.net - Unit Test fails on Run All but not on Run Selected Tests with all selected

Question

I'm getting some weird behavior. If I click "Run All" in Test Explorer then 1 of my unit tests fails, but if I select all tests and click "Run Selected Tests" then the unit test passes.

The test that is failing is throwing a reflection error: System.Reflection.TargetException: Non-static method requires a target. on a type which is defined in the dll code which I'm testing. There doesn't appear to be anything odd with the class - there are plenty of other classes defined in the dll which reflection is happy with. I've included the test stack trace below.

NB It is a complex test - it reads inputs & expected answers from an .xlsx file, populates a LocalDb with data from the xlsx, performs a calculation using the data in the LocalDB and then compares the calculated to expected values. However as I say it is working and works when I run all tests (using select all > Run Selected Test).

What is different about Run All? Any insights would be appreciated.

I have tried a clean and rebuild with no luck. Catching and logging reflection error indicates that the GetValue call is throwing for every property I attempt to access on the type - but only when run by "Run All" and only on this one type? (if I catch the error then all GetValues succeed on all other types).

Stack Trace

Test Name:  IT_CheckCashOnly1DepositOutputValues
Test FullName:  Lib.AE.Tests.Integration.CalculationTests.IT_CheckCashOnly1DepositOutputValues
Test Source:    c:
etreturn.co.zaMainNetReturnLib.AE.TestsIntegrationTestsCalculationTest.cs : line 23
Test Outcome:   Failed
Test Duration:  0:00:00.1661906

Result Message: 
Test method Lib.AE.Tests.Integration.CalculationTests.IT_CheckCashOnly1DepositOutputValues threw exception: 
System.Reflection.TargetException: Non-static method requires a target.
Result StackTrace:  
at System.Reflection.RuntimeMethodInfo.CheckConsistency(Object target)
at System.Reflection.RuntimeMethodInfo.InvokeArgumentsCheck(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at System.Reflection.RuntimePropertyInfo.GetValue(Object obj, BindingFlags invokeAttr, Binder binder, Object[] index, CultureInfo culture)
at System.Reflection.RuntimePropertyInfo.GetValue(Object obj, Object[] index)
at Lib.AE.Xlsx.XlsxHelper.Compare[T](T expected, T calculated, ExcelWorksheet ws, Int32 r, Int32 colStart, Boolean& valid) in c:
etreturn.co.zaMainNetReturnLib.AEXlsxXlsxHelper.cs:line 101
at Lib.AE.Xlsx.XlsxWorkSheet_SharePNL.CompareXlsx(ExcelPackage pck, List`1 expectedXlsx, ValuationCalculation calc) in c:
etreturn.co.zaMainNetReturnLib.AEXlsxXlsxSharePNL.cs:line 143
at Lib.AE.Tests.Integration.CalculationTests.CheckCalculationResults(String xlsxDocToLoad, WorkSheets testingScenarios) in c:
etreturn.co.zaMainNetReturnLib.AE.TestsIntegrationTestsCalculationTest.cs:line 64
at Lib.AE.Tests.Integration.CalculationTests.IT_CheckCashOnly1DepositOutputValues() in c:
etreturn.co.zaMainNetReturnLib.AE.TestsIntegrationTestsCalculationTest.cs:line 23

---

Solution

So this turned out to be (a) my problem - my unit test shared state with another unit test and (b) an order problem. Be aware that it's not obvious which order TestExplorer will run your tests in. I created a new UnitTestProject with 2 x UnitTest .cs files and three TestMethods each i.e.:

UnitTest1.cs

    [TestMethod]
    public void ONE_AAA() {}

    [TestMethod]
    public void ONE_BBB() {}

    [TestMethod]
    public void ONE_CCC() {}

UnitTest2.cs

    [TestMethod]
    public void TWO_CCC() {}

    [TestMethod]
    public void TWO_BBB() {}

    [TestMethod]
    public void TWO_AAA() {}

Then ran these tests via two methods i.e. (1) Run All (2) Select all and Run Selected Tests, and logged the order in which the tests were started by TestExplorer. The results for Run Selected Tests are fairly unintuitive:

-- Run All
2013-01-16 11:53:47.4062 INFO TestInitialize: ONE_AAA
2013-01-16 11:53:47.4122 INFO TestCleanup: ONE_AAA
2013-01-16 11:53:47.4122 INFO TestInitialize: ONE_BBB
2013-01-16 11:53:47.4122 INFO TestCleanup: ONE_BBB
2013-01-16 11:53:47.4122 INFO TestInitialize: ONE_CCC
2013-01-16 11:53:47.4282 INFO TestCleanup: ONE_CCC
2013-01-16 11:53:47.4282 INFO TestInitialize: TWO_CCC
2013-01-16 11:53:47.4282 INFO TestCleanup: TWO_CCC
2013-01-16 11:53:47.4282 INFO TestInitialize: TWO_BBB
2013-01-16 11:53:47.4282 INFO TestCleanup: TWO_BBB
2013-01-16 11:53:47.4282 INFO TestInitialize: TWO_AAA
2013-01-16 11:53:47.4282 INFO TestCleanup: TWO_AAA

-- Select All > Run Selected
2013-01-16 11:55:26.0139 INFO TestInitialize: TWO_BBB
2013-01-16 11:55:26.0139 INFO TestCleanup: TWO_BBB
2013-01-16 11:55:26.0249 INFO TestInitialize: ONE_BBB
2013-01-16 11:55:26.0249 INFO TestCleanup: ONE_BBB
2013-01-16 11:55:26.0249 INFO TestInitialize: TWO_AAA
2013-01-16 11:55:26.0249 INFO TestCleanup: TWO_AAA
2013-01-16 11:55:26.0249 INFO TestInitialize: TWO_CCC
2013-01-16 11:55:26.0249 INFO TestCleanup: TWO_CCC
2013-01-16 11:55:26.0249 INFO TestInitialize: ONE_CCC
2013-01-16 11:55:26.0249 INFO TestCleanup: ONE_CCC
2013-01-16 11:55:26.0249 INFO TestInitialize: ONE_AAA
2013-01-16 11:55:26.0249 INFO TestCleanup: ONE_AAA

question from:https://stackoverflow.com/questions/14349512/unit-test-fails-on-run-all-but-not-on-run-selected-tests-with-all-selected

Answer 1

I have been caught out in the past with running multiple unit tests in that the order in which the tests run may not be in the order in which they are declared in the test class, and may in fact be in order of test method name. e.g. if I have

[Test]
public void PreviousTest()
{
}

[Test]
public void LaterTest()
{
}

Then LaterTest gets run first as its name appears before PreviousTest when ordered alphabetically.

This won't matter if all of your tests are entirely independent, but if they are modifying shared resources then you may get unusual behaviour if you were expecting that LaterTest's changes wouldn't have any effect on PreviousTest due to it being declared second.