Red For The Right Reason: Fail By Assertion, Not By Anything Else

   1: [Test]

   2: public void An_Interaction_Test()

   3: {

   4:   var anotherObject = MockRepository.CreateMock<IAmThatThing>();

   5:   var myObject = new MyObject(anotherObject);

   6:   myObj.DoSomething();

   7:   anotherObject.AssertWasCalled(ao => ao.ThatThing());

   8: }

</div> </div>

If the DoSomething method throws a NotImplementedException here, the method called ThatThing() on the IAmThatThing interface may not be called, but we don’t actually know that it was not called. We are not proving that the method is not called and that the assertion can fail the test. Rather, we are only proving that an exception can fail the test.

Complex Test Scenarios

f the object being tested has a private method and let put a NotImplementedException into that private method, we have not proven that the method being called is not yet implemented. In fact, we have no certainty of which method in the object is not implemented until we look at the stack trace that is produced by the exception being thrown. If, on the other hand, the private method is expected to return a specific value under the specific circumstances that were set up by the public method being called from the test then we can return a value that is known to be incorrect and the assertion will fail for the right reason.

If we are dealing with a more complex set of dependencies and interactions then the NotImplementedException becomes more and more of an obstruction to the tests for every assertion that we have. Having an exception will fail the test when we have 5 asserts against interactions means that we have 5 tests that are failing for an invalid reason.

False Positives, False Negatives

A false positive exists when a test passes for the wrong reasons, and a false negative exists when a test fails for the wrong reasons. Thomas said this about false positives in his comment:

because with a NotImplementedException a test will NEVER become green for the wrong reasons, and therefore there will be no danger that it is overlooked in the (sometimes hectic) production process.

He is right about that, no doubt. A NotImplementedException certainly will prevent false positives while it is in place, but there are two problems that I see with this statement:

1. You are introducing false negatives, which are just as dangerous as false positives. Since the test is failing because of an exception and not because an assertion found an abnormality, the test failure is a false negative
2. You have not prevented false positives, only postponed them. Once the NotImplementedException is removed, the code is open to the possibility of false positives

We must be diligent about preventing false positives and false negatives, both, not just false positives.

When To Test For Exceptions

There are times when it’s important to do both positive and negative testing around exceptions… when your code throws them. This example shows that we expect a process to not throw any exceptions under one condition, and we expect it to throw an exception under another condition.

   1: [Test]

   2: public void Dont_Throw_Exceptions()

   3: {

   4:   var myObject = new MyObject();

   5:   Exception caughtException = null;

   6:   try

   7:   {

   8:     myObject.DoSomething(withThisCondition);

   9:   }

  10:   catch(Exception ex)

  11:   {

  12:     caughtException = ex;

  13:   }

  14:   Assert.IsNull(caughtException);

  15: }

  16:  

  17: [Test]

  18: public void Throw_Exceptions()

  19: {

  20:   var myObject = new MyObject();

  21:   Exception caughtException = null;

  22:   try

  23:   {

  24:     myObject.DoSomething(basedOnAnotherCondition);

  25:   }

  26:   catch(Exception ex)

  27:   {

  28:     caughtException = ex;

  29:   }

  30:   Assert.IsNotNull(caughtException);

  31: }