Acceptable test failures

As Derick Bailey pointed out in my last post, one of the annoyances with ReSharper is the NotImplementedException it puts in when you generate a method.  Going from the TDD side, this is exactly what we don’t want when we’re doing Red-Green-Refactor.  There’s probably a setting somewhere, but I haven’t found it.  It does bring up a larger issue – what is an acceptable test failure?  That’s fairly simple:

A test should have only one reason to fail.

When we’re doing Red-Green-Refactor, we fail the test on purpose, mainly to prove that our test is correct.  Without a test failing for the reasons we want, we’re not entirely confident in the correctness of our tests.  Normally, I’ll do something like this:

• Create the test/spec method
• Write the client code and behavior I want, even if the members don’t exist
• Use R# to create members that don’t exist, providing implementations that fail assertions

So I’ll wind up creating a test like this:

[Test]
public void Should_mark_order_as_processed()
{
    var orderProcessor = new OrderProcessor();
    var order = new Order();

    orderProcessor.ProcessOrder(order);

    order.IsProcessed.ShouldBeTrue();
}

At this point, the ProcessOrder method does not exist.  The problem comes in when I use either VS or R# to create the method:

public void ProcessOrder(Order order)
{
    throw new System.NotImplementedException();
}

Now, R# at least highlights the entire row, so I can remove the NIE block fairly easily.  But I’d rather not see it at all, it violates what a good unit test should be.  Unit tests should only fail because of a failed assertion.  I want the above test to fail when and only when the last line, the assertion, executes and fails.

Executing the test as-is gives me a test failure, but not the correct failure.  Things like NotImplementedExceptions or worse, NullReferenceExceptions, cloud the triangulation of the Red-Green-Refactor steps.  When I don’t get the test to fail for the reasons I want, I never really know if the test is correct.  It may pass later, but I don’t know if it will fail correctly.  This is one of the more insidious test smells, as it can lead to a false sense of confidence in the correctness of my specifications.  Accidental truths in my system mislead developers on the true behavior of the system.

When tests fail, and not because of an assertion failure, it’s a smell that I need to write another test.  I then need to go back and ensure my test can fail for the reasons I specify, and only then do I have a valid test.  By following true Red-Green-Refactor steps, and being vigilant on creating new tests when aberrant tests failures pop up, I can have well-founded confidence in the behavior of my system.