Acceptable test failures

As Derick Bailey pointed out in my last post, one of the annoyances with ReSharper is the NotImplementedException it puts in when you generate a method.  Going from the TDD side, this is exactly what we don’t want when we’re doing Red-Green-Refactor.  There’s probably a setting somewhere, but I haven’t found it.  It does bring up a larger issue – what is an acceptable test failure?  That’s fairly simple:

A test should have only one reason to fail.

When we’re doing Red-Green-Refactor, we fail the test on purpose, mainly to prove that our test is correct.  Without a test failing for the reasons we want, we’re not entirely confident in the correctness of our tests.  Normally, I’ll do something like this:

  • Create the test/spec method
  • Write the client code and behavior I want, even if the members don’t exist
  • Use R# to create members that don’t exist, providing implementations that fail assertions

So I’ll wind up creating a test like this:

[Test]
public void Should_mark_order_as_processed()
{
    var orderProcessor = new OrderProcessor();
    var order = new Order();

    orderProcessor.ProcessOrder(order);

    order.IsProcessed.ShouldBeTrue();
}

At this point, the ProcessOrder method does not exist.  The problem comes in when I use either VS or R# to create the method:

public void ProcessOrder(Order order)
{
    throw new System.NotImplementedException();
}

Now, R# at least highlights the entire row, so I can remove the NIE block fairly easily.  But I’d rather not see it at all, it violates what a good unit test should be.  Unit tests should only fail because of a failed assertion.  I want the above test to fail when and only when the last line, the assertion, executes and fails.

Executing the test as-is gives me a test failure, but not the correct failure.  Things like NotImplementedExceptions or worse, NullReferenceExceptions, cloud the triangulation of the Red-Green-Refactor steps.  When I don’t get the test to fail for the reasons I want, I never really know if the test is correct.  It may pass later, but I don’t know if it will fail correctly.  This is one of the more insidious test smells, as it can lead to a false sense of confidence in the correctness of my specifications.  Accidental truths in my system mislead developers on the true behavior of the system.

When tests fail, and not because of an assertion failure, it’s a smell that I need to write another test.  I then need to go back and ensure my test can fail for the reasons I specify, and only then do I have a valid test.  By following true Red-Green-Refactor steps, and being vigilant on creating new tests when aberrant tests failures pop up, I can have well-founded confidence in the behavior of my system.

Related Articles:

Post Footer automatically generated by Add Post Footer Plugin for wordpress.

About Jimmy Bogard

I'm a technical architect with Headspring in Austin, TX. I focus on DDD, distributed systems, and any other acronym-centric design/architecture/methodology. I created AutoMapper and am a co-author of the ASP.NET MVC in Action books.
This entry was posted in TDD. Bookmark the permalink. Follow any comments here with the RSS feed for this post.
  • http://creedcultcode.blogspot.com Dale Smith

    But isn’t your test implicitly asserting that no exception will be thrown, since you didn’t mark it with an [ExpectedException()] attribute? And if so, does the failure qualify as an assertion failure? Does this test not count as one that notifies you of behavior you didn’t expect?

  • http://jimmybogard.lostechies.com Jimmy Bogard

    @Dale

    That’s what I meant by “accidental” failures, you also see “accidental” passes. In an accidental failure, I never see the test fail because of the assertion failure. If I get an accidental failure, fine, but create a new test for it. I want all tests to fail only because of the assertion failure.

  • http://creedcultcode.blogspot.com Dale Smith

    So your saying that the tests should positively express the code’s expected behavior, right? Totally agreed. To be honest, I was just playing devil’s advocate. Please don’t tell my pastor.

  • http://www.rasmuskl.dk Rasmus Kromann-Larsen

    Hey Jimmy,

    Just noticed this in my Resharper Live Templates explorer and thought about this post.

    It seems newly generated methods (by resharper) use the live template called “defaultBodyTemplate” under Predefined Templates -> No Language -> System

    If you change this – no more NotImplementedExceptions…

    Enjoy.

  • http://www.rasmuskl.dk Rasmus Kromann-Larsen

    Btw I’m really impressed with your 58 million readers according to your feedburner thingy… 58393K.. :-P