The many faces of ExpectedException

I’m not a fan of the style most xUnit frameworks suggest for verifying exceptions.  After seeing the problems with using an attribute to perform assertions, many frameworks have come up with their own way of their own way of doing it.  Outside using a different xUnit framework, you can also perform this verification manually inside your test.  It’s not quite as explicit, but it’s possible.

Suppose we have the following OrderProcessor that we’d like to test:

public class OrderProcessor
{
    private readonly INotificationService _notificationService;

    public OrderProcessor(INotificationService notificationService)
    {
        if (notificationService == null)
            throw new ArgumentNullException("notificationService");

        _notificationService = notificationService;
    }

    public void ProcessOrder(Order order)
    {
        if (order == null)
            throw new ArgumentNullException("order");
    }
}

Note that the same exception type is thrown from two different locations.  What I’d like to do is create a test specify some behavior that when ProcessOrder is called, it errors when the Order is null.  There are several ways of doing this, some that work better than others.

Using attributes

To set an expectation that a block of code will throw a certain type of exception, you need to decorate your test method with an ExpectedException attribute.  Both NUnit, MSTest, and MbUnit provide this attribute (all with the same name).  For example, here’s an NUnit test verifying the exception test:

[Test]
[ExpectedException(typeof(ArgumentNullException))]
public void Should_throw_ArgumentNullException_when_the_Order_is_null()
{
    OrderProcessor processor = new OrderProcessor(null);

    processor.ProcessOrder(null);
}

This test passes, but for the wrong reason.  When I pass in the null argument for the OrderProcessor constructor, the constructor throws the exception, and the ProcessOrder method is never even executed!  This tests passes, but only accidentally.

Additionally, it’s not clear from the body of the test what specifically is supposed to fail.  As long as anything inside the method body throws the correct exception, the test will pass.  I don’t like tests passing accidentally and I don’t like opaque tests.

Although NUnit’s ExpectedException attribute has additional parameters for checking messages, etc., it’s still a kludge around the real problem of using an attribute to perform assertions.

Manual checking

This is my preferred method when using NUnit, MSTest or MbUnit.  In this style, I put a try-catch block explicitly around the statement of code I expect to throw the exception.  Using NUnit:

[Test]
public void Should_throw_ArgumentNullException_when_the_Order_is_null()
{
    OrderProcessor processor = new OrderProcessor(null);

    try
    {
        processor.ProcessOrder(null);
        Assert.Fail("Should throw ArgumentNullException");
    }
    catch (ArgumentNullException)
    {
    }
}

If the test code gets past ProcessOrder, I force a test failure (and provide a meaningful message).  This test now fails, as the constructor is the statement throwing the exception.  I get a fast, meaningful failure when ProcessOrder does not throw the expected exception.

I test a specific exception type by only including that exception in the catch block.  If some other exception is thrown, the exception won’t get caught by my specific catch block, and the test will fail.

There are several variations of the manual checking, but the basic idea is the same.  Put a try-catch block only around the statement that should throw, and put in specific checking to make sure the test passes only when the exception you want is thrown.

xUnit.net style – Assert.Throws

Instead of using an attribute or doing manual try-catch blocks, xUnit.net introduces an Assert.Throws method, where I can supply behavior through a delegate.  This is all a mouthful, it’s easier to see the code:

[Fact]
public void Should_throw_ArgumentNullException_when_the_Order_is_null()
{
    OrderProcessor processor = new OrderProcessor(new NotificationService());

    Assert.Throws<ArgumentNullException>(() => processor.ProcessOrder(null));
}

I supply the block that should throw in the lambda expression, but this could be an anonymous delegate or function name.

I really like this style as it combines the assertion, the exception type, and the block that throws into one readable statement.  Although the parameter-less lambda block can look strange, I’m able to confine the block that expects the exception to the one statement that should throw, just like I was able to do in the manual example.

Looking at the source code behind the xUnit.net implementation, behind the scenes it does something very similar to the manual example.  It just wraps it up into one nice, neat call.

NBehave style – fluent interface

As we’re pushing towards the 0.4 release of NBehave, we’re trying more fluent syntax using BDD-style extension methods that wrap existing xUnit implementations.  For example, a fluent wrapper over MSTest would now look like:

[TestMethod]
public void Should_throw_ArgumentNullException_when_the_Order_is_null()
{
    OrderProcessor processor = new OrderProcessor(new NotificationService());

    processor.WhenCalling(x => x.ProcessOrder(null)).ShouldThrow<ArgumentNullException>();
}

In the earlier example using lambdas, I was forced to use a parameter-less lambda that looked a little strange.  In this case, the specification is meant to be as human-readable as possible, almost sentence-like.  With some generics work, the lambda gets strongly typed to the class under test, so I get all of the IntelliSense and refactoring goodness.  (I’m still trying to reverse the order of the methods…not as simple as it seems.)

So many choices…

When it comes down to it, you want a test that:

• Fails for the reasons you want
• Can be immediately understood on what it’s testing

Whichever choice you go with, as long as it conforms to the above criteria, you should come out fine in the end.  For me, the attribute method is the only one that doesn’t achieve either of the above.  Exceptions seem to have been made a special case, but since lambdas have made anonymous delegates much easier to deal with, better alternatives exist for the verbose or broken mechanisms we had to use previously.