The many faces of ExpectedException

I’m not a fan of the style most xUnit frameworks suggest for verifying exceptions.  After seeing the problems with using an attribute to perform assertions, many frameworks have come up with their own way of their own way of doing it.  Outside using a different xUnit framework, you can also perform this verification manually inside your test.  It’s not quite as explicit, but it’s possible.

Suppose we have the following OrderProcessor that we’d like to test:

public class OrderProcessor
{
    private readonly INotificationService _notificationService;

    public OrderProcessor(INotificationService notificationService)
    {
        if (notificationService == null)
            throw new ArgumentNullException("notificationService");

        _notificationService = notificationService;
    }

    public void ProcessOrder(Order order)
    {
        if (order == null)
            throw new ArgumentNullException("order");
    }
}

Note that the same exception type is thrown from two different locations.  What I’d like to do is create a test specify some behavior that when ProcessOrder is called, it errors when the Order is null.  There are several ways of doing this, some that work better than others.

Using attributes

To set an expectation that a block of code will throw a certain type of exception, you need to decorate your test method with an ExpectedException attribute.  Both NUnit, MSTest, and MbUnit provide this attribute (all with the same name).  For example, here’s an NUnit test verifying the exception test:

[Test]
[ExpectedException(typeof(ArgumentNullException))]
public void Should_throw_ArgumentNullException_when_the_Order_is_null()
{
    OrderProcessor processor = new OrderProcessor(null);

    processor.ProcessOrder(null);
}

This test passes, but for the wrong reason.  When I pass in the null argument for the OrderProcessor constructor, the constructor throws the exception, and the ProcessOrder method is never even executed!  This tests passes, but only accidentally.

Additionally, it’s not clear from the body of the test what specifically is supposed to fail.  As long as anything inside the method body throws the correct exception, the test will pass.  I don’t like tests passing accidentally and I don’t like opaque tests.

Although NUnit’s ExpectedException attribute has additional parameters for checking messages, etc., it’s still a kludge around the real problem of using an attribute to perform assertions.

Manual checking

This is my preferred method when using NUnit, MSTest or MbUnit.  In this style, I put a try-catch block explicitly around the statement of code I expect to throw the exception.  Using NUnit:

[Test]
public void Should_throw_ArgumentNullException_when_the_Order_is_null()
{
    OrderProcessor processor = new OrderProcessor(null);

    try
    {
        processor.ProcessOrder(null);
        Assert.Fail("Should throw ArgumentNullException");
    }
    catch (ArgumentNullException)
    {
    }
}

If the test code gets past ProcessOrder, I force a test failure (and provide a meaningful message).  This test now fails, as the constructor is the statement throwing the exception.  I get a fast, meaningful failure when ProcessOrder does not throw the expected exception.

I test a specific exception type by only including that exception in the catch block.  If some other exception is thrown, the exception won’t get caught by my specific catch block, and the test will fail.

There are several variations of the manual checking, but the basic idea is the same.  Put a try-catch block only around the statement that should throw, and put in specific checking to make sure the test passes only when the exception you want is thrown.

xUnit.net style – Assert.Throws

Instead of using an attribute or doing manual try-catch blocks, xUnit.net introduces an Assert.Throws method, where I can supply behavior through a delegate.  This is all a mouthful, it’s easier to see the code:

[Fact]
public void Should_throw_ArgumentNullException_when_the_Order_is_null()
{
    OrderProcessor processor = new OrderProcessor(new NotificationService());

    Assert.Throws<ArgumentNullException>(() => processor.ProcessOrder(null));
}

I supply the block that should throw in the lambda expression, but this could be an anonymous delegate or function name.

I really like this style as it combines the assertion, the exception type, and the block that throws into one readable statement.  Although the parameter-less lambda block can look strange, I’m able to confine the block that expects the exception to the one statement that should throw, just like I was able to do in the manual example.

Looking at the source code behind the xUnit.net implementation, behind the scenes it does something very similar to the manual example.  It just wraps it up into one nice, neat call.

NBehave style – fluent interface

As we’re pushing towards the 0.4 release of NBehave, we’re trying more fluent syntax using BDD-style extension methods that wrap existing xUnit implementations.  For example, a fluent wrapper over MSTest would now look like:

[TestMethod]
public void Should_throw_ArgumentNullException_when_the_Order_is_null()
{
    OrderProcessor processor = new OrderProcessor(new NotificationService());

    processor.WhenCalling(x => x.ProcessOrder(null)).ShouldThrow<ArgumentNullException>();
}

In the earlier example using lambdas, I was forced to use a parameter-less lambda that looked a little strange.  In this case, the specification is meant to be as human-readable as possible, almost sentence-like.  With some generics work, the lambda gets strongly typed to the class under test, so I get all of the IntelliSense and refactoring goodness.  (I’m still trying to reverse the order of the methods…not as simple as it seems.)

So many choices…

When it comes down to it, you want a test that:

  • Fails for the reasons you want
  • Can be immediately understood on what it’s testing

Whichever choice you go with, as long as it conforms to the above criteria, you should come out fine in the end.  For me, the attribute method is the only one that doesn’t achieve either of the above.  Exceptions seem to have been made a special case, but since lambdas have made anonymous delegates much easier to deal with, better alternatives exist for the verbose or broken mechanisms we had to use previously.

Related Articles:

Post Footer automatically generated by Add Post Footer Plugin for wordpress.

About Jimmy Bogard

I'm a technical architect with Headspring in Austin, TX. I focus on DDD, distributed systems, and any other acronym-centric design/architecture/methodology. I created AutoMapper and am a co-author of the ASP.NET MVC in Action books.
This entry was posted in TDD. Bookmark the permalink. Follow any comments here with the RSS feed for this post.
  • http://scottic.us Scott Bellware

    I prefer an opposite style for fluent interface, where the exception is documented earlier in the statement, like:

    typeof(Exception).IsThrownBy(someDelegate)

    Ultimately, when I have my druthers, I prefer to capture the exception during setup and test it later:

    SetUp()
    {
    _exception = someDelegate.GetException()
    }

    should_not_be_allowed
    {
    _exception.ShouldNotBeNull()
    }

    should_raise_System_Exception
    {
    _exception.Is(typeof(System.Exception))
    }

    Jeez. You guys and your angle brackets. Just when you thought it was safe to go back in the configuration DSL, angle brackets now attack C# developers right here on DRY land :)

    Generics. Meh. Give me Ruby and lets just let angle brackets fade into history.

  • http://jimmybogard.lostechies.com Jimmy Bogard

    @Scott

    But what else will satisfy my techno-fetishism besides angl-ey brackets?

    I do like the “Is(typeof”, I had been using something like “ShouldBeOfType(typeof(” and the “typeof” seemed redundant. Back to the drawing board…

  • http://www.bluespire.com/blogs Christopher Bennage

    This is good stuff. It’s got me thinking.

  • http://www.bgeek.net Owen

    In NSpecify (https://sourceforge.net/projects/nspecify/) there is the availability to do Specify.That(Repository.SaveChanges).Must.Be.Disallowed
    I’m just thinking of gathering some changes from this post so that it reads a little better
    [context]
    public class WithANewWidget
    {
    Widget widget;
    [BeforeAll]
    public void GivenANewWidget()
    {
    widget = new Widget();
    }
    [Specification]
    public void TheWidgetShouldNotBeSavable()
    {
    Specify.That(Widget.Save).Must.Be.Dissalowed.WithError(typeof(ValidationException));
    Specify.That(Widget.Save).Must.Be.Dissalowed.WithError(“The Widget is not valid”);
    Specify.That(Widget.Save).Must.Be.Dissalowed.WithError(typeof(ValidationException)).AndMessage(“The widget is not valid”);
    }
    }

  • http://blog.alecl.com/ Alec Lazarescu

    I’ve also run into the same question on how to neatly handle checking for exceptions in unit tests. I’ve come up with the following helper methods.

    public class SpecHelpers
    {
    public static void ExpectException(Action methodCall) where T : Exception
    {
    bool caught = false;

    try
    {
    methodCall();
    }
    catch (T)
    {
    caught = true;
    }

    if (!caught)
    Assert.Fail(String.Format(“Expected exception {0} not thrown”));
    }

    public static void ExpectExceptionMessageStartsWith(Action methodCall, string startsWith) where T : Exception
    {
    bool caught = false;

    try
    {
    methodCall();
    }
    catch (T ex)
    {
    if (!ex.Message.StartsWith(startsWith, true, null))
    Assert.Fail(String.Format(“Expected exception {0} thrown, but message didn’t start with \”{1}\”. Message: \”{2}\”",
    ex.GetType().FullName, startsWith, ex.Message));

    caught = true;
    }

    if (!caught)
    Assert.Fail(String.Format(“Expected exception {0} not thrown”), typeof(T).FullName);
    }

    public static void ExpectExceptionMessageContains(Action methodCall, string contains) where T : Exception
    {
    bool caught = false;

    try
    {
    methodCall();
    }
    catch (T ex)
    {
    if (ex.Message.IndexOf(contains, StringComparison.InvariantCultureIgnoreCase) < 0)
    Assert.Fail(String.Format("Expected exception {0} thrown, but message didn't contain \"{1}\". Message: \"{2}\"",
    ex.GetType().FullName, contains, ex.Message));

    caught = true;
    }

    if (!caught)
    Assert.Fail(String.Format("Expected exception {0} not thrown"), typeof(T).FullName);
    }
    }

    quick example:

    private bool ThrowEx()
    {
    throw new Exception();
    }

    private bool ThrowExContains()
    {
    throw new NullReferenceException("a special message");
    }

    [Test]
    public void TestExceptionWrapper()
    {
    // .NET 3.5 style with lamba notation
    SpecHelpers.ExpectException(() => ThrowEx());
    SpecHelpers.ExpectExceptionMessageContains(() => ThrowExContains(), “special”);

    // .NET 2.0 style with anonymous delegate
    SpecHelpers.ExpectExceptionMessageContains(delegate() { ThrowExContains(); }, “special”);
    }

    Full post: http://blog.alecl.com/archive/2008/10/25/syntactic-sugar-for-thrown-exception-checking-with-generics.aspx

  • http://jimmybogard.lostechies.com Jimmy Bogard

    @Alex

    Looks good, thanks for the link!