Cleaning Up Test Failure Noise

The Fixie test framework has no built-in assertion library, since assertions are an orthogonal concern to the nuts and bolts of test discovery and test execution. Although I stand by the decision to avoid including a built-in assertion library, there is one neat thing NUnit does with its own assertions in order to simplify output of test failures. Today, we’ll see Fixie’s answer to that feature.

NUnit simplifies its own output when exceptions are thrown by its own assertion library infrastructure. For instance, when NUnit’s Assert.AreEqual(int, int) fails by throwing an exception, the output deliberately excludes stack trace lines pointing within the implementation of AreEqual, and deliberately excludes the name of the exception type. This filtering allows typical test failure output to remain as simple and direct as possible, pointing the developer to the line where their own test failed.

Since Fixie has no assertion library of its own, you may instruct it which types make up your assertion library’s implementation details. Consider some hypothetical assertion library:

namespace Hypothetical.Assertion.Library
{
    public static class Assert
    {
        public static void AreEqual(int expected, int actual)
        {
            if (expected != actual)
                throw new AssertionException(String.Format("Expected {0} but was {1}.", expected, actual));
        }
    }

    public class AssertionException : Exception
    {
        public AssertionException(string message)
            : base(message)
        {
        }
    }
}

Out of the box, Fixie doesn’t distinguish AssertionException from any other Exception (How could it?), so all of the exception details appear in the output. Consider a test class with some tests that will surely fail, and the corresponding default verbose output:

using Hypothetical.Assertion.Library;

public class CalculatorTests
{
    readonly Calculator calculator;

    public CalculatorTests()
    {
        calculator = new Calculator();
    }

    public void ShouldAdd()
    {
        Assert.AreEqual(999, calculator.Add(2, 3));
    }

    public void ShouldSubtract()
    {
        Assert.AreEqual(999, calculator.Subtract(5, 3));
    }
}

Test 'CalculatorTests.ShouldAdd' failed: Hypothetical.Assertion.Library.AssertionException
    Expected 999 but was 5.
    Conventions\DefaultConvention.cs(10,0): at Hypothetical.Assertion.Library.Assert.AreEqual(Int32 expected, Int32 actual)
    CalculatorTests.cs(14,0): at CalculatorTests.ShouldAdd()

Test 'CalculatorTests.ShouldSubtract' failed: Hypothetical.Assertion.Library.AssertionException
    Expected 999 but was 2.
    Conventions\DefaultConvention.cs(10,0): at Hypothetical.Assertion.Library.Assert.AreEqual(Int32 expected, Int32 actual)
    CalculatorTests.cs(19,0): at CalculatorTests.ShouldSubtract()

The implementation details of assertion libraries are rarely interesting to the developer. A custom convention can be instructed to simplify failure output by listing the types that make up the assertion library:

public class CustomConvention : Convention
{
    public CustomConvention()
    {
        Classes
            .NameEndsWith("Tests");

        Methods
            .Where(method => method.IsVoid());

        HideExceptionDetails
            .For<AssertionException>()
            .For(typeof(Assert));
    }
}

Rerunning the failing tests, Fixie simplifies the output, directing the developer to the actual failing line of test code:

Test 'CalculatorTests.ShouldAdd' failed:
    Expected 999 but was 5.
    CalculatorTests.cs(14,0): at CalculatorTests.ShouldAdd()

Test 'CalculatorTests.ShouldSubtract' failed:
    Expected 999 but was 2.
    CalculatorTests.cs(19,0): at CalculatorTests.ShouldSubtract()

In addition to identifying the types which make up your assertion library of choice, your custom convention may also list assertion extension classes defined in your own projects, further simplifying your output during failures.

Related Articles:

Post Footer automatically generated by Add Post Footer Plugin for wordpress.

About Patrick Lioi

I am a Senior Consultant at Headspring in Austin, TX. I created the Parsley text parsing library and the Fixie test framework.
This entry was posted in Uncategorized. Bookmark the permalink. Follow any comments here with the RSS feed for this post.
  • Sarah Nelson

    Hi All, I am bit confused whether to opt for Scrum credentials or attend a PMP prep course / PMP classes preparing to take PMP certification exams . Whats your take?