Fail Loudly

I ended my last post, Generating Test Cases at Runtime, with a pop quiz: “Can you spot the bug? It’s possible to write test methods that never get invoked.” Today, let’s cover the bug as well as its fix.

Reproducing the Issue

At the end of the last post, we had a working test class and a convention which could match up a static IEnumerable generator with a test accepting a T. Each value yielded by the IEnumerable would be passed to the test as a single pass-or-fail-result. It worked just fine for that example: TestWithNoParameters() was called once, and TestWithInterestingParameter(Person) was called once for every Person yielded by the static Person-generating methods.

If we add a third test method, though, one whose incoming T type doesn’t happen to match any input-generating static methods, something weird happens:

That’s terrible! Our new test doesn’t pass, and it doesn’t even fail. A test runner’s prime directive is to provide reliable pass/fail results. Anything that is a test should have a result, and this new method is most certainly a test according to the convention’s discovery rules:

The Diagnosis

Over my last few posts, we’ve dealt with Fixie’s parameterized tests hook: a Func<MethodInfo, IEnumerable<object[]». When you want to define the meaning of a parameterized test method, you must yield an object[] once for every time you want the test method to be called. Until now, I’ve been assuming that the convention author’s Func will always yield at least once. We even saw an attempt to do so in last week’s convention:

When a test method had no parameters at all, it would explicitly return a single empty object[], meaning “Call the test method once with no args.” FindInputs(…), on the other hand, could still yield zero object[], and thus zero requests to call the method. Here’s a bug that causes silent failures, and the worst failure is the one that keeps on happening without anyone knowing. We often hear the advice to “fail fast”, but there’s an implicit “…and fail as loudly as possible” in there, too.

The Fix: Fail Loudly

In order to fail loudly, Fixie needed to treat test methods with unsatisfied parameters as failures, with a failure message explaining that fact. A nice side effect is that it’s easier for convention authors to do the right thing: yield when you have a meaningful input, don’t when you don’t, and let Fixie autofail any test method that still couldn’t be called.

Our convention from last time gets simpler, now that we don’t have a lurking edge case to care about, and the output now correctly complains about the new test method:

The fix is available in Fixie 0.0.1.101.