Getting value out of your unit tests

Unit tests, with TDD in particular, are the most efficient way I’ve found in creating behavior for my application.  For lasting value, beyond just the safety net of “if I change something, will something break”, requires extra discipline, and a more refined manner in which we go about doing TDD.  One of the most frustrating aspects of TDD done wrong is dozens of unit tests that provide no other value than they might fail.

Back when Scott and others were refining TDD into Context/Specification BDD-style, I remember him posing the question, “if we actually treat tests as documentation, what is the result?”.  For a lot of unit tests, that documentation is a muddled, miserable mess.  Nothing can be more frustrating than a failing unit test that provides zero insight into why the test exists in the first place.

My first year or so of doing TDD produced exactly that, a mess.  No insight into the what or why of my system’s behavior, but merely a retrospective look on what each individual class did.  But by adding a few rules, as well as personal guidelines, I’ve noticed my tests have started to provide what I really wanted – a description of the behavior of the system.  These rules injected that value in my tests that would have otherwise made the tests just dead weight, code I avoided after I wrote them.

Rule #1 – Test names should describe the what and the why, from the user’s perspective

One way to do this easily is the Context/Observation(specification) naming style.  I’m not the greatest at explaining exactly the BDD style, I’d rather defer to JP Boodhoo, Scott Bellware’s article, Aaron Jensen and Raymond Lewallen on this one.  But the general idea is that an outside developer should be able to read the test/class name, and clearly understand what the intended observable behavior is, for a given context.  Here’s a bad, bad example:

[Test]
public void ValidationShouldWorkCorrectly()

Validation should work correctly.  Hmm.  “Correctly” is in the eye of the beholder.  “Correctly” depends on the circumstances.  “Correctly” depends on your definition of “correct”.  The kicker is, if this test fails in the future, how will I know if it is because the test is wrong or the code is wrong?  I won’t.  And at that point, I have to make a judgment call on whether the test is holding its water or not.  Sometimes, I’ll just make the test pass, sometimes I’ll remove the test, and sometimes I’ll spend a lot of wasted time trying to figure why the heck the test was written in the first place.

That’s perhaps the most frustrating aspect.  A unit test that was supposed to provide value when it failed, instead only caused confusion and consternation.

Rule #2 – Tests are code too, give them some love

Whether you think so or not, you will have to maintain your tests.  You don’t have to model your tests like you would, but duplication matters in your tests.  Do yourself a favor, right now, go order the xUnit Test Patterns book.  TDD is a craft, and the xUnit book is a How-To.  Veteran or neophyte, you’ll learn a new angle that you didn’t know before.  One thing it showed me is that tests are code, which I have to maintain, and showing them love pays almost just as well as it would refactoring production code.

Refactoring tests does many things for you:

  • Eliminates test smells such as brittleness, obscurity or erraticism, and more
  • Provide insight into common contexts and behavioral connections (if several tests have the same setup, then their assertions/observations are related somehow)
  • Reduces nausea

I hate, hate long, complex tests.  If a test has 30 lines of setup, please put that behind a creation method.  A long test just irritates and leaves the developer cross-eyed.  If I don’t have long methods in production code, why would I allow this in our test code?  A test has three parts: Setup, Execute, Verify (and sometimes Teardown).  Following the Context/Specification test pattern already groups these three parts for you, so you can’t get yourself in trouble.

Bottom line, tests are code, love them, and they’ll love you back.

Rule #3 – Don’t settle on one fixture pattern/organizational style

This assumption killed me for a quite a long time.  For some reason or another, I always assumed that I should have a test fixture pattern of One Class, One Fixture.  One problem of course, this makes understanding behavior of the system as a whole, and of its parts, quite difficult.  Systems with hundreds of classes aren’t very easy to see how the pieces fit, and finding usages and following control flow just doesn’t cut it.  But aren’t our tests supposed to be documentation?  Aren’t they supposed to describe how the system works?

If you’re blindly following one fixture per class, that ain’t happening.

One fixture per class leads to some pretty crappy test fixtures.  I’ve seen (and created) test fixtures literally thousands of lines long.  That’s right, thousands.  We would never, ever, ever tolerate that in production code, why is our test code exempt?  Simply because we assumed that there is one pattern to rule them all.  Well, I don’t construct every object with a no-arg constructor, why should I tie both hands behind my back with One fixture pattern?

Or better yet, why just one organizational pattern?  Do unit tests have to match the existing code base file for file, class for class, just with “UnitTests” somewhere in the namespace?  That creates absolute insanity.

Sometimes classes have a lot of behavior that belongs in one place.  Big fixtures can indicate I need to refactor…unless it’s my fixtures that are the problem.  Repositories that provide complex searching are going to have a lot of tests.  But do yourself a favor, look at alternative grouping, such as *gasp* common contexts.  The really cool thing about exploring alternate organization is that it fits perfectly with Rule #2.  Explore other organizational styles and fixture patterns.  Try organizing by user experience contexts, not strictly by classes.

When you start seeing behavior in your system not through your classes’ eyes, but from the expected user experience, you’re on the road to truly valuable, descriptive tests.

Rule #4 – One Setup, Execute and Verify per Test

Another side-effect of blindly using one fixture per class are tests that either:

  • Have a lot of asserts
  • Have a lot of Setup/Execute/Verify in one test

Tests should have one reason to fail.  Asserting twenty different pieces isn’t one reason to fail.  If we’re following Rule #1, that test name is going to get very, very long if we try and describe all of the observations we’re doing.  Have a lot of asserts?  Pick a different fixture pattern.  Test fixture per feature and test fixture per fixture are great for breaking out separate assertions into individual tests.  The better I can describe the observations (from the user experience side), the more the code being created will match what is actually needed.

The multiple execute/verify in a single test is also indicative of assuming one fixture per class.  Here’s one example:

[Test]
public void ValidationShouldWorkCorrectly()
{
    var user = new User();

    user.Username = "   ";
    user.IsValid().ShouldBeFalse();

    user.Username = "34df";
    user.IsValid().ShouldBeFalse();

    user.Username = "oijwoiejf^&*";
    user.IsValid().ShouldBeFalse();
}

Blech.  Not only does the test name suck (which with this many asserts, did we expect any different?), but I have zero insight into the different contexts and valid observations going on here.  What if the second assertion fails a month from now.  How exactly am I to know if the test is wrong?  Or the code is wrong?  Another headache.

Whatever test fixture pattern you go with, you have to stick with one Setup Execute and Verify per test.  If your current fixture pattern doesn’t allow you to adhere to this rule…change patterns.

Keepin’ it clean

Tests can be documentation, but only if you try.  If you’re in the “write and forget” category, your tests will become a deadweight, some even causing negative value.  Thousand-line test fixtures, hundred-line tests, incoherent and illegible tests, bad test names, all of these both contribute waste and detract from the maintainability of your system.  So what, a test failed.  Congratulations.  But can you understand why it failed, what behavior is being specified, and why it was important under what circumstances?  If not, what value are your tests giving you, except for a future headache?

Related Articles:

Post Footer automatically generated by Add Post Footer Plugin for wordpress.

About Jimmy Bogard

I'm a technical architect with Headspring in Austin, TX. I focus on DDD, distributed systems, and any other acronym-centric design/architecture/methodology. I created AutoMapper and am a co-author of the ASP.NET MVC in Action books.
This entry was posted in TDD. Bookmark the permalink. Follow any comments here with the RSS feed for this post.
  • http://realfiction.net Frank Quednau

    I wouldn’t mind seeing as counter-example to your 4 how you would rewrite the test that apparently attempts to assert that certain user names are not valid.

  • http://www.codebucket.org Lee Brandt

    @Frank: This would be my stab at it. If this is what you’re asking.
    [Test]
    public void ShouldValidateUsername()
    {
    var user = new User();
    user.Username = ” “;
    user.IsValid.ShouldBeFalse();
    }

    @bogardj
    Great article. Great points. For me, BDD has been the language shift that pulled me away from my bad TDD habits. I think the reason people get sucked into the fixture/class pattern is it makes them feel safe about code coverage. If they make one fixture per class and one to n tests per method, they’ll have great coverage. This, of course, leads to the problem you explained above of leading you away from the REAL purpose of TDD which is to verify and document the behaviors of the system.

    Always enjoy reading your posts,
    Lee

  • Michael Reuß

    Following rule #1 I’d go for

    [Test]
    public void EmptyUsernameShouldNotBeValid()
    {
    var user = new User();
    user.Username = ” “;
    user.IsValid.ShouldBeFalse();
    }
    etc.

    Right?

    Micha.

  • Tosh

    I’m with Frank: add some more value to the post by including examples of good or excellent tests that address the issues pointed out in the article.

  • http://www.lostechies.com/members/bogardj/default.aspx bogardj

    @Micha

    Yep, that’s much more like it.

    @Frank, @Tosh

    Workin on it, thanks for the heads up!

  • http://www.notesfromatooluser.com Mark Levison

    What about this case:

    [Test]
    public void EmptyUsernameShouldNotBeValid()
    {

    var user = new User();

    user.Username = ” “;
    user.IsValid.ShouldBeFalse();
    assertEquals(” “, user.getName());

    }

    This is a lousy example – but I often find useful to assert two or threes properties of an object, based on calling on a single production method.

    In some cases maybe I want to assert invariants.

  • http://www.lostechies.com/members/bogardj/default.aspx bogardj

    @Mark

    One assert per test. If you have a common context/setup, like this one, pull that into its own setup method, or test fixture class, and have tests for each assert, with good test names describing what the user experience should be.

  • http://www.managed-world.com Jason Olson

    A style I’ve liked lately is the idea of having a folder named after concern of the application. Then the test class/fixture is named after the context (WhenNewStackIsCreated, for instance), and then the tests are essentially the assertions you are making (ItShouldBeEmpty(), ItShouldNotBeIndexable(), etc.).

    Then, the constructor of the test fixture (at least in xUnit.net) sets up the context that every single test will run under.

    I personally find that this way of thinking helps organize my own thoughts on what I’m testing. And it often leads to test fixtures that are smaller (since they are about a specific context), and each test has a minimal number of asserts. Combined with AAA (Arrange/Act/Assert) style testing, it’s a methodology I’ve come to love.

    Just my $0.02.

  • http://blog.typemock.com Gil Zilberfeld

    Jimmy,

    Thanks for the great article.
    Taking Mark’s example and your response: The name of the test is vague enough to include the properties. If I read the test I understand when the user name is invalid. Keeping them together, allows me to conceptually relay that they go together.Granted, that from code maintenance point of view, the separation is needed.

    But from the POV of someone new, or someone visiting the code after 6 months, wouldn’t this test convey more?

    Thanks

    Gil Zilberfeld

  • http://www.lostechies.com/members/bogardj/default.aspx bogardj

    @Gil

    This example wasn’t so bad, but I’m sure you’ve seen ones where you have a bazillion assertions, and even worse, lots of setup-execute-verify in one test. Instead of thinking about it, I tend to just break up tests. It’s easier than deciding for me.

  • Daniel Fernandes

    Validation tests are bit of a pain and I don’t have much of a problem with multiple assertions as long as it’s easy to know what’s tested, there aren’t too many assertions and the context is explcit.
    There may be situations that instead of having to create small unit tests that try testing individual rules, the business representative would submit valid test cases and expected outcomes for the validation. This is similar to Fit but you simply embed those test cases.
    Daniel