Where TDD fails for me

TDD is by far the sharpest tool in my belt.  The simplicity of client-driven design combined with the safety net of unit tests allow me to build software at a remarkable constant pace.  At the edges of most of the applications I’ve worked on are areas where TDD has completely fallen flat on its face (for me).  It’s a little disheartening when these areas are always around frameworks that I can’t change.  These are areas where adding unit tests provides coverage, but completely fails in the “tests as documentation” category.  Or, it’s in an area where testing is difficult or impossible, regardless of the available tools at your disposal.

My current TDD failures are around NHibernate and ASP.NET MVC, but they both center around a common theme – I’m deep in the extensibility points of someone else’s framework.  These frameworks offer great extensibility points, but often at the cost of the final result making any sense whatsoever.  Perhaps it’s how I practice TDD, as I like to start on the outermost visible behavior, and let client-driven code direct the design underneath.  Often the outermost behavior leaves little point to TDD internal implementation details.  Other times, the outermost visible behavior is voodoo to get set up, and verification impossible for the next developer to understand.

Example 1 – Extensibility through inheritance

In NHibernate, mapping from types to the database at the property level is done through a set of IType implementations.  These mappings provide the logic to map from System.Decimal to something out of an IDataReader.  Often, we need to provide custom mapping types to do things like map values from enumerations, custom Value Object types, or dealing with legacy databases.  NHibernate is fantastic in that regard, as there has not been a problem I’ve thrown at it that I haven’t been able to solve with an obvious extensibility point.  Side note – this is common with good OSS frameworks – feedback from the community funnels back in to further refine the design.

The one issue with these extensibility points is that it is completely non-obvious to unit test one of these implementations.  Here’s one example of an implementation:

public class DummyCustomType : PrimitiveType
    public DummyCustomType()
        : base(new SqlType(DbType.String))

    public override object Get(IDataReader rs, int index)
        var o = rs[index];
        var value = o.ToString();
        return value;

    public override object Get(IDataReader rs, string name)
        int ordinal = rs.GetOrdinal(name);
        return Get(rs, ordinal);

    /* etc etc etc */

Blah blah blah.  I can TDD individual pieces, but notice that I’m inheriting from PrimitiveType – an NHibernate extensibility point.  But are unit tests valuable here’?  I think not, as the true test as to whether this custom type works is in the context of where it is used – NHibernate.  Instead of testing the individual class, I’ll define the behavior I want against NHibernate, usually by loading up entities that match up to the scenarios I’m interested in.

So it’s very rare that I TDD an extensibility point driven through inheritance.  The voodoo going on underneath in the base type would put too much knowledge into the test of an implementation that I can’t often see without Reflector, so where is the value?  I don’t really see any.  Some of the pieces I have to implement I don’t even know why I need, so I leave “throw new NotImplementedException” in, and wait for my application to blow up and tell me why I need that piece.  It’s another reason I see 100% coverage as a goal that has to be balanced against other concerns.

In cases of extensibility through inheritance, it’s only the macro behavior I care about.  I could care less how the specific extensibility is used – all I care about is that the eventual result of my cog in the giant machine works as specified.

Example 2 – Thorny observations

How do I TDD a custom WebForms control?  Or an ASP.NET MVC ActionFilterAttribute?  I know I can create the crazy dependencies required – such as the ActionExecutingContext – but what exactly is this telling me?  The final result, or verification, is the action filter plugged in to the complete pipeline.  Otherwise, I’m only verifying that it’s doing exactly what I told it to do – not what I need it to do.  In domain model specifications, I describe how I want the model to behave under specific conditions.  I could TDD this:

public class AdminRoleFilterAttribute : ActionFilterAttribute
    public override void OnActionExecuting(ActionExecutingContext filterContext)
        if (filterContext.HttpContext.User.IsInRole("Administrator"))
            filterContext.Result = new RedirectResult("/unauthorized");

But honestly, only after a spike.  And if I did spike this, how would I get back here?  TDD would lead to a big disconnect between what I was verifying (that some Result is set under certain conditions) and what I actually want to happen.  TDD’ing this part requires knowledge of how the final result is used, leaving the eventual maintainer of the code to guess that this result hopefully has the desired effect.

What I’d really like to verify is that a user gets redirected when they are not an administrator.  However, I have to poke around a lot of framework pieces to get that.  I understand those actions are part of using a framework, but it’s up to the next developer to make the jump that hopefully, I did my homework and that setting that Result to that value will have the desired end behavior.  End behavior I can’t test easily, as it’s really a browser-level interaction test.

I usually still TDD these scenarios, but only after a spike.  And all I’m really doing is TDD’ing back to the point where I knew things were working in my prototype.  At that point, I still do manual, one-time-only verifications that what I did actually created the behavior I wanted.  Because the unit tests didn’t really do that, they just verified I’m using the framework in the way I specified.


In the end, I’ve noticed my tests around framework interactions have the least amount of value.  They definitely have value, but the act of TDD is severely stunted as I’m playing by someone else’s rules.  I can verify my little cog behaves exactly how I specify in isolation, but that’s pointless if it fails when inserted back into the big machine.  Sometimes I can verify the output of the big machine, as in NHibernate, and sometimes it’s very difficult, or slow, as in ASP.NET MVC’s example (and other web frameworks).

Not only do these tests have less value, but they tend to be far more brittle than tests against POCOs, or even services where I have the Dependency Inversion Principle in play.  I create some custom NHibernate type, hooray!  But what about the dozen other scenarios for legacy data I don’t know about?  Or, when the only true verification is a manual or slow verification of an ASP.NET MVC implementation detail?

The unit tests themselves still provide value, but even tools like TypeMock wouldn’t solve the issue.  The issue isn’t that I can’t mock the framework, it’s that the framework in action is the only true verification of the intended behavior.  I’m not getting nearly as much, if any, client-driven design benefits when TDD’ing framework extensibility points.

About Jimmy Bogard

I'm a technical architect with Headspring in Austin, TX. I focus on DDD, distributed systems, and any other acronym-centric design/architecture/methodology. I created AutoMapper and am a co-author of the ASP.NET MVC in Action books.
This entry was posted in TDD. Bookmark the permalink. Follow any comments here with the RSS feed for this post.
  • Framework designers take note: Favor composition over inheritance. Prefer thin sliced services loaded through a container versus deep, painful hierarchies which must be maneuvered around.

    Design your framework using TDD and with a container in mind, and you will have generally end up with a well-designed framework exhibiting the best feature of all: True Extensibility.

  • Good post. I agree, framework nitty-gritty and GUI are often the pain points for me.

    RE: Chad – How would composition help him in the usertype case? Even if it was made with composition, the framework would still be pulling all the strings and calling his methods in some way (probably chatty too). It might be somewhat easier, but I think the main problem prevails.

    One of my favorite places to TDD in regards to frameworks is my NHibernate mappings – those cascades always get me. But I guess that’s more “Sanity-Test”-Driven-Design than TDD :-)

  • @Rasmus: I’ll toss it back to you, why does PrimitiveType need to be a class instead of an Interface?

    I checked the source and it basically does nothing. It derives from ImmutableType which in turn turn derives from NullableType, then AbstractType which is the top of the chain. These types of deep inheritance trees can be very painful to work with.

    Perhaps this is the only/best way to do this, in which case we have to deal with it, but I’m guessing that a decorator approach or chain of responsibility would’ve been better applied here than crushing inheritance.

  • @Rasmus

    I definitely TDD my mappings – but it’s the framework extensions that I only TDD through mapping tests.

  • although i wouldn’t call it a unit test, i would still TDD the primitive type inheritence. i would set up an integration test that uses NHibernate, to prove that I need a Primitive Type implementation by showing that NHibernate does not do what I want natively, then i would set up the primitive type implementation to make the test pass.

  • I believe such things should be tested through integration tests, e.q. in a real environment or close to it. So, all stuff that is not related directly to the 3-rd party component should be extracted to another classes.

    But it is great if 3-rd party developer provide some helpful classes for testing with their components (like Monorail’s BaseControllerTest)

  • @Derick

    Yeah, I still TDD those, but they don’t create unit tests. I keep forgetting that TDD != creation of unit tests.


    Even with the 3rd party classes – they sometimes don’t make sense outside the context of the framework. I do love the test helpers – it shows that the framework devs know the testing pains