Defining unit tests

I don’t know where I got off the tracks on this one, but I’m really liking Michael Feathers’ definition of a unit test:

A test is not a unit test if:

  • It talks to the database
  • It communicates across the network
  • It touches the file system
  • It can’t run at the same time as any of your other unit tests
  • You have to do special things to your environment (such as editing config files) to run it.

    Tests that do these things aren’t bad. Often they are worth writing, and they can be written in a unit test harness. However, it is important to be able to separate them from true unit tests so that we can keep a set of tests that we can run fast whenever we make our changes.

  • It doesn’t say “must only test one class” or “by jove, every test method name should include the name of the method being exercised”.  Going back to the original Test-Driven Development book, there are no instructions to test only one class at a time and that every dependency must be some kind of test double for the test to be a true “unit test”.

    The xUnit Test Patterns book does go in to building top-down tests like these, but that book is mostly a collection of patterns rather than prescriptive guidance.

    Looking at dependency injection, the component’s constructor describes what the component needs for it to function.  Those are implementation details.  When I’m testing that component, is that really something I need to be concerned about?  In some cases, yes, when I’m concerned about interactions and that behavior is interesting to me.  But other times, the level of isolation through a strict mock-everything approach leaks those implementation details of not only what the dependencies are but how those dependencies are used leads to a coupling of test and implementation.

    Six years of doing TDD and I think I might be getting it.

    Related Articles:

    Post Footer automatically generated by Add Post Footer Plugin for wordpress.

    About Jimmy Bogard

    I'm a technical architect with Headspring in Austin, TX. I focus on DDD, distributed systems, and any other acronym-centric design/architecture/methodology. I created AutoMapper and am a co-author of the ASP.NET MVC in Action books.
    This entry was posted in TDD. Bookmark the permalink. Follow any comments here with the RSS feed for this post.
    • http://www.blogcoward.com jdn

      “Six years of doing TDD and I think I might be getting it.”

      I understand the point you are trying to make, but if you think about it, that’s kind of scary.

      I mean, you are smart. What about the rest of us trying to do TDD?

      That’s why BDD, Context Specification, is so much better. But I digress.

    • Josh

      I disagree with that statement.Since I see a large value in testing the whole system instead of the pieces that don’t interact with other systems.

      1. A database contains code just like any other application. Why should it be shunned from testing?
      2. Having socket/network failures in a unit test is a good thing. That means you have to change your code to better handle a situation that rarely occurs. Your unit test could still fail, but the error is “Network unavailable”, vs, “Object reference not found”. I had a bug in a c# 3270 emulator I couldn’t figure out, after two weeks of automated testing I finally found the issue and fixed it. If I took the advice above that bug would of made it to production.
      3. Isn’t it ideal to run unit test on another machine, so doing so introduces a great deal of latency for any build server to run no matter how fast your unit tests run. Its great to be able to run them locally but you still have to check on another machine to prevent the “works on my machine” syndrome.
      4. If you code is running that slow for database/network communications, is there a chance that maybe its slow code is the problem. TDD gives you the ability to unit test a specific portion and profile it which could improve the system performance and your unit test completion time.

      Just my 2 cents.

    • Roco

      “I understand the point you are trying to make, but if you think about it, that’s kind of scary.”

      My sentiments exactly. I had to sit there and let that sink in for a few minutes. Jimmy comes off as a very sharp developer, so if something like testing, which should be so fundamental in our craft, takes that long to truly “get”, then there is some seriously wrong with our industry. Or more specially, the guidance around how we should test (which seems to change every damn day) is seriously flawed.

    • Pete D.

      @Josh

      So we should let our unit tests fail because calling X webservice to get some data which we then process is down or broken? How many developers will continue to run the tests then?

      “A database contains code just like any other application. Why should it be shunned from testing?”

      Not sure what you mean by code, maybe DDL, stored procs, views etc. But hopefully you don’t mean logic.

    • Steve

      @Josh,

      1. That is integration testing, not Unit Testing. The definition quoted even said that testing the database is valid, but it’s just don’t call it “Unit Testing”

      2. If having a socket failure is rare, how can you guarantee it’ll happen when you are running your Unit Tests, just get lucky? Better to simulate it failing to make sure that your code fails gracefully than hope that a Unit Test catches it.

      3. I don’t understand this point at all, build servers run Unit Tests on another machine all the time. That is one of the main benefits of Unit Tests, I can pull down the code on any machine, and (in Visual Studio anyways) go to Tests -> Run -> All Tests in this Solution. No fiddling with config files, worrying about databases or anything else. Unit Tests just run.

      4. If running all “Unit” tests that hit the database causes slowness, it’s generally not due to the code, but due to the fact that you are constantly setting up and tearing down data, that and your database is always contending for the same data.

      I’m all for a nightly integration test that hits the DB and pretty much runs the full gamut of code, but that isn’t Unit Testing. Unit Tests need to be fast and quick, they need to test that the code that I just wrote does what it says on the tin. Anyone needs to be able to run it without knowing anything about the code, and most importantly a CI server needs to be able to run it dozens, if not hundreds of times a day (for a large project), and that just isn’t tenable if you are constantly setting up and tearing down databases.

      Where I work now, we have 70 developers working on a large project, our Test suite ran over 200 times yesterday without any problems. If a test breaks, the developer who broke it knows within five minutes and can fix the problem. We also run a daily, full fledge set of integration test runs to confirm that everything plays nicely together from soup to nuts, but if we ran that on every check in, our build server would probably explode.

    • http://www.lostechies.com/members/bogardj/default.aspx bogardj

      @jdn

      It was a bit of a joke. I also don’t think I’ve truly mastered OOP, TDD, architecture etc etc. and that I still have a lot to learn. It’s meant to underscore that I hit things that fundamentally shift my outlook. And I fully expect to change my mind in a year.

      @Josh

      I whole-heartedly value full-system tests. In fact, I don’t see nearly as much value in unit tests if there isn’t a full-system test to back it all up.

    • http://www.blogcoward.com jdn

      @Jimmy

      I understand that, but it is hard to swing a dead cat and not come across similar posts with sentiments like “I’ve been doing TDD for three years, and most of the time, I’ve been doing it wrong.”

      I think there’s a reason for that.

    • http://dorkasaurusrex.blogspot.com Chris B

      I’m not sure I can completely agree with that definition of unit test either. I hope the people working on NHibernate write tests that talk to databases, and I would consider those to be “true” unit tests. You can easily come up with analogs for the file system and the network as well. I also find it interesting that a unit test is defined in terms of what it is not. I can’t imagine how difficult it would be to define a circle this way…..

      I think the database, file system, and network could be better classified as external systems. Web services and cooperative processes seem to be notable omissions that fall into this category. External systems are one of the areas where we usually want to introduce seams into the application so that we do not depend on the external system in order to verify ours. We introduce seams into applications so that we can achieve new levels of abstraction and independence.

      My best shot at defining a unit test would be: A unit test is a test that verifies one unit of functionality at a single level of abstraction, but may specify behavior at a seam. Once a test crosses a seam, it becomes an integration test.

      A unit of functionality could be a method, class, user story, etc…. The SUT may use objects at multiple levels of abstraction to do its job, but the test is only aware of one of them. Mocks/stubs are allowed as stand-ins only at the seam boundary. The test’s abstraction level will probably be the same abstraction level at which the SUT exists. I’m not sure if that is necessary though.

      I think this also speaks to the problem you mentioned in your previous post. The tests were aware of both the desired results (high abstraction level) and the implementation details (low abstraction level) of the SUT. If the test only used mocks/stubs at seams, refactoring would probably be easier, as long as the APIs to the seams are less likely to change.

    • Josh

      @Pete D.

      If the web service is down or broken, that at least alerts someone to the issue. 9 out of 10 times I bet the exception is something generic and has no meaning if someone looked at it. Which means you write the exception ABCCompanyWebServiceDownAfter5Retries, vs the Remote Host Down after you get tired of looking up the error or decide to throw an “inconclusive”. I even bet that there is 0 retry logic in 99% of the code out there. Also if you have a single if statement in a stored proc, its code. So a simple “update if exists, insert if it doesn’t” stored proc now contains logic. So it has to be tested to make sure everything is working as expected.

    • http://blog.belfryimages.com.au Ben Scott

      I don’t think the point of @mfeathers post (or @bogardj’s analysis) is that we shouldn’t write the integration (as in data layer, network, fs) tests at all. The point was that these (slow) integration tests should be separated into another test harness, leaving the faster unit tests. Getting fussed about calling them ‘integration tests’ vs ‘unit tests’ is just semantics.

      This doesn’t require duplication of tests or mocking data access in unit tests.

      Say we had two methods to test. “CalculateCartCheckoutPrice” is a complex calculation using a number of domain objects, and doesn’t do any data access. “SaveCartContentsToDatabase” mainly does data access (saving the domain objects to a database) and not much logic.

      “CalculateCartCheckoutPrice” tests would be considered unit tests, but “SaveCartContentsToDatabase” tests would be integration tests. The integration tests are then mainly testing the design of the database (tables, stored procedures) and data interface. The unit tests are separately testing the logic, and are where you want faster iterations of tests.

    • http://www.lostechies.com/members/bogardj/default.aspx bogardj

      @Chris

      That might be my favorite definition – things like a repository crosses a stream, which I’d want to stub most of the time in a unit test. But peer objects, objects of the same abstraction, I’d want the real-deals in play.

    • aNt

      I was introduced to “The Way Of Testivus” ( http://www.agitar.com/downloads/TheWayOfTestivus.pdf ) as part of last year’s GTAC.

      Write the test that needs to be written.

    • Josh1

      That definition given by Michael Feathers is the same one he uses in his book Working Effectively with Legacy Code (highly recommended by myself). In it he also states this about unit tests:

      1. They run fast.
      2. They help localise problems.

      He goes on to say that a unit test that takes 1/10 sec to run is a SLOW unit test. The slower the tests, the less likely is it that a developer is going to keep running them. Which defeats the purpose.

      Reducing the “integration” scope of a unit test from several classes to a few or a single class will speed up these tests. You can also see that DB or network access will totally kill the speed (and reduce localisation of problems).

      I think his point about speed is very crucial.

    • http://dorkasaurusrex.blogspot.com Chris B

      After pondering on it overnight, I realized that my definition of a unit test may cause the seams of the application to become visible at every level of abstraction above the seam. It seems that it would be ideal if seams were only visible at the next higher level of abstraction.

      For example, suppose you have Widgets that are read/written through an IWidgetRepository, and you also have a WidgetProcessor that does some sort of magical processing on the widgets. Suppose your SUT exists at the same abstraction level as the WidgetProcessor and that it uses the WidgetProcessor. How would the SUT know what IWidgetRepository implementation to use to instantiate the WidgetProcessor?

      If the SUT accepts an IWidgetRepository as a dependency, the seam has bubbled up to another level of abstraction. Now all objects that use the SUT must somehow resolve an IWidgetRepository implementation. The SUT’s dependence on IWidgetRepository is only indirect, it only serves to allow the instantiation of the WidgetProcessor.

      On the other hand, if the SUT accepts an IWidgetProcessor as a dependency, an artificial seam has been created in the application. This seam adds a small degree of abstraction by hiding the WidgetProcessor’s dependencies, and some independence by letting you mock/stub the widget processor, but it isn’t much. The IWidgetProcessor will probably only have one implementation, and exists only for the purposes of registering WidgetProcessor in the DI container. I think this is a contributing factor to “mocks everywhere” and “interfaces for everything”, and can result in the types of tests which you mentioned previously.

      There are also other solutions, like Service Locator and Factory patterns, but these are very similar to accepting the dependency as an argument. How is the factory or locator resolved? Do I have to write one for every environment (UT vs production)?

      I’m not really sure how to modify the definition so as to prevent this, or if there is a definition that would do so. I do think that this is one of the biggest problems and areas of confusion when it comes to defining a UT strategy. Maybe someday I’ll have some sort of epiphany on how to solve it.

    • http://nguyducthuan.com Thuan

      Regarding Michael Feathers’ definition, a colleague of mine said that it is too strict. He proposed a slightly change to it: A test is not a unit test if the SUT:
      – Talks to…
      – Communicates…

      In other words, it will be acceptable, for example, to read a file to prepare data for a test case.
      What do you think?

    • http://www.ntcoding.co.uk ntcoding

      Jimmy,

      I’m loving your exploration of testing and I am on the journey with you. But how about chucking the odd example in here and there?

      Judging by your comments, people seem to be drawing their own conclusions. I myself wonder how your theory translates to practical application.

      And finally thanks for writing these intriguing posts.