Continuous Integration Tip of the Day

We wasted quite a bit of time not following this rule today:

If the CI build fails, roll back the previous revision.

The build failed, and from the exception we assumed it was an environmental issue.  Well, a few commits later, and the build kept failing.  We couldn’t tell if it was the original commit or later commits that made the build keep failing.  We tried reverting individual revisions, but no go.

Eventually, the only thing that worked was reverting back to the last successful build.  Thus wasting quite a bit of everyone’s time.  Lots of excuses on why the build should be green when it wasn’t, but a green build is a green build.  A red build is a red build.

This is one of those lessons it seems like we have to learn every few months.  Hopefully, this bad taste will stick in our mouths for longer next time.

About Jimmy Bogard

I'm a technical architect with Headspring in Austin, TX. I focus on DDD, distributed systems, and any other acronym-centric design/architecture/methodology. I created AutoMapper and am a co-author of the ASP.NET MVC in Action books.
This entry was posted in Rant. Bookmark the permalink. Follow any comments here with the RSS feed for this post.
  • Rather then throwing code at the wall and hoping it sticks, why not just profile the build, do a root cause analysis and fix it in one checkin?

    Of course another possibility would be to implement gated checkin like OpenGauntlet and prevent evil changesets from being committed to the branch in the first place.

  • How about a CI server like build-o-matic (which will do a binary search to find the checkin that broke) or Team City (which will do simulataneous builds?)

  • @julian

    I haven’t looked at those yet – though not breaking our CI rules would have fixed the issue. Never check in to a broken build, unless you’re fixing the build.