Transactions and the check-in dance

At our current project, I’m on the largest individual team I’ve ever worked for.  I’ve been in larger groups working on the same codebase, but not where everyone was actually committing code every day.  One thing that’s been a constant for me in XP/Agile teams is CI and the check-in dance.  The check-in dance is roughly as follows:

  • Update from source control
  • Merge any conflicts
  • Run the local build (NOT just compile)
  • If success, commit
  • Make sure automated build succeeds

The check-in dance is transactional, where it all succeeds or it all fails.  If the automated build fails, we roll back the previous commit.

Transactions can use a variety of concurrency to manage conflicts.  In our case, for whatever reason, we used pessimistic concurrency around our check-in dance transaction.  Pessimistic concurrency means we establish a “lock” on committing, and no one else can integrate during the time we commit.

As those familiar with database transactions can attest, pessimistic concurrency can hinder performance when many people want to change the same data, as additional requests have to be queued up until the transaction is complete.

What this meant for our team was that the end of the day was not too much fun.  Since we have a large team, the chances are pretty high that at least a handful of folks want to check in at the end of the day.  Since each check-in dance can take up to 10 minutes of update, merge, build, commit, we had a lot of waiting at the end of the day.

At the recent ALT.NET conference, Ayende suggested that we move to a looser, optimistic model.  Instead of locking the check-in dance process, we would let anyone commit at any time, and just deal with merge conflicts as they occur.

This works because our source control (SVN) detects conflicts on commit, and will fail a commit if the local version does not match the server version.  We’ve been on the optimistic concurrency mode for the past few days, and lock contention has definitely gone down.

The only thing we really need to worry about is making breaking changes to core models.  Since we no longer put a locked boundary on updates and commits, the probability goes up that someone commits after I update, and introduces a change that breaks me that I’m not aware of.

We do have two things on our side here: team communication and the automated build.  Rolling back a commit is annoying, but only takes about a minute.  We can also give each other the heads up when we’re about to introduce a breaking change.

To be honest, the next time someone announces they have a breaking change, you can be sure that I’m going to try and slip my commit in before theirs, because that’s how I roll.

Related Articles:

Post Footer automatically generated by Add Post Footer Plugin for wordpress.

About Jimmy Bogard

I'm a technical architect with Headspring in Austin, TX. I focus on DDD, distributed systems, and any other acronym-centric design/architecture/methodology. I created AutoMapper and am a co-author of the ASP.NET MVC in Action books.
This entry was posted in Continuous Integration. Bookmark the permalink. Follow any comments here with the RSS feed for this post.
  • http://www.JeremyJarrell.com Jeremy Jarrell

    Not sure what you’re using for your automated builds, but if you’re aren’t married to it you may want to check out TeamCity. It ships with a Visual Studio plugin that basically provides gated commits to your source control.

    When you commit using the visual studio plug in, it intercepts the commit and tries to run the full automated build with the new code, if the build succeeds it commits the code to source control on your behalf, if it fails it rejects the commit, tells you, and leaves the codebase pristine. It integrates fine with subversion.

    we used it on my team when we first introduced subversion and CI (sort of like training wheels for continuous integration) and it worked out great.

    good luck!
    jeremy

  • Jeremy Gray

    You may also want to investigate the reasons behind the end-of-the-day commit rush. It can usually be avoided (or at least minimized) through better work decomposition (reducing both the number and complexity of commits at the end of the day) and strong, frequent reminders to the team that the version control system is not a workstation backup solution.

  • http://www.lostechies.com/members/bogardj/default.aspx bogardj

    @Jeremy Jarrell

    That’s awesome, I’ll check that out. Rollbacks are still annoying.

    @Jeremy Gray

    It’s something we’ve looked at, but we all just seem to get busier on certain days and certain times. We’ll keep an eye on it.

  • Jeremy Wiebe

    We pretty much follow that pattern for check-ins except that we add one more step in between “Run the local build” and “If success, commit”. Between these two we do another Update from source control”. If any new changes come down we return to “Run the local build”.

    This allows us to catch changes that have been checked in since we started the check-in dance and helps us reduce conflicts between commits.

  • http://igorbrejc.net Igor Brejc

    Since you’re using SVN, you could take a look at SVN-Monitor tool (http://www.svnmonitor.com/default.shtml) – it monitors for changes in the repository and then shows a very useful taskbar icon when your working copy is not up to date. It also has some other useful features. And best of all – it’s free.

  • http://www.sweebs.com Kris

    Hello,

    Please add your site at http://www.sweebs.com. Sweebs.com is a place where other people can find you among the best sites on the internet!
    Its just started and we are collecting the best found on the net! We will be delighted to have you in the sweebs listings.

    Regards
    Kris