Speeding up a local build

For the past few years, I’ve been fairly strict about following a rigorous Continuous Integration process.  That means we run a local build (NOT just a solution compile) before we check in.  However, our local build was getting rather long.  For us on a large team, five minutes is too long.  It’s just long enough that people start to get bored.  It’s just long enough that we attached a build chime to the end so that we had audible cues that our build was done (no one likes to stare at a console screen).

Once things got bad enough, we looked at ways of speeding up our local builds.  There are quite a few sources on the interwebs, and here are a few that we found greatly sped things up.

1) Tame the solution structure

Using multiple projects to enforce dependencies is a noble cause, but quite inefficient.  I’ve seen teams, with the desire to enforce a certain architecture, explode their project count from a handful to dozens.  All in the name of trying to make sure I didn’t call the data access layer from the UI layer.  But build time is largely affected by the number of projects far, far more than the number of files.  What’s more insidious about large numbers of projects is that it introduces dozens and dozens of delays in day-to-day development.  I can deal with a long local build time, but if it takes a minute or more to simply run my application locally, my noble cause has introduced far more pain than it has solved.

The quickest way to solve this naturally is to collapse your solution structure down.  Try to focus your projects more on deployment dependencies and targets rather than projects or files.  You can still structure your project using *gasp* folders however you like.  But with all that file copying and compiling over and over again, large numbers of projects.  Concerned about relationships and dependencies?  Try NDepend.

2) Use file-based dependencies

Project A depends on Project B.  Project B depends on Project C.  I make a change in Project C, but now I have to copy files to two places, Project A and Project B.  As your dependencies deepen, Visual Studio can’t know when projects need to be re-compiled.  Instead, you can change your project configuration to output all assemblies to one common folder.  Control build order not through project references, but through compilation order, which is easily configurable through the solution options of Project Dependencies and Project Build Order.

Using file-based dependencies eliminates all of the needless file copying going on, and lets Visual Studio be smarter about when items need to be recompiled.  Putting this in practice in your local build will also speed things along.

3) Batch targets in MSBuild and NAnt calls

There is a significant performance hit when starting up MSBuild, and to some extent, NAnt.  But both of these allow for passing in multiple build targets in one call.  Instead of:

msbuild.exe /t:Clean
msbuild.exe /t:Build

Do this:

msbuild.exe /t:Clean /t:Build

We get the same effect, two targets run, but now we don’t have to start MSBuild twice, have MSBuild parse the XML build files twice, and so on.

4) Use in-memory databases for testing

On our local builds, we run a suite of integration tests against a local database, which are orders of magnitude slower than unit tests.  If you’re using an ORM like NHibernate, switching databases is as easy as changing a configuration file.  This is something we’re working towards on our team (and isn’t there yet), but I’ve heard quite a few great things about tremendous speed improvements, 10X in some cases.  Check out Justin Etheredge’s post on the subject for a detailed writeup:

http://www.codethinked.com/post/2008/10/19/NHibernate-20-SQLite-and-In-Memory-Databases.aspx

The interesting part I hear is “but I’m not really testing against my database anymore!”  The only problems I’ve run into when switching between Oracle and SQL Server were Oracles absolutely asinine naming restrictions.  And guess what, we’ve got a test for that too!

On our server, we run against the real databases we integrate with.  But locally, since we don’t take advantage of database-specific grammar, functions or things of that nature, it makes sense to make this optimization.

5) Asynchronous build tasks

Our build steps go something like this:

  • Clean output directories
  • Reset local test database
  • Compile solution
  • Run aspnet_compiler to compile views
  • Run unit tests
  • Run integration tests

Some tasks obviously need to be run before others.  I have to compile before I test.  But some tasks don’t have dependencies on ones that come before it.  Because we’re all running on multi-core machines, we can take advantage of Jay Flower’s asyncexec NAnt task.  This task is identical to the “exec” NAnt task, with one difference – it doesn’t wait for the process to complete.  At the end of our build, we’ll have another task, that waits for all asynchronous tasks to complete.

In our case, we made the aspnet_compiler task run asynchronously, right after our compilation.  Because unit tests and integrations can take a while, it turns out that doing that resulted in an immediate 90 second drop in the time in our build, almost 20% for us at the time.  Quite significant!

You can’t make everything asynchronous, as disk I/O time and the number of cores (as well as dependencies between tasks).  But a couple of well-placed asynchronous tasks in place of “exec” can have quite a large impact.

Wrapping it up

When we started our build improvements, I was at around 390 seconds locally, completely unacceptable.  After our build improvements (and we still haven’t put in in-memory databases), we’re down to 180 seconds, that’s over a 50% drop.  Even more important is that the build time is now down to a time where it doesn’t distract from my work, and I don’t have to worry about the time to integrate my changes.

These aren’t the only ways to speed up builds, nor speed up integrations.  Other ideas include using a staging server, developer branches, gated commits, and others.  All great ideas in improving our efficiency in delivering value.

Related Articles:

Post Footer automatically generated by Add Post Footer Plugin for wordpress.

About Jimmy Bogard

I'm a technical architect with Headspring in Austin, TX. I focus on DDD, distributed systems, and any other acronym-centric design/architecture/methodology. I created AutoMapper and am a co-author of the ASP.NET MVC in Action books.
This entry was posted in Continuous Improvement, Continuous Integration. Bookmark the permalink. Follow any comments here with the RSS feed for this post.
  • http://www.blogcoward.com jdn

    “When we started our build improvements, I was at around 390 seconds locally, completely unacceptable.”

    While I understand what you mean, this is kinda funny for those of us who are in situations where builds take 30+ minutes. On good days.

    But thanks for this. It’s a good list of improvements.

  • Jeremy Gray

    Regarding #1, there seems to have been a growing hatred of breaking things up into multiple assembly projects in recent years. At some point along the way I became a convert. Now, however, I don’t buy the black and white statements people keep making any more. Frankly, they are doing a serious disservice to the community. The reason is this:

    I finally had a chance to work on a project with enough test coverage that breaking things up into multiple assemblies made the moment-to-moment development process go _faster_, not slower, for the simple reason that once enough unit test coverage is in place you’ll find yourself searching for ways to speed the _combined_ build/test process. Compilation time isn’t everything. On a project with good coverage it may not even be noteworthy. More critically, the solution structure may affect the now-dominant test time far more than the now-unimportant build time.

    Here’s what I’d rather see being suggested: Just as breaking _everything_ up was an unjustified blanket statement, so is sticking everything together. Break up _your_ solution to the degree and in the manner that makes _your_ regular build/test cycle run quickest given an average code change. For some solutions, this will result in you having one and only one project. For others, you may end up with dozens of projects supported (for example) by shell scripts that conditionally execute only what needs to be tested based on only what needed to be built. Revisit your choice from time to time, as your project’s needs will likely change over time.

  • http://www.lostechies.com/members/bogardj/default.aspx bogardj

    @jdn

    Don’t worry, I’ve been there too. What we wound up doing is figuring out larger dependencies, splitting things into multiple solutions, and only running the build on solutions we changed. It put more trust in the CI build, for sure, but helped us see our local changes much quicker.

    @Jeremy

    You could also look into multiple solutions and local builds, for each solution. It’s all about understanding what changed and what was affected.

  • Bruno Martínez

    I’m intrigued about the point on memory databases. Visual Studio 10 uses SQL Server Compact Edition to store intellisense, so it can’t be that slow.

  • http://erichauser.net/ Eric Hauser

    Jimmy,

    You may be doing this, but msbuild /m is your friend. Also, I recently upgraded my workstation which was a Core 2 Duo with 4 GB of RAM — not a bad box — to a low end Core i7 with 6GB of RAM. A previous large .NET solution that took ~35 secs to compile now compiles in 10 secs flat. Hardware is cheap relative to developer time.

    Also, are you running your unit tests in parallel or single threaded? For non-sequential tests, it would be interesting to see what parallelizing your tests did.

  • http://weblogs.asp.net/sfeldman Sean Feldman

    We used to have SQLite in memory testing for the DB, but we stopped it. SQLite was passing tests locally and failing them against the real database (Foreign keys, etc). Our solution was to use local DB per developer. This is not as bad as going to some server. As long as there’s a 100% compliance between in-memory and production database, you can do it. Otherwise – no.

  • http://sharpbites.blogspot.com alberto

    Nice article.

    It would be great if you could share your build scripts.

  • Jeremy Gray

    @Jimmy – Using multiple solutions is something I’ve done on many larger projects. To be able to do that you still have to go against the frothing masses and break things up. That the break-up usually is best cut along the very same lines that define tiers and partitions, published abstractions and private implementations, is very telling IMHO.

    Don’t get me wrong, I’m not advocating cutting along EVERY single line. People need to determine their own priorities and then cut where and only where appropriate based on those priorities, whether they be build time, test time, published abstraction version stability, tiering, partitioning, IDE performance when working on subsets of projects, all sorts of options.

    The blanket statements just need to stop because they aren’t helping anyone and will actively harm many. The first thing to happen on a blindly-minimized-project-count solution with a large enough number of tests in it is that the devs will stop running the tests, and we all know where that leads. And, no, the fact that the build process will still run them is inadmissible because, while true, it kinda misses the point. ;)

  • http://www.lostechies.com/members/bogardj/default.aspx bogardj

    @Jeremy

    Here’s my case study – a recent gig we went from 72 to 12 projects. Many of the projects were exactly one file, even though there were only 4 deployed applications. Pointless, in that case.

    If a team has problems with understanding layers and responsibilities, and you have to force a project structure to make that understanding, then I’d say you have other bigger problems. I’d rather the focus be on deployments and dependencies between deployments rather than trying to enforce layers, or “i don’t want to reference System.Data from my Domain project”. That’s hand-holding.

  • http://www.lostechies.com/members/bogardj/default.aspx bogardj

    @Sean

    Thanks! We’ll keep an eye on it. On the other hand, I don’t see a situation where dev’s shouldn’t have their own DB. Sharing DB’s is messy.

  • http://www.lostechies.com/members/bogardj/default.aspx bogardj

    @Eric

    I’m on a laptop supplied by our client – not too much I can do in that side. I forgot about the /m switch, thanks!

  • Padda

    Hi,

    Excellent tips.

    On point 2. Is there any way to control build order other than via project dependencies? In a solution with lots of projects, sometimes a small change means all dependent projects are recompiled. This increases local build time significantly. So, I copy all projects output to a ReferencedAssemblies folder, and reference dependencies from there (with CopyLocal = false). But when I build a solution, the dependent projects still build, even if no changes have been made. I’d like to decide the project build order without using dependency ordering. Is this possible? I’m using VStudio 2008.

    Thank you.

  • Charlie Barker

    Giving your developers SSD drives will also help

  • XioPod.Net

    Very good advice! We do most the same things in our build system. In addition, we found the following also useful:

    - Keep an eye on the build output. if it lists warnings either fix them or disable the warning number.
    - Setting warning level to 0 can save a ton of time but can hide simple fixes from dumb/lazy developers. we had a project spitting out 14K+ warnings, changing the level to 0 reduced the build time by more then 60% (8 mins down to 3)
    - If you don’t use the XML docs don’t generate them
    - turning off vshost processes can help reduce file locks
    - RAM and SSD’s are getting cheap and can really help. we actually run two stripped SSD’s and 24GB+ ram in all our build servers.