Estimation in consulting

Some of our big early agile projects in consulting gave us quite a big perspective in what works well in estimation techniques in our consulting business, and what doesn’t. We already ditched the group estimation techniques of planning poker, finding the time to be largely a waste and its only benefits (group design, etc.) to be better served with other, more targeted activities. But what we kept for a while was exposing our estimations of sizing to the customer.

Regardless if we did sprints or not, we would show the customer the estimated cost of stories:

  • Story 438 – 13 points
  • Story 34 – 8 points
  • Story 424 – 5 points
  • Story 453 – 13 points

This led to quite a bit of back and forth with the customer with the question of “what the heck is a point?!?!?” The customer wants to know how much a feature is likely to cost. But a point is a measure of complexity, not effort. Talking about estimated effort is a completely different conversation.

Our customers don’t write checks in points. They write checks in dollars. For a customer to make an educated decision on priority etc. they need to have an understanding on how much a feature is likely to cost them, in a term that actually makes sense to them.

Transitioning to hours

Nearly all of our contracts these days are structured as time & materials. We bill hourly, and work in mutual risk plans in our contracts with things like escape clauses for the client and so on. We like to compare our inventory to the fish market. In a fish market, any fish not sold at the end of the day is trashed, waste. Any hour we don’t sell to a client is (potential) waste. Ultimately, our inventory is hours, and that’s what we sell to clients.

Coming back to estimation, cost in points caused nothing but confusion and frustration with client after client. So, we ditched estimating cost in points. It’s still extremely important to set cost expectations with a client. No one likes to take their car in for a brake check and see a bill on the other side for $3000. As long as we’re upfront about uncertainty and risk, it’s critical to set proper expectations on how much a feature will cost, in actual dollars. And as long as features/stories follow the INVEST guidelines, we can reduce the risk of missing the target.

By including all activities in our story flow boards, we include all activities in getting a story to production when estimating cost, such as:

  • Feature Analysis
  • System Analysis (for existing systems)
  • Modeling
  • Design
  • Development
  • Testing
  • Stabilization
  • Deployment

Hands on the keyboard typing are merely one phase of a story’s lifecycle, but all the other phases include time, effort, manpower and ultimately money. These phases add value and are part of delivering a story, so we include all these in our estimates. To keep with the spirit of our Fibonacci story sizes, we followed powers of 2 in our cost estimates:

  • 1 hour
  • 2 hours
  • 4 hours
  • 1 day
  • 2 days
  • 3 days

This would be in each of the areas of effort for delivering a story:

Feature Analysis 2 hours
System Analysis 8 hours
Modeling 4 hours
Design 4 hours
Development 2 days
Testing 2 hours
Deployment 1 hour
Total 37 hours

This total cost would be labeled something like “optimistic cost estimate” or “unrealistic cost estimate”, to convey that there is still uncertainty. Then, depending on the client, system, environment and past history, we’d have an uncertainty percentage, ranging from something like 10% to 50%. This is all about factoring variance of the final costs of stories. Some projects have wildly varying costs from the original estimate, some don’t. We’d then include this in our cost estimate of a story:

Unrealistic Estimate 37 hours
Risk factor 25%
Realistic Estimate 46.25 hours

We can then track to see based on each story how close we are to the varying estimates, and make adjustments to our risk factors as necessary. The tech lead/principal (who is also a developer) provides these estimates to the customer, and it’s usually a starting point on discussions of priority, complexity and so on.

All about communication

In consulting, we sell hours of time. For a customer to make an informed decision on choosing stories, managing complexity and budget, they have to have a realistic expectation of cost. While story points are good at estimating complexity, they aren’t good at cost. Instead, we adapted our story estimation techniques to our delivery models to come up with a solution that our customers actually undertand. They may not like our estimates, but we’ve at least set the proper expectations against a number that actually makes sense to someone signing a check.

Related Articles:

Post Footer automatically generated by Add Post Footer Plugin for wordpress.

About Jimmy Bogard

I'm a technical architect with Headspring in Austin, TX. I focus on DDD, distributed systems, and any other acronym-centric design/architecture/methodology. I created AutoMapper and am a co-author of the ASP.NET MVC in Action books.
This entry was posted in Agile. Bookmark the permalink. Follow any comments here with the RSS feed for this post.
  • Carlos Ribas

     Jimmy, how do mitigate the risk of regressing to waterfall when you start talking in terms like these?  What do you do when the client demands increasing levels of certainty once you start talking in terms of hours (which they immediately divide by 8 and then by number of staff to calculate a deadline)?

    The path to perfect knowledge of cost is paved with upfront planning, no?

    • Anonymous

      Scheduling is a whole different ball of wax. We have to make sure that we don’t present costs in terms of a Gantt chart, or if we ever see one, we reset expectations. Cycle time is a different measure than cost. “How much does this cost” is a different question than “when will this be delivered”.

      Otherwise, it’s still the same story flow board, we’re just estimating the number of hours spent in each phase. We manage the size of the stories, but it’s really not much different than what we did with points. 

      • http://twitter.com/kristofclaes Kristof Claes

        How can you give your clients a price without doing a rather detailed upfront waterfallish analysis?

        I guess the requirements should be very detailed and complete to avoid miscommunication (“Oh, you meant it like _that_?!”), but how detailed should the analysis be in order to provide a reasonably accurate price?

        I can really see the benefit of an agile approach, but giving your clients a price without shooting yourself in the foot looks really difficult without getting caught by the waterfall.

        • Anonymous

          Just like the other items we estimate in development, we try to get a reasonable idea of cost on all the phases. It’s not an easy process to be sure, but we still want the client to have the ability to change direction. That still happens with a cost estimate, but now they have some better information about making a decision.

          These are cost estimates too, not mini contracts. We’re just trying to set cost expectations.

          • http://twitter.com/kristofclaes Kristof Claes

            Does that mean that it is possible that you give your client a cost estimate of let’s say $1,000 but that you actually end up charging them $1,350?

            When we offer a price to our customers, they want to pay that price, nothing more.

          • Anonymous

            Yes, but it’s not a surprise. If the client is surprised by a bill, AFTER work has been done, it’s the fault of the person doing the work not raising the flag sooner. 

  • http://mutedsolutions.com Derick Bailey

    i think the big assumption or un-stated state of your team / client relationship, is trust and open communication. without these, things fall apart quickly, no matter how you estimate. some of the questions that others are raising are partially answered by this, as well. “what do you when the client …”, “we communicate … and our relationship of openness, honesty and trust kicks in”

    • Carlos Ribas

      Good point.  So, how to manage clients who are naturally more skeptical of software development (perhaps they’ve worked with too many programmers who cannot complete FizzBuzz)?

      • Anonymous

        One way we do it is to break up the deliverables into an initial period, where we do 2-4 weeks of development to deliver something of value. The client can then decide to initiate more work or stop and move on. It’s like a “trial period” where we prove we can deliver working software. 

  • Bob Cravens

    Because story points provide a relative measure of complexity, they are easier to determine than providing hour estimates. Breaking stories into tasks and estimating hours on each task is quite resource intensive. Typically we only do this for stories that are about to enter the production pipeline (sprint). Breaking down all the stories up front, represents a waste in terms of lean.

    Yes you owe it to your client to convert story points to hours.  Typically, the conversion factor from story points to hours estimate is your team’s velocity. Are you calculating team velocity? If yes, then where do you feel the lack of accuracy in velocity is coming from?

    • http://profiles.yahoo.com/u/6BKSFCTQBEUGFPGCJORKT4UBVY Andrew

      Bob, the difference here is that a Client is asking for an estimate to determine if the new work is going to enter the pipeline.  It’s a bit different when dealing with external clients.  You can’t expect to tell a client that you can’t do an estimate until you are about to do the work, because a client will tell you that they don’t want you to do the work until they have an estimate.  It’s a nasty chicken/egg scenario.

      It’s one of the costs of doing business. 

      • Bob Cravens

        I am not suggesting you don’t or can’t supply an estimate. You certainly can and should. I am suggesting the following:

        1. Collect user stories that define the work to be done.
        2. Estimate these using story points (much faster than breaking down into tasks and estimating hours)
        3. Use your team’s velocity to convert the story points into the lead time for the work.
        If a team has reached gone through the forming/storming/norming/performing phases, then their velocity should be known enough to provide the estimates within the certainty discussed in the article (10-50%).

        My question to Jimmy (or others) is using the velocity as a conversion flawed? If so, how?

        • Anonymous

          Our customers don’t sign checks with points or lead time estimates.  That’s not to say that what you lined out isn’t important or valuable, but that ultimately cost is what counts in a lot of cases. “How much will this cost” is a different question than “when can I have it”. 

          • Bob Cravens

            I don’t understand. You are breaking your estimate down into time (hours) to get to the the final cost. I propose that you can get the same estimate for time from story points and velocity. Then perform the same math to convert the time to cost.

            v = your team’s average velocity in story points per spring
            t = length of sprint in hours
            sp = total story points for work

            hours = sp / (v * t)

            cost = hours * billing factor

            The second equation we both end up using. I contend that calculating your hours from story points / velocity takes less time and provides an equivalent accuracy.

            Not being a PITA, just curious as to if you are calculating velocity? If so, what issues were you having using it for your estimates?

          • Anonymous

            We don’t do sprints. It’s all pull-based flow and lead time calculations.

            Group estimation was a big waste. Sprint planning meetings were a waste. In fact, by the end of the scrum project, we weren’t doing scrum, we found a faster way to deliver without timeboxed sprints.

            The ceremony around sprints were a waste, so we eventually ditched all of the activities. All those group sessions are really, really expensive and we found better usages of our time.

          • Bob Cravens

            I personally like the pull-based paradigm also. With your pull-based flow, do you have metrics for lead time? For example, a cumulative flow diagram plotting story points versus time for the stories started / finished can provide lead time per story point. This can be used to convert story points to time in this flow-based paradigm.

            I am glad that you have found something that works. I am curious still about the need to perform the detailed analysis (break down into tasks and provide hour estimates). It still seems a bit excessive to get to the 10-50% estimate.

  • http://profiles.yahoo.com/u/6BKSFCTQBEUGFPGCJORKT4UBVY Andrew

    What bothers me about Agile is all the ceremony about “complexity isn’t effort” or “effort isn’t hours”.  When asked about how complicated something was, if you answered “Well, that’d take me about a day” you’d get slapped down because you are associated hours with story points.  But really, as much as we all try and abstract it, it all boils back down to hours.  For Jimmy, he just takes one additional step and converts hours into money that a client spends.

    That’s why I like this approach, it eliminates unnecessary frustration by saying “Feature X will cost you $5,000″.  That way a client can make a good, educated decision about it.  Is adding this feature to my website worth 5k?   

    Any well written contract will have contingency baked in (Jimmy’s risk factor), so it’s unlikely that you’ll get screwed  if there are unknowns that alter the time lines.  If the client changes their requirements, you rip up the estimate and start again.  

    You also have to remember that if you get decent at estimating, you should be under budget sometimes as well, so even if you do have to eat a little margin because your estimate was 5% off on an individual feature, you potentially will make it up on the next.  Of course, if you are always underestimating, then you need to adjust your factors a bit, but that’s a different issue.

     

  • http://www.hollman-alu.nl Aluminium Kozijnen

    Yeah that is correct.. Our customers don’t write checks in points. They write checks in dollars..

  • Joshua Barker

    What is the difference from modeling and design?