Success and statistics

The 2009 CHAOS Summary has some interesting numbers:

This year’s results show a marked decrease in project success rates, with 32% of all projects succeeding which are delivered on time, on budget, with required features and functions

Wow, terrible numbers.  A 32% success rate?  68% projects fail?  Quite depressing, but I see a different picture.  Let’s look at the definition of “project success” here:

On time, on budget, with required features and functions

Against this, I like Mike over at the Leading Answers blog, who has a great list of projects that did not fit any of these criteria, yet were considered a success.  Here’s one example:

Titanic
(The 1997 film not the original ship). This film was six months late, massively over budget and finished with a bloated 194-minute running time. Seemingly not a good performance given the original schedule, budget and scope requirements. Yet the film turned into an enormous critical and commercial success, winning eleven Academy Awards, including Best Picture and became the highest-grossing film of all time.

Clearly these measures matter to studio execs, as flops like Waterworld can sink entire studios.  But software, like movies, have a different definition of success that isn’t strictly limited to the waterfall-ish metrics set out at the beginning of the project.

On time?  That release date was no more than a wild guess.

On budget?  Again, a wild guess based on limited knowledge of scope.

Required features and functions?  Says who, the original RFP?  The requirements document?  Or the end users, who actually use the system?

I’m no expert in measuring success, but I know what a happy user is.  Clearly success has many more vectors than the tired “on-time, under budget” mantra.  I’ve worked on deathmarch projects vastly over budget, under estimated and released months late.  Yet it generated new sales, new clients, new opportunities and new relationships.  It was an awful experience, and we learned quite a bit from the project, but it was still a company success.

But then you do have stinkers like Waterworld, where indicators like budget and missed deadlines can show a project in trouble.  But these are only one indicator, which need to be balanced against others like user feedback, client interest, and so on.

Budgets and deadlines are important, but not the final answer in customer, personal and company success.  It’s time we stop limiting ourselves in our definition of success.