I need some ‘wisdom of the crowd’ here.

I’m guessing just about ever single one of you (at least the software developer types) reading this have deployed software to a production environment at least once in your career. One of the products we’re working on is getting near to a major milestone and will start seeing wider deployments than the beta and preview deployments we’ve seen so far. I’m in charge of developing a list of different things we’ll need to do to the product to start wrapping it in a bow for wider distribution and deployment.

What sorts of things, in your opinion and experience, should I be most concerned about?  Some of things I have on my list are:  Good logging and error reporting, well-defined and practiced strategy for handling updates and patches going forward (including incremental DB schema migration), etc.

I’m also looking for any “Whatever you do, DO NOT do *this*” where *this* is some thing you tried and ended up being a terrible idea or blew up in your face.

If you comment, please keep your advice short and to the point.  Though I would encourage you, if you have a funny and/or insightful story to share, to blog and link it back to this post for me and others to read!

About Chad Myers

Chad Myers is the Director of Development for Dovetail Software, in Austin, TX, where he leads a premiere software team building complex enterprise software products. Chad is a .NET software developer specializing in enterprise software designs and architectures. He has over 12 years of software development experience and a proven track record of Agile, test-driven project leadership using both Microsoft and open source tools. He is a community leader who speaks at the Austin .NET User's Group, the ADNUG Code Camp, and participates in various development communities and open source projects.
This entry was posted in Advice, Deployment. Bookmark the permalink. Follow any comments here with the RSS feed for this post.
  • P

    Have a solid version-ing strategy; this may not be a problem if you can push updates with every release, but when different customers have different versions, it can be a nightmare if a version strategy wasn’t thought about.

  • What is it? A site? An installable app? A site with an installable app? An installable site with and app?

  • The best thing you can do is thing of 5 things that could go wrong. A SQL query goes bad, the disk fills up, etc. What is your plan for dealing with that? Can you redeploy a new version of the application quickly with better logging? Do you have tracers that you or your customer can turn on to see what is going on? Do you have access to your environment to debug in production?

    If you have no plan for dealing with these things, that is the biggest problem. Something will go wrong, you can’t really plan for it because it is usually some weird thing that no one thinks about. But you can have a plan for what it is you are going to do when someone calls and says your app is not working.

  • Philbo

    Do not release at 6pm on a Friday. You’ll be there for hours whilst everyone else is in the pub.

  • KevDog

    What I find to be one of the biggest helps in the early phases of deployment is to add a very quick means of gathering user feedback. We have a silverlight/wpf control that we add to products in development so that clients/users can give us feedback and impressions. It writes to a dedicated database that lists the function or page they are using, along with what there notes and complaints are. Some products have a “send feedback” link, but we have found that having them go outside the system is a barrier to getting more information.

    It’s been invaluable in turning around user feedback into action items.

  • KevDog

    The most famous mistake is “Never get involved in a land war in Asia, but only slightly less well-known is: don’t go in against a Sicilian, when deployment is on the line”.

  • 1. Never underestimate the importance of a good installation process.
    2. AutoUpdate is gold.
    3. Test on EVERY platform you support (seems silly, but is overlooked). Basically, test on as many machines you can. Best case, find a teenage girl with a social media addiction and run it on her computer. If it can run there (with ever virus known to man) you are probably doing good.
    4. The amount of testing required for an external release is double that of an internal release. Plan accordingly. Internal releases can be (more easily) backed out and get a second chance. No such luck for external releases. You just look stupid.
    5. Gold Standard: don’t just log errors, automatically upload them to your server.

  • mendicant

    The worst thing that happens to us is when we have an app go bad and none of the Ops guys are around to get us into the server and get logs, etc.

    Make sure you have a good way of not only logging the errors, but getting access to them (if possible), or at least an easy way for them to forward the bad ones to you.

    In the end, we have started putting elmah onto our prod apps with only the devs having access and it’s helped us provide far better support for our clients.

  • Joe

    From a totally different point of view, and this all depends on your app and audience, of course:

    1. Support in place to handle a bunch of questions from new users, many of whom will do things you would have never thought of.

    2. Administator(s?) to handle accounts, roles, permissions

    3. Some kind of doc/video showing how to use the application.

  • phat shantz

    By the time your program grows up, graduates, and moves off to production, your company and product infrastructure should be as solid, inviolable, and rigorous as any assembly line.

    Production can be a very revealing time if pre-production has taken a different tack or has been approached with less rigor or professionalism. You fight like you train. You release in production like you release in pre-production. If that frightens you, then go back, change your pre-production patterns to those that more-resemble full production and try to release another beta. If it is much harder then you have done well. If it is the same, either you are too rigid in pre-production or you don’t know what it’s like to run a production pathway.

    Your exception-noting loop must contain communications, discrimination (what is worst, what is next), replication and remediation pathways, remediation plans, and full version release processes.

    The release process includes feature/bug tracking, feature addition/bug fix, release definitions, change control, versioning, test candidates, test harnesses, final builds, test results, and release candidates with their release installers.

    The only thing worse than introducing a bug in software is introducing more while you fix the first. For this reason, the version release process can never short-circuit any of your testing and durability design validation processes.

    Application revisions must be strictly (but that is not to say tightly) controlled. Any build and release process must follow the same steps regardless of urgency. A release candidate must be a complete build. It must be built using the exact same process as previous releases (of the same major version). It must be tested with the same (or more) unit, data, interactivity, automation, and other testing. It must only be released if verified 100% by the same person tasked with releasing every build.

    Feedback from the customer is vital. So, too, is personal and compassionate response to customer feedback. It is essential that every company offering consumer products have a consumer advocate on staff. In a software company, this person should intimately understand the software, but still be able to take the customer’s side. “We can’t make it do that,” is not a valid response to a bug report.

    Before a release, take a team poll. Everyone must agree that this release candidate satisfies all requirements and durability for a production release. If not, then do not release the product. Go back. You still have plenty of time until the moment you release. Then you are now and forever out of time.

    Give it some thought.

  • Venu
  • Never do a live data migration / schema change / bulk update after midnight.
    Script it, test it, save it for the morning.

  • Great feedback everyone. Lots of good wisdom here. Thanks a ton!

  • Caladin

    If you don’t have it, set up elmah to mail all errors to a
    email account that the lead gets forwarded to him.

    Also set it up to store the errors locally on the servers as XML so you can browse tehm later (though with email you almost never use this)

    Make sure your logging on it’s most verbose level logs the entry and exit of every function, especially in your libraries.
    this way you’ll almost always be one page of code from your error immediatlye after browsing the call stack & the logs in the exception. Once things are stable, you can turn this down to get more performance. (though you will pretty quickly become addicted and want to keep it on all the time)