Starting a new project – refactoring an existing system

Well, the team is starting on a new project. Our company is in a situation that is probably familiar to many people. We have some existing code, we have data in the database, everything ‘sorta’ works. But now we want to add some new functionality, and there is fear. Fear that changes to the existing system will break things that currently work.

So far we have spent about a month trying to figure out the existing system. I spent some time meeting with developers who have worked on the existing code, spent lots of time meeting with the product owner, and lots of time looking at the code. It is definitely going to be a tricky problem. One of the most difficult things for me has been trying to work this project in an agile method. We have created stories and added them to a backlog, but a lot of them are very vague – “Understand and document Document table” is one example.

Here are some of the challenges we have already identified:

  • The whole system is basically a document management system that stores scanned images (Well Logs) and data about those well logs.
  • The system is comprised of loaders that read data coming from somewhere else that evaluate the data and then store the meta data in the database and the files on a fileserver.
  • Right now, there are at least 3 loaders in active use, each of which is similar to the others, but with a lot of copy-paste code reuse.
  • The database schema has a concept of a generic document, but has been ‘polluted’ over time with well log specific columns.
  • The is very little of the SOLID principles in use – we have jsp’s that do presentation, database queries, and filesystem operations, for example.
  • There are very few tests of the existing system.
  • The document management system is fairly central to the company, and manages much more than just the well logs, so any changes made for well logs have to take other documents into account also.
  • There is a pricing system in place that charges customers for downloading the logs (naturally, with duplicated logic in various places).

One key thing we are trying to keep in mind as we work on this project is that we want to deliver business value as quickly as possible, but we also want to improve the system and reduce development effort, as we are fairly certain that there will be additional features and datasets to process in the future. So we’re trying to strike a balance between a full rewrite of large parts of the system and shipping something that provides value quickly.

I’ll keep writing as we progress on this project. It has definitely been one of the more difficult things I have ever taken on. I’d welcome anyone’s feedback on how they have taken on similar projects.

This entry was posted in Uncategorized. Bookmark the permalink. Follow any comments here with the RSS feed for this post.
  • Though I haven’t done it in practice I’ve recently had to think about a similar situation and I’ve been looking at utilising an anti-corruption layer to provide an interface between the old code and the new. There’s a decent explanation of it in the Evan’s DDD book.

  • That sounds tough. Michael Feathers’ book Working Effectively With Legacy Code might help–it helped me add functionality to a legacy system. In that situation, I mostly added unit tests around code related to functionality we wanted to change, then refactored and made the changes. But I’ll admit that sometimes just getting things unit-testable involved having to make “careful” changes, like pulling methods out of the jsp/aspx in order to get tests around them at all. And having to work like that always feels slow, too.

  • That’s a nightmare and a half. May your team be strong, optimistic, and persistent, your clients friendly, and any managers above you patient. Judging from what you’ve said you’re already on the right track.

    I personally haven’t been through a refactor/rewrite of that magnitude, so I can’t cover all of the angles. I’m afraid that the only thing I can suggest, is something that you probably already know–

    Since you want business value as quickly as possible, and that value is something that is determined only by your clients, I think it would be advantageous to be aware of which changes to the system they will quickly and easily recognise and adore. For end-to-end systems, I’d say this is likely to be anything visual followed by anything improving perceived performance. I think expectations management is going to be key as the whole project seems very volatile.

    Provided you choose to follow such a strategy, I’d be hesitant putting everyone on the chosen front of greatest initial impact, as you may lose reconnaissance on other ends of the system which could eventually chuck a monkey-wrench in your solution. Might depend on the size of your team? Perhaps I’d prioritise subsystem work order based on their value-add factor.

    In general, I can imagine that it might be advantageous for the team to focus most of their energy on one part of the system at a time and move across the system linearly if possible, so you eliminate the problems that can come with meeting in the middle.

    Hopefully some of this is helpful. Good luck!

  • @Garry – thanks – I think we’re going to need multiples of those!

    @Amy – Yes, the Feathers book is definitely a favorite around here. We actually had all the developers in the company go through it using a book club format about a year ago. Trying to get some characterization tests around things is where we are starting.

    @Michael – thanks for the good thoughts. We are a team of 5 developers (including myself – I also server as team lead/scrum master), 1 Project Owner, 1 Project manager, who helps out with analysis also, and 0 testers.

  • Please keep us updated, Steve, with what you decide to do and your success with your process. Good luck!