Concepts and features – an example

One of my favorite posts by Ayende covers the idea of concepts and features.  Design occurs at the concept level, whereas features build on top of concepts.  This can be a little abstract, but I recently ran into a situation where the presence of a concept in one area allowed for feature development, while the absence of a concept in another area made change difficult.

Adding features to concepts

In this system, we integrate with many 3rd party vendors through file drops.  Some data has changed in another system, or users have interacted with the system, or system-level events have occurred.  When this happens, our system consumes these files and makes decisions based on what the data in the file represents.

These files come in a variety of formats, CSV, tab-delimited and XML.  But one common concept in all of these files was that they represented a set of messages.  We already introduced the concept of executing a “batch job”:

public interface IBatchJob
    void Execute();

We then had a batch job executor that could execute any batch job.  When a requirement came up to log exceptions when a batch job executed, this was a trivial task.  We already created a system-wide concept of executing a batch job, it was now just a matter of adding that feature.

The next concept we needed to augment was the idea of handling a message.  Each entry in the file was transformed into a message object, and a message broker located and executed the appropriate handler:

public interface IMessageHandler<T>
    where T : IMessage
    void Handle(T message);

We needed to know specifically when a handler failed, to log the specific message and exception so that the exception could be examined and the message potentially re-processed.  Again, because all control for executing message handlers went to a single message broker, this is trivial to implement.  Applying the Single-Responsibility Principle and concepts of orthogonal design meant that no implementation needed to change to add this concept.

Absent concepts

The tipping point for identifying concepts comes when adding of a kind of feature becomes difficult, and needs architectural design work.  In our case, it was the need to record the history of processing files.

We already have the concept of executing batch jobs and handling messages.  The missing concept was processing a file.  That is, locating a file, parsing it and converting to messages, and actually executing those messages.

There are about five different ways of doing this at the moment.  In some cases we use a persistent batch object to represent the workflow of processing the file.  In other cases, we just have some parser class that then invokes the message broker.

But in no place in our system architecture was the concept of processing a file.  When it came time to add features to this concept, and the concept didn’t exist in our system, it became pretty daunting to augment a non-existent design.

In this case, we waited too long to formalize the concept of parsing files.  We hadn’t needed to add new features to this absent concept, but building out the concept itself with the existing shape of disparate features would have made adding subsequent parsers much easier to think about and develop.

Concepts are quite similar to building blocks.  Building on top of a concept means I already have a set of building blocks to start from.  Without concepts, I have a blank canvas.  This can be appropriate sometimes, but other times it leads to wildly different implementations.

Even though each implementation of the file parsing was self contained, overall ease of maintainability suffered because there was no common theme, no unifying concept, no design.

Balancing design and YAGNI

It’s easy to put off concept design with the excuse of YAGNI.  We waited until we needed the feature of recording the parsing outcome before we went down the road of designing the concept.  Unfortunately, that left quite a bit of rework and redesign, as we had so many different ways of parsing files.

Instead, recognizing the concept of parsing files early would have made subsequent file parsing easier to understand.  We wouldn’t have had to think about what pieces we needed to invent before putting together the puzzle.  The outlines would have already been there, and we would have just needed to fill in the blanks.

I’m all for starting with blank canvases.  But on a regular basis, introspection is needed to search for common concepts to build upon.  Otherwise, we lead ourselves into wildly heterogeneous architecture.

About Jimmy Bogard

I'm a technical architect with Headspring in Austin, TX. I focus on DDD, distributed systems, and any other acronym-centric design/architecture/methodology. I created AutoMapper and am a co-author of the ASP.NET MVC in Action books.
This entry was posted in Design. Bookmark the permalink. Follow any comments here with the RSS feed for this post.
  • Hm… sounds like DDD would have helped. If you work according to DDD then concept identification is part of understanding requirements, because concepts are part of the Ubiquitous Language. This does not mean you need to implement all concepts, but at least you´re clear early on which concepts there are.

    And I´d and from personal experience that then architecting a system as consiting of processes (yes, I know, doesn´t sound very OO ;-) but it works nevertheless) does help to translate concepts into code. So whatever you see in an implementation is part of the Ubiquitous Language – and the other way round.


  • Nolan Egly

    This is why it’s so important to have a good understanding of the scope of a new application. Unfortunately, that’s hard :)

    It’s difficult to balance “analysis paralysis” against “YAGNI – oh wait, yes we DO need that”, and even then the client is going to change their mind several times after swearing the requirements are frozen. Design is an iterative process.

    Past experience with similar systems helps a lot (Jimmy will probably make file processing more explicit in his next EDI system). There are some good general books on architecture out there. I like to read project post mortems to understand what system design challenges other teams have run into and how they solved them, but good ones are hard to find.

    How else do other readers “search for common concepts to build on”?

  • Great post. It’s something I’ve been thinking about quite a lot lately. I notice you said, “we waited too long to formalize the concept,” not “we should have discerned and modeled this concept at the outset.” That would be YAGNI and paying a debt that might never come due.

    My own “concept needs formalization” alarm is getting better tuned with experience, I feel. For me it works best like this. The product owner orders a feature. I notice right off that it’s like some existing feature. Okay, let’s keep it DRY then. So I fire up Resharper and set to refactor the existing code to support the new feature. When it starts to feel sticky and procedural, the the concept alarm rings. Let’s model it. Ah, that feels better.