Training resources regarding CQRS and ES

At our company we use CQRS and ES. For the newer products fully implemented in this new architecture we do not use any database but just the file system as data store. This new way of developing requires a significant and somewhat radical mind shift. What used to be best-practice in the traditional way of coding does not apply any more. Things are solved differently.

Definitions

DDD: Domain Driven Design
CQRS: Command Query Responsibility Segregation
ES: Event Sourcing

Web-Sites

• Good site about DDD/CQRS/ES: http://domaindrivendesign.org/cqrs
• DDD sample of event sourcing in C# (accompanies Vaughn’s new book on IDDD): http://lokad.github.com/lokad-iddd-sample/
• DDD/CQRS Community around the world: http://cqrsguide.com/world
• Case studies of CQRS/DDD projects: http://cqrsguide.com/case-studies

Videos

• 6-hour video about CQRS by Greg Young: http://www.viddler.com/v/dc528842
• Video about DDD/CQRS by Greg Young: http://www.youtube.com/watch?v=KXqrBySgX-s
• Search for other videos by Greg Young, e.g. on Sills Matter
  • http://skillsmatter.com/podcast/design-architecture/simple-is-better,
  • http://skillsmatter.com/podcast/design-architecture/cqrs-not-just-for-server-systems?goback=.gde_2470540_member_49148769

Videos of Greg Young, Udi Dahan, Eric Evans, et. al.: http://skillsmatter.com/event/design-architecture/ddd-exchange-2011

People and Blogs

Important people to follow

• Greg Young, http://codebetter.com/gregyoung/,he coined the term CQRS
• Rinat Abdullin, http://abdullin.com/,we use a customized version of his “framework” (Lokad.CQRS – http://lokad.github.com/lokad-cqrs/)
• Jonathan Oliver, http://blog.jonathanoliver.com/
• Udi Dahan, http://www.udidahan.com/?blog=true

Discussion Groups

Important discussions about and around DDD, CQRS and Event Sourcing happen in this Google group: http://groups.google.com/group/dddcqrs?pli=1

Most of the people having some relevance in this area are present on this group and answering questions.

Post Footer automatically generated by Add Post Footer Plugin for wordpress.

Abstract

In our company we were looking for a way to radically simplify the way we implement our enterprise software. We wanted to get rid of accidental complexity introduced by using complex and expensive middleware and server software like RDBMS. In this talk I’ll give an overview on how we use CQRS and event sourcing and how we were able to significantly improve our velocity and at the same time concentrate more on the business requirements and less on technical implementation details.

See you there!

Post Footer automatically generated by Add Post Footer Plugin for wordpress.

In this post we will discuss how events generated by our aggregates are serialized and then stored in the event store. Remember, we do not use any database to store data and thus have to provide our own persistence mechanism.

Note: the code snippets presented in this post represents a simplified version of code found in Lokad.CQRS. This code is used to show the core concepts.

Serializing and deserializing an event

Since we are not going to use a database to store our data we are now on our own. Let’s first choose a serialization format that suits our needs. What do we need?

• Serialization and deserialization should be fast
• The serialized data should be as compact as possible
• The serialization process should be tolerant for changes in the events, e.g. allow us to rename properties of the event or add new properties to the event

It turns out that Google’s protocol buffer format is the ideal choice. And luckily we find a Nuget package which gives us an implementation of the serializer/deserializer for .NET.

Let’s define an interface that provides what we need

We have a method SerializeEvent which accepts an event and returns the serialized event as array of bytes. Of course we then need the counterpart, which does the opposite. The method DeserializeEvent accepts an array of bytes and returns the deserialized event.

Now please note that the SerializeEvent method accepts any event that implements IEvent<IIdentity>. The serialization of the event is no problem, but to be able to deserialize the event we need to know the concrete type of the event we had previously serialized. Thus we need to somehow serialized the type or rather the contract name of the event together with the content of the event. This fact slightly complicates the whole process. But as you will see, it is still no rocket science involved. Each step is simple.

Lets first define a helper class Formatter which contains a contract name of an event and a delegate to serialize and another to deserialize this event

As you can see, the serializer delegate takes an object and serializes it into a stream. The deserializer delegate takes a stream (containing the serialized event) and deserializes its content and returns it as object.

We want to create an instance of Formatter for each event that we have in our system. To get all events we can use code similar to this

The result will be our know event types. Note that line 18 will be evident in a minute.

The Formatter class introduced above is hosted by the EventSerializer class which is responsible for the effective event serialization/deserialization. We inject the known event types via constructor into this class. The EventSerializer takes these known types and creates two dictionaries out of it

• one that gives a formatter instance provided the event type and
• the other gives the event type provided its (contract-) name

We use the RuntimeTypeModel class of the protobuf-net library to get a formatter (the instance that serializes/deserializes the event to an array of bytes). We also use an extension method GetContractName to get the contract name of the event type. It is defined as follows

In the above method we take the namespace of the event from the [DataContract] attribute with which we have to decorate each event in order to make it serializable using the protocol buffer format (see our NewTaskScheduled event).

With all this preparation the actual serialization of the event is quite easy

The method Serialize shown above takes an event instance and its type and serializes it into the given destination stream.

The deserialization is a two step process. First we have the contract name of the event and want to get the corresponding (event-) type

Having this type we can deserialize the event

The above method gets the stream from which it reads the serialized content of the event as well as the event type. The method returns the deserialized event (as object).

That was not so bad, wasn’t it? No magic or rocket science needed so far.

Ok, then we can now discuss the implementation of the IEventStreamer interface that I introduced at the beginning of this section. This class that we will now discuss not only writes the (serialized) content of the event to a stream but also some message contract information (or message header; where an event is a message).

First of all the EventStreamer class uses our EventSerializer

Let’s now show the SerializeEvent method and then discuss the various parts of it.

The method consists of 3 parts

• line 22-27: we use the event serializer class discussed above to serialize the event (=content)
• line 29-36: we serialize a message contract which contains the event type name (=contract name), the length of the content as well as the content position (=messageContractBuffer)
• line 38-45: we open a stream and first write the (serialized) message header contract into it (line 41). Then we append the messageContractBuffer to the stream and finally we append the content to the stream. Last we return the content of the stream (line 44)

The DeserializeEvent method has to do the exact opposite of the above method. Let’s have a look at it

On line 50 we create a memory stream around the buffer containing the serialized data. Then we have again our 3 steps

• line 52-53: read and deserialize the message header contract. From it we get the length of the message header that will be deserialized in step 2
• line 55-58: read and deserialize the message header. From the previous step we know exactly how many bytes we have to read (header.HeaderBytes)
• line 60-65: read and deserialize the event. From step two we know the length of the content and thus how many bytes we have to read (contract.ContentSize).

To be complete I also have to show the MessageContract and the MessageHeaderContract classes. the MessageContract class contains information about the event (i.e. the event).

We specifically need the contract name and the size of the event when it is serialized (the content length). Since the message contract is serialized by using protocol buffer we have decorated it with [DataMember] attributes.

The MessageHeaderContract contains information about the MessageContract, namely the length of the message contract when it is serialized. It also contains logic to write and read itself to a stream. We do not need protocol buffer here since it is trivial and of fixed length (just an long – which is 8 bytes long).

With this we have the basis to be able to store events in the event store and subsequently read them back from the event store when needed. Let’s now look at the event store in detail.

Saving events to the event store

We want to create a file per aggregate instance which contains all events generated by this particular aggregate. Any new event is serialized into an array of bytes (as discussed in the previous section) and then appended to this file. With each event we also store

• the length of the serialized event (the data length),
• the version of the event (starting from 1 for the first event in the life cycle of an aggregate) and
• the hash code of the serialized event to recognize whether the data is corrupt or has been tampered.

Let’s start and create a class that allows us to append an array of bytes (the serialized event) to the file which contains all events of an aggregate. This class has an Append method which accepts as parameter the said array of bytes.

Note that for write operations the file is opened in a mode that allows a single writer but many concurrent readers (line 16).

We use a helper class TapeStreamSerializer to do the actual write operation. Note that in this first draft we always write version = 1 to the file, no matter how many events we already have stored before.

Lets now look into the TapeStreamSerializer class. The WriteRecord method uses a binary serializer to write the record into a memory stream. We also use the SHA1Managed class of the .NET framework to calculate the hash code of the serialized event.

On line 22 to 24 we create a header containing the length of the serialized event (the data array) and write it into the memory stream. Then on line 26 we write the actual data array into the memory stream and on lines 27 to 31 we add a footer section to the stream. The footer contains once again the length of the data array, the version of the event and the hash code computed from the data array.

Once we have everything written to the memory stream we append this data to the file (line 34) and we are done.

The above method uses two simple helper methods to write a 64bit integer (lines 23, 28 and 29) and a hash code (line 30) into the memory stream.

Reading events from the event store

We now want a way to read all existing events of a given aggregate from the event store. For this purpose we implement the ReadRecords method in our FileTapeStream class.

On line 42 we make sure we arrived at the end of the file and no more records can be retrieved. On line 45 we again use the helper class TapeStreamSerializer to do the actual reading of a single event record from the file.

The above method returns an array of TapeRecord items. A TapeRecord item contains the serialized event as well as its version

Let’s now look into the ReadRecord method of the helper class. We basically have to revert the write operation we described earlier. First we try to locate/read and validate the header information (line 72-74). The header has a fixed length, thus we know exactly how many bytes to read. Then we read the data (lines 76 and 77). We know how many bytes we need to read since the data length was stored in the header. Finally we read and verify the footer which is also of fixed length (line 79-92). Specifically we make sure that the stored hash code corresponds to the ad-hoc calculated hash code of the data array (line 83-91).

We use the following helper method to read a 64bit integer

and this one to read the hash code

and finally this one to read and verify a specific signature like e.g. ‘header start’ or ‘footer end’.

Summary

In this post we discussed in detail how events generated by aggregates are serialized and the appended to the event store. I also showed how those serialized events can be read from the event store.

In my next post I will discuss how we can integrate this code into our sample application. Stay tuned.

Post Footer automatically generated by Add Post Footer Plugin for wordpress.

In our company we were looking for a way to radically simplify the way we implement our enterprise software. We wanted to get rid of accidental complexity introduced by using complex and expensive middleware and server software like RDBMS.

In part 1, part 2 and part 3 of this series I introduced some of the core elements of our CQRS/ES architecture. In part 4 I discussed some of the questions that readers and coworkers have when confronted for the first time with the ideas of command-query-responsibility-segregation (CQRS) and event sourcing (ES).

Tying it together

As promised in part 3 of this series I’ll try to bring everything together in this post. From part 1 (the magic triangle) we know that the client sends commands to the domain. Lets have a look at what happens here

The Silverlight client creates a command and sends it via a WCF service to the server. The WCF service on the service side forwards the command to a command dispatcher which in turn uses the IoC container to resolve the command handler that can handle this command. There can only be exactly one command handler per command registered in the IoC container. This is a direct consequence of the fact that a command always must have a single target. If no command handler is found, or more that one command handler for this particular command exists then this is a fatal error.

The command dispatcher then forwards the command to the command handler. In our system the command handler is implemented as a chain of responsibilities (also called a pipeline).

When the command enters the pipeline in the first step we test whether it would violate any business rule. If yes then the execution is aborted and the command result type is set to ValidationFailed and the violated business rules (text messages) are returned.

In a second step we test whether this particular command needs an electronic signature (ESIG) and if yes we verify that a valid ESIG is provided. Again, the command execution is aborted if the ESIG is missing or wrong and an error message is returned to the client.

There are more steps in our system which lay outside of the scope of this post.

Finally, if everything is fine so far the command arrives at step n, the command execution action. Here the command is executed. In our case the command is forwarded to the aggregate factory. The aggregate factory executes the following tasks

• analyze the incoming command and determine which aggregate instance the command targets.
• load all existing events for this particular aggregate instance from the event store.
• Create a new instance of the aggregate and re-hydrate the aggregate with the events loaded in the previous step
• forward the command to the aggregate instance

Contract

Before I can continue with code I have to introduce a couple of contracts (interfaces) that we use in our infrastructure to make the code as simple as possible.

First of all we recognize that every command or event that we deal with in our system is a message. Thus we introduce a message interface. Since the interface IMessage is already used by other code (e.g. messaging system) we use ICqrsMessage as the base interface.

It is just an empty (marker) interface. But now we want to distinguish commands and events and we introduce the following two contracts

and

It is important to note that both interfaces are covariant (out) in the generic parameter TIdentity. This makes it possible to cast an ICommand<Foo> to an ICommand<IIdentity> if Foo implements IIdentity.

Finally we have the interface which all identities (IDs) used in our system implement

Having introduced these interfaces (or contracts) we can now continue to show the aggregate factory. The command execution action gets the aggregate factory injected

here the interface that each command action implements is simple and look like this

note that the Handle method is a void method. A command execution never returns data in CQRS.

In order to make our code compile we need to adjust the definition of the ScheduleNewTask command and the TaskId identity that we introduced earlier. The TaskId is an identity and needs to implement the interface IIdentity

and the command needs to implement the command message interface

Now we should be able to compile and can continue with the implementation of the aggregate factory. We have this

On line 13 the ID of the aggregate that is target of the command is determined. Since we use typed IDs throughout the system the ID not only gives us the value but also it tells us which type of aggregate it targets (Task aggregate versus e.g. Animal aggregate).

On line 14 we load the events from the event store that have been previously stored for the aggregate with the given ID. This collection can be empty if the command is the first command in the life cycle of the aggregate.

On line 15 we get the observer that later on is injected into the aggregate. The observer is an infrastructure function that takes the event(s) that the aggregate triggers and handles them. Specifically the observer stores the event(s) in the event store and then publishes them asynchronously.

On line 17 we determine whether the target is a task or not. If yes, we create the state of the task from the events (line 20) and then we create the aggregate and inject the state and the observer to it (line 21). Finally we forward the command to the aggregate for execution (line 22) and return (line 23).

If we have other aggregate types in our system we repeat line 17 through 24 for each aggregate type, e.g.

For the moment I do not want to discuss the details of the two methods LoadEventsFromEventSource(…) and GetObserver(). I will explain the details of these in detail in a future post. Suffice to say that the observer has the following signature

It’s an action that has one input parameter, the event that it handles. We have learned above that each event is an IEvent<IIdentity>>.

Let’s now revisit our Task aggregate and aggregate state that we introduced in an earlier post. Let’s start with the state.

the state has a constructor with one parameter, the collection of events injected by the aggregate factory. The state re-hydrates itself with these events (see the foreach loop).

The aggregate on the other hand gets the state and the observer injected

The aggregate also implements the Execute method which is called by the aggregate factory

The RedirectToWhen is a static helper class with uses reflection to determine to which When method of the aggregate the command has to be forwarded. The code is very simple due to the fact that we use the convention to call all our methods on an aggregate that execute a command ‘When’ and that each of these methods has one single parameter, that is the command they handle.

Of course, in our production system the code looks slightly more complex since we cache the meta data to avoid repetitive use of (slow) reflection. But this series of posts it about the principles and not so much about technical details!

We do not have to forget to make our events implement to events contract, e.g.

That’s it, simple, isn’t it? Well you might disagree with me here… but the apparent complexity stems from the fact that the whole process had so many steps. Each step in isolation is really really simple, agreed? There is absolutely no “magic” or “rocket science” involved. We need no expensive tools of frameworks so far to achieve our goals. And that is one the essence of this series of posts: we can achieve our goals by using simple code and leaving alone expensive middleware or server software.

I have to admit, so far there is a pretty important piece missing in the puzzle. It is the queuing system that we use. But this will be the target of my next post.

Look back

Let me try to visualize once again how a command flows through the system

Code

You can find the sample code on GitHub.

Till next time!

Post Footer automatically generated by Add Post Footer Plugin for wordpress.

Eventual consistency

Full consistency (also called immediate consistency) is overrated, period!

When I have some doubt about whether something “is right” or “can possibly work” then I try to find analogies in the real world. And it turns out that when looking for analogies in the business world where eventual consistency applies then we find those everywhere! Even in scenarios which are “business-critical” or “life-threatening”. But then if real world works in an eventual consistent way, why are we tying to achieve full consistency at all cost in our LOB applications?

Let me give some real world samples of eventual consistency:

• you are ordering a book at an online book store. After you have checked out and your payment by credit card has been accepted the application tells you that your order has been placed and will be shipped asap. At the moment you receive the confirmation the online store has not yet reserved the book for you nor is it guaranteed that this very book is on stock and if not that it is still available from the publisher. It some times happens that the book is not on stock and thus the delivery is delayed. In this case the online shop will notify you by sending an email. If the book is not available any more then the shop might offer you alternatives and compensate you with a special bonus.
• The unit price of some share traded at the stock exchange changes. This change in price is appearing on the stock tickers only with a certain (short?) delay; maybe a couple of seconds or even minutes.
• You order a coffee at your local coffee shop. The cashier forwards your order to the barista immediately and then takes your credit card to check out. To check the credit card and approve the payment might take a while. In the mean time the barista has already started to brew your coffee. It turns out that your credit card is invalid and you have no alternative payment method. The coffee cannot be undone and as a “compensating action” the barista gets the order to throw the coffee away.
• A terrible accident happens. You call 911 and explain that there is a person having life threatening injuries. The person answering the call promises that EMS and fire fighters will be there as quickly as they can. At the moment you end the call the fire fighters and EMS have not yet been sent on their way nor are any surgeons put on hold that will take care of the injured person. All this is organized asynchronously. When the injured person finally arrives at the hospital he gets treated as soon as a team  of surgeons/doctors is available. The patient might even have to wait for some time if no doctor is available.

All these samples are not made up by me but are very real and happen daily! Thus I can assure you that we can develop applications that are eventual consistent and at the same time happily fulfill all business requirements. We just need to address the problem a little bit different. A recommendation is to design your application in a task oriented way. Each task should be very focused and well defined. Avoid having “fluffy” or vague use cases.

After a successful transaction I need to refresh/reload the screen

With CQRS and ES this is not possible since the read model is only eventually consistent with the domain model. The delay can be very short (a couple of milliseconds) or quite long (seconds to minutes) depending on the type of transaction.

But this is not a problem! Why not?

We design our application in a way that commands sent to the domain model have a high probability to succeed. This can be achieved by e.g. pre-validating the commands prior to sending them to the corresponding aggregate of the domain model. Assuming now that the command succeeds we can update the screen to reflect the new state without reloading everything from the database/data source. We already have all the necessary data at hand, otherwise we would not have been able to create and send the command in the first place. As a sample: if we send the command “PublishTask” to the domain model and the domain model sends back an ACK then we can set the status of the corresponding task on the screen to “published” without querying the database.

But what about if it takes a long time (seconds to minutes) until the read model is in sync with the domain model after a transaction has been executed?

In this case it is often the case that the person that triggers the transaction is not the same person/user that needs to see the result of the transaction (other than it has been completed successfully). As a sample take again our online book store. The manager decides to reduce the unit price of all books in stock by 10% to better compete with the competitors. Now who is most interested to see those changes in price? It is not the manager since he knows what he has just done. He does not have to browse through the list of all books to have a confirmation of the change. But the clients of course benefit from this price reduction. But it doesn’t matter if the price change is only reflected in the online catalog minutes after the manager has executed the price reduction.

Why not use a database for the event store or the read model

Well, first of all, please read the title of my series of blog posts: “How we got rid of the database”. This series is not only about the theory behind CQRS and event sourcing but also about the concrete application of an architecture based on CQRS/ES as well as our goal to reduce the overall complexity of the system. If we can achieve our goals with simpler means why not do it. Why shouldn’t we strive for simplicity when solving problems and achieving solutions?

It turned out that we did not leverage nor need all the features an RDBMS a la Oracle or SQL Server could offer. It was as if I took a truck when a bicycle would be sufficient or search for a calculator to calculate 5 + 5, instead of using my brain.

What we need when doing CQRS is

• in the domain model: an efficient way to access a stream of events identified by the unique ID or the aggregate instance which produced the events.
  Solution –> we serialize all events as they occur into an append only file whose full name is a codified version of the ID.
• in the read model: an efficient way of accessing (denormalized) instances (=the current state) of view models by their respective unique ID in addition to (secondary) indices that are e.g. serialized hash tables (or dictionaries). I will talk more about indices in a future post.

All that and more can be easily achieved by avoiding any database (relational or document) and just using some simple code that can read and (atomically) write single files.

How can we enforce referential integrity?

We cannot, at least not at the data level!

Integrity of data is solely guaranteed by our application. It is the responsibility of the application to maintain integrity and consistency of the data.

One important aspect in this regard is that we eliminate the concept of “delete” from our vocabulary (I know, I know, there are always exceptions…). In the business world the concept of “delete” does not exists

• we do not delete an employee but rather release him
• we do not delete an order but rather cancel it
• we do not delete a coffee in a coffee shop but rather discard it (the ingredients have already been used)
• an accountant does not delete a wrong journal entry but rather compensates it with a journal correction

If ever we have to “delete” an item we do not physically delete it but mark it “as deleted”.

But if we never physically delete anything from our data store we do not run into problems like “orphaned children” or “missing references”.

As an added benefit we are better prepared for any audit, since no data is ever lost.

What about backup?

We only ever need to make sure that our event store is backed up. We really(!) don’t care about the read model since it can always be rebuilt from the event store in case of catastrophic failure. And if this is not sufficient, we can always create the read model redundant; we just send the events (which are used to build the projections) to at least two different servers.

The event store on the other hand is just a folder full of files and can be backed up by any decent backup program out in the market. No special and expensive software needed!

Post Footer automatically generated by Add Post Footer Plugin for wordpress.

In this post I want to show how these published events can be used by observers (or projection generators) to create the read model.

Querying

When the user navigates to a screen the client sends a query to the read model. The query handler in the read model collects the requested data from the projections that make up the read model and returns this data to the client. The client never sends queries to the domain. The domain model is not designed to accept queries and return data. The domain model is optimized for write operations exclusively.

Assuming the user is navigating to the screen where existing tasks can be edited the query that the client triggers could be

and the data returned by the query handler would look like this

where

contains the full name and id of the candidates that are assigned to this task and

contains all the details of an animal that is target of the task.

Where does this data come from I hear you ask… Well, that’s the topic I’ll discuss next.

Generating the read model

Whenever we design a new screen we need data to display on this screen. Thus we define projection(s) that are tailored in a way that best suits our needs. Ideally we want to create a projection that allows us to get all the data we want with one single read operation to the data store. That is the ideal, but in reality that is not always possible and thus we just want to state this principle: design the projections in such a way that we can retrieve the data needed for a screen using the minimal amount of read operations, ideally one operation only.

This principle requires as a consequence that we store our data in a highly de-normalized way. We regard data duplication in the read model as a necessary consequence and do not try to avoid it. Storage space is extremely cheap nowadays.

Since we are not using an RDBMS to store our read model the projections do not have to be “flat”. We can projections that are made of object graphs.

A first approach would thus consequently be to define a projection that looks somewhat similar to the query response object, that we define above. Let’s do so

Please note that I use the typed Id (TaskId) introduced in part 1 in my view.

Now we need to define a class, that creates the task details projection for us. We want to make the implementation of this class as simple as possible. The class should be a POCO and only depend on a writer object

The writer that we inject into our projection generator class is responsible to physically write our views into the data store. In our case the data store will be the file system but it could be as well a table in an RDBMS or a document database or a Lucene index. From the perspective of this generator class it doesn’t really matter what type of data store it is.

The definition of the IAtomicWriter interface is very simple

For convenience we can then write some extension methods to this interface

Why not add these methods to the interface directly instead of writing extension methods you might ask. The reason is that we should always try to keep our interfaces as simple as possible such as that our code remains more composable and less coupled.

Having defined the interface and also added the above extension methods we can now continue to implement our projection generator class.

Each projection is created by events. All data that make up a projection are provided by events. In our sample the first event in the life cycle of a task is the NewTaskScheduled event that we defined in part 1. Let’s add code to handle this event in our projection generator

We use the same convention as we already introduced with the aggregate: we call all our methods When. Each of those methods has exactly one parameter, the event that it handles. This convention makes it easier for us to later on write some tools around our read model. I will discuss this in detail in a later post.

Note that we have used the Add (extension-) method of the writer since at the time when the NewTaskScheduled event happens the corresponding view does not yet exists.

Later on other events of interest might be published by the domain and our projection generator can listen to them. Let’s take as a sample the TaskPublished event. We add the following code to the projection generator

very simple, isn’t it?

The resulting file on the file system could look similar to this

(remember that we are using JSON serialization).

In the next post I’ll try to wire up all pieces that we have defined so far such as that we can run a little end-to-end demo. Stay tuned…

Post Footer automatically generated by Add Post Footer Plugin for wordpress.

In part one of this series I started to explain what we do, when e.g. a user (in the particular case a principal investigator) wants to schedule a new task. In this case a command is sent from the client (the user interface of the application) to the domain. The command is handled by an aggregate that is part of the domain. The aggregate first checks whether the command violates any invariants. If not the aggregate converts the command into an event and forwards this event to an external observer. The observer is injected into the aggregate by the infrastructure during its creation. The observer accepts the event and stores it in the so called event store.

It is important to realize, that once the event is stored we can commit the transaction and we’re done. What(?) – can I hear you say now; don’t we have to save the aggregate too?

No, there is no need to save (the current state of) the aggregate; saving the event is enough. Since we store all events that an aggregate ever raises over time we have the full history at hand about what exactly happened to the aggregate. Whenever we want to know the current state of an aggregate we can just re-create the aggregate from scratch and re-apply all events that we have stored for this particular instance.

We can extend the diagram that I first presented in my last post  as follows

If we now look at just the events of one single task instance we would have something like this

If we create a new task aggregate and apply all the n events that we find in the event store then the result is a task aggregate that has been completed.

It is important to know that new events are always only appended to the stream of existing events. Never do we delete or modify an already existing event! This is equivalent to the transaction journal of an accountant. The accountant always only adds new journal entries. He never modifies an existing one nor does he ever delete a previous entry.

But how do we store these events? There are various possibilities, we could serialize the event and store it in a column of type BLOB (binary large object) or CLOB (character large object) of a table in a relational database. We would also need to have a column in this table where we can store the ID of the aggregate.

We could as well store the (serialized) event in a document database or in a key value store (e.g. Hashtable).

But wait a second…, if we serialize the event why can we not just store it directly in the file system? Why do we need such a thing as a database which only adds complexity to the system (it is always funny to point out that a database ultimately stores the data in the file system too).

Let’s then define the boundary conditions:

• we serialize the events using Google protocol buffer format
• we create one file per aggregate instance; this file contains all events of the particular instance
• any new incoming event is always appended at the end of the corresponding file

Advantages

• Google protocol buffer
• produces very compact output
• is one of the fastest ways to serialize an object (-graph)
• is relatively tolerant to changes in the event signature (we can change names of properties of add new properties without producing version conflicts)
• since we usually only operate onto a single aggregate per transaction we need only one read operation to get all events to re-construct the corresponding aggregate
• append operations to a file are fast and simple

Disadvantages

• we have to decorate our events with attributes to make them serializable using Google protocol buffer

If we are using protbuf-net library, which is available as a nuget package, to serialize our events then we can use the standard DataContractAttribute and DataMemberAttribute to decorate our events

Note that the order of the properties and there data type is important but not their name.

Assuming we have our event serialized and it is available as an array of bytes we can use the following code to append it to an existing file

The code is a simplified version of the productive code we use in our application. What is missing is the code that  also stores the length of the data buffer to append as well as its version. Whilst incomplete the above code snippet nevertheless shows that there is no magic needed and certainly no expensive database required to save (serialized) events to the file system.

Since we did not want to reinvent the wheel we chose to use the Lokad.CQRS “framework”. If you are interested in the full code just browse through the FileTapeStream class.

Loading the stream of events from the very same file is equally “easy” and the implementation can be found in the same class.

In my next post I will talk about how we can query the data. Stay tuned…

Post Footer automatically generated by Add Post Footer Plugin for wordpress.

I want to write a series of posts which describe in detail how we do things in my company. What architecture do we use, which patterns do we follow, and more specifically, how do we implement those features. I invite you to join me on this journey…

Note: the core ideas follow the various examples of Greg Young, Rinat Abdullin, et al.

It all start with a simple picture

A similar image was used by Greg Young in his 6-hour video about DDD, CQRS and Event Sourcing. This image nicely shows us what is happening in an application following CQRS. The client sends commands to the domain (-model). The domain either accepts a command and hence sends ACK (=acknowledged) or refuses it altogether and sends a NACK (not acknowledged) back to the client. If the domain model accepts a command then it produces an event as a result of the execution of the command. The event is consumed by the read model. The read model in turn is used by the client to get its data in the form of DTOs.

Lets write some code to demonstrate how we do things. I will start from scratch, that is from a blank solution, and incrementally develop the  code we need to fulfill the requirements.

But first we have to give a somewhat realistic user story to start with:

As a principal investigator I want to schedule a new task. The task has a group of animals as targets upon which the executor of the task acts. The task is addressed to a group of crew members from whom one can chose to execute the task…

Evidently a user (= the principal investigator) wants to schedule a new task. This is done by the system by sending a command (ScheduleNewTask) to the domain. After discussion with the domain expert it is clear that the command should look like this

The name of the command gives us the context. From it it is clear what the intent is. The command name should thus be unambiguous.

The command is a DTO and its payload usually contains the minimal data necessary to successfully fulfill the request. In our case these are

• The (client side generated) unique ID of the new task. Note, that we do not just use a simple type for the ID like int or Guid but rather a class TaskId.
• The name of the task (in combination with the due date this should be unique)
• A description of what the executing person is supposed to do
• The date (and time) when the task is due
• A list of IDs of the candidates (staff) that can choose to accept (and execute) the task
• A list of IDs of the animals which this task targets.

Now let’s look a little bit closer at our ID. The code to is looks similar to this

Evidently the “real” id is of type Guid and there is also an associated Tag (=”task”). The tag is mainly used to output a human readable version of the id. For this purpose the ToString() method is overriden. Having an ID of type TaskId for a task and e.g. StaffId for a staff makes those IDs distinguishable. Has we only used Guids for both type of IDs we wouldn’t have been able to tell which one belongs to which type of entity.

The command is targeting the domain. Inside the domain there is an aggregate which handles the command. We call this aggregate TaskAggregate.

To make things simple and discoverable we use some simple conventions. In the case of an aggregate we call the method who handles a command When and we always pass exactly one parameter to this method – the command itself. This results in very readable code (at least regarding the method signatures).

The method first makes sure that the command doesn’t violate the aggregate’s invariants, e.g. like this

In the above code snippet the aggregate uses its internal state to determine whether this command is the very first command (Version = 0) or not. If not then we have a problem since the system wants to re-create an already existing task instance; which of course we do not allow.

Once the invariants are guaranteed the aggregate transforms the command into an event. The event tells any observer(s) that the corresponding command has been successfully executed. The name of the event is in past tense, i.e.

In our case we have a one-to-one matching of the command’s payload with the event payload. This one-to-one matching is often the case but by no means the only possibility. Often an event can be enriched by additional data if needed by any observer.

It is very important that we always name our events in past tense. This makes it clear that an event is something that has already happened and cannot be undone!

Now lets look again at the aggregate. We extend the method’s code as follows

Note that once we have created the event from the command, we are call a private helper method Apply. The method Apply forwards the event to the aggregate’s state and also to an external observer

The initial aggregate state and the external observer are injected by the infrastructure into the aggregate at its creation time

The (external) observer injected by the infrastructure will take this event, store it in the event store and forward it to our queuing system which eventually will dispatch it asynchronously to any registered consumer of the event. Typical consumers are the writers who create and update the read model.

In part 2 we will discuss some more details about how events are stored in the event store. Till then, have a great time.

Post Footer automatically generated by Add Post Footer Plugin for wordpress.

Overview of Nhibernate 3 Beginner’s Guide

· Clear, precise step-by-step directions to get you up and running quickly

· Test, profile, and monitor data access to tune the performance and make your applications fly

· Reduce hours of application development time and get better application architecture and performance

Read more about this book and download free Sample Chapter

How to Enter?

All you need to do is head on over to the book page and look through the product description of this book and drop a line via the comments below to let us know what interests you the most about this book. It’s that simple

Product description for NHibernate book: http://www.packtpub.com/nhibernate-3-beginners-guide/book

Winners from the U.S. and Europe can either choose a physical copy of the book or the eBook. Users from other locales are limited to the eBook only.

Deadline

The contest will close on 30/12/11 PT. Winners will be contacted by email, so be sure to use your real email address when you comment!

Post Footer automatically generated by Add Post Footer Plugin for wordpress.

Yesterday Anders Hejlsberg gave a talk about future directions of C# and VB.Net. He started with look back and then immediately dive into the new features of C# 5.0 which is part of VS 2011 and .NET 4.5. The most prominent new feature is of course the direct language support for asynchronous programming. The compiler provides two new keywords async and await which doesn’t sound exciting at all but in reality dramatically simplifies the way we developers can write “fast and fluid” code. Instead of dealing with threads, thread synchronization and callback functions ourselves we can now write fully asynchronous code in a sequential way. The compiler will do the heavy lifting for us and convert this “sequential and seemingly synchronous” code into a sequence of asynchronous calls and their corresponding continuations.

If you think that this is not worth talking about then try to write a sequence of asynchronous calls that depend on each other. And make sure that you provide adequate exception handling.

During his presentation Anders Hejlsberg also revealed some details about the future version of C# and he announced that a first developer preview will be available in about 4 weeks. The main part of the upcoming version of C# is a completely new compiler which is written in C# and can be used as a service. The project has the code name Rosalyn.

This “compiler as a service” will be of tremendous value for various scenarios. All kinds of tools, utilities or plugins will be able the leverage the compilers capabilities to their favor. Some samples are

• Intellisense, code coloring will leverage the output of the parser part of the compiler
• Refactoring tools like Resharper do not have to provide their own parser any more and can leverage the compiler
• Compile and execute scripts or code snippets (provided e.g. as string) on the fly
• Future versions of Visual Studio will leverage the compiler and provide an interactive console for C#
• Automatically translating code from one language to another; Anders demoed the translation from C# to VB.NET and vice versa
• Meta programming a la Boo or Ruby will be possible
• etc.

I think this new compiler will once again dramatically improve the development experience with C# (and VB.NET).

The new XAML editor in VS 2011

If you have worked with WPF or Silverlight in Visual Studio 2010 or 2008 you know that the XAML editor was extremely slow and limited in its functionality. Now that has changed dramatically in VS 2011. First of all the new XAML editor is backed by the same engine (or at least part of it) as Expression Blend. The editor now runs in its own process to not compromise the stability and availability of VS. The editing capabilities have tremendously improved and members of the VS team  confirmed me that they are tuning the editor as much as possible and make it performing well also in very large project with many assets.

Expression Blend for HTML5/CSS applications

Expression Blend is now available in a new flavor. HTML5/CSS3 based Metro style applications can now be visually designed in Blend. Many of the new CSS3 features are fully supported by this new editor. Not only is Expression Blend a layout tool but it fully supports the interactive mode (that is JavaScript is executed) which is very important in scenarios where most of the page is rendered dynamically, driven by JavaScript.

Post Footer automatically generated by Add Post Footer Plugin for wordpress.