Comments on: How we got rid of the database–part 3

By: Steve Friend

Steve Friend — Wed, 20 Jun 2012 10:47:00 +0000

”every job I’ve had data has needed to be updated almost in real-time; if
Joe Customer places an order in our store, we need an agent to follow
up with it relatively fast, if not immediately. ”
Really? How do you cope with peak hours, do you increase the number of agents you have so they can respond immediately? Do your customers place order by ‘phone?

I work for a large-ish online retailer (think ~$500m sales/year) and we don’t need to respond to orders immediately.

By: Jimmy Bogard

Jimmy Bogard — Wed, 20 Jun 2012 02:01:00 +0000

One thing to keep in mind is that the user typically doesn’t think of screens as reports. Putting a date on the screen doesn’t disabuse them of the desire to treat data as data, if that’s what they really want.

Eventual consistency is a recent invention. Before computers, eventual consistency was across departmental boundaries, but not internally. In good task based UIs, users reason about tasks and think about success/fail in terms of requests, not reports.

By: darylmer

darylmer — Tue, 19 Jun 2012 21:53:00 +0000

Just want to put in a little note here about some of the questions that people have and mention that the Lokad CQRS sample project has been updated and is slightly different in a few areas. As for the views, the way I usually describe the eventual consistency scenario is to think about those views almost like a report. When you run a report is it only current at the instant it completes. By the time you see it on screen or printed it should be considered old and that is why most people put the date and time a report was run on it. People generally accept that a report is outdated basically the instant it completes.

The views are updated pretty darn quick. The only scenario where they wouldn’t be is if there was a fairly significant number of commands that got dropped in just before your command. I have been working on a project to convert data from an old system to a new system via issuing the old data as new commands and this is where things can get bogged down. I am looking at finding a better way to deal with views used as indexes because an add to the index requires a full read/full write of the entire view as an atomic operation which bogs as there are more items added to the view. I have thousands of commands issued in a a very short time(this is not a normal situation and is only for conversion) and you really see the impact when it has to open the index view update the data and then write it all back for each add/update. This is an optimization situation where you have to get creative. Under normal circumstances this isn’t and won’t be an issue for me, however, if you plan on running under pretty heavy load you will have to find a better way to deal with this which may mean keeping a lightweight traditional database only for the indexed views.

By: Anonymous

Anonymous — Tue, 19 Jun 2012 18:01:00 +0000

Thank you John, you hit the nail! I wouldn’t have been able to explain it any better

By: John Teague

John Teague — Tue, 19 Jun 2012 17:33:00 +0000

This really depends on implementation and what you mean by immediate feedback. One way to give immediate feedback even if the command and event process takes a while is to update the UI as if the command will succeed. A core tenant to CQRS is to make commands as small as possible with high probability of success. If the command passes validation, the chances of failure should be very small. So you can update the user’s view with the result as if it did succeed and in the case of failure, take a “compensating action.” Like alerting the user if they are still online or sending an email message if they are not.

A common example of this is credit card validation. While the number is validated in real time, more verifications must take place and are done after the order is places, like credit limit checks. If there is a problem you get an email letting you know your order was not placed and why.

By: John Teague

John Teague — Tue, 19 Jun 2012 17:28:00 +0000

I hear this all the time too when I talk about CQRS. There’s a couple of things I always talk about when I’m asked. The first thing I say is that updates in real time are myth, at least it’s so difficult to really do it usually not worth it.

The example I give is a user is looking at an update screen, with the “current” data populated so they can change it. The user then takes a 20 minute phone call and then comes back makes the change. Did you check before making that update there were no changes while they were on the phone? It’s not guaranteed to be consistent. We’ve sold to the users this myth that that they have consistent data when it’ really not,unless you go through great pains to ensure that’s the case.

So assuming you aren’t guaranteeing perfect consistency, with an eventual consistency approach, you now must ask the business what level of consistency they need (explaining at 0ms consistency is not possible). You’ll be surprised at the answers you get, 5 minuts, 20 minutes, 1 hour. I’ve got applications that update data from different systems once a day, as requested from the business owners. It will be rare that they give an answer that is under 1 second. If it is, then your job is describe the time and resources required to meet that SLA.

Once you know what your consistency contraints are, you can now give solutions to the problem based on an actual business requirement.

In your comment you said “almost real time” is that real time from the users perspective or from the machine perpective? Is 5 seconds “almost” or is 5 milliseconds. You can send a message to the other side of world in 5 seconds, 5 milliseconds is a different story. The point is, until you ask you don’t know.

By: Monsters X

Monsters X — Tue, 19 Jun 2012 14:41:00 +0000

Why not use couchdb or something, same storage (JSON objects) but all sorts of easy fanciness (like clusters, atomic transactions, etc..)

By: Anonymous

Anonymous — Tue, 19 Jun 2012 13:03:00 +0000

That is a concern most newcomers have. I did too! But in reality it turns out not to be a problem; why? I’ll dedicate a upcoming post to that topic – eventual consistency. Stay tuned.

By: Wayne M

Wayne M — Tue, 19 Jun 2012 12:24:00 +0000

I’m just a little bit confused by this although it seems interesting. How do you “sync” the data between the read model and the write model? In a real-world scenario users are constantly updating the data (or customers are placing orders, whatever) and this usually has to be reflected fairly quickly in a back-end system with the “read model”. If you’re using let’s say Sql Server for the write model and RavenDB for the read model, how is the data meant to be synced without impacting performance or making users wait a day to get the current day’s orders, etc.

I guess the main confusion point for this whole CQRS type of architecture is that I don’t quite get how it applies to real scenarios; every job I’ve had data has needed to be updated almost in real-time; if Joe Customer places an order in our store, we need an agent to follow up with it relatively fast, if not immediately. Is this type of architecture just not suited to the majority of transactional use cases or am I missing something crucial?

By: Oded

Oded — Tue, 19 Jun 2012 08:41:00 +0000

@cbp – Eventual Consistency. There is no guarantee for seeing _immediate_ results.