Comments on: Efficient transactional processing

By: Vasilio Ruzanni

Vasilio Ruzanni — Sun, 26 Jun 2011 13:44:00 +0000

Totally agreed on differentiation between clearly OLTP-scenarios and batch-processing of data.

By: Charlie

Charlie — Tue, 03 May 2011 19:38:00 +0000

We have implemented a solution like this that processes 25k card payments at the end of every month. We run a query against the database to get the 25k requests and push them into a queue. We then set the number of threads on our message handler to match the number of available channels on our 3rd party’s web service. We plan to introduce a distributor to load balance across two physical servers for fault-tolerance.

By: Scooletz

Scooletz — Sat, 23 Apr 2011 06:14:00 +0000

Have you considered using sagas and timeouts for each of the subscriptions? Then, there’d be no need of the mentioned query.

By: Carlos Ribas

Carlos Ribas — Fri, 22 Apr 2011 18:48:00 +0000

Well I was thinking along the lines of the original problem — it is calling out to some kind of credit card processing system. Problems beyond data manipulation (which, if done against a set, often can be done at the database, especially in a system where projecting from SQL to DTOs/bypassing the entity’s behavior for reads is deemed safe/acceptable) can’t be reduced to doing it at the SQL level.

Definitely agree with the cleanliness of a unit-based approach. And doing that on top of a message architecture is typically a good idea because it gives you flexibility. But the original point was about performance, so I was just calling out that maybe an in-process threading approach (where each worker is still just doing one aggregate) would make more sense since with a high unit cost (1 second) the host machine would be doing a lot of waiting anyway…

The part about multiple entities being updated at the same time is NH-related implementation detail, right?

By: Josh Stone

Josh Stone — Fri, 22 Apr 2011 17:02:00 +0000

I should mention, in our case the number of aggregates we needed to process was suitable for a single server. But triggering processing via a new published message is a good approach since it treats the processing as a new use case, with a new request, transaction, etc.

By: Josh Stone

Josh Stone — Fri, 22 Apr 2011 16:59:00 +0000

I actually dealt with a scenario like this recently. One transaction per aggregate is indeed the way we went, though it doesn’t need to be distributed. Our solution just involved a thread pool does the processing for each aggregate in turn, each with its own transaction.

By: Anonymous

Anonymous — Fri, 22 Apr 2011 16:53:00 +0000

No, you’re right. The other valid option I see is a set-based approach, doing this at the SQL level. But looping through more than one aggregate root to do work – I don’t see that as a real valid option any more.

We might still do this synchronously, through in-process handlers, but still have the process generate messages.

I don’t even think that it’s a performance question – separating into separate handlers reduces complexity, since I’m only dealing with one operation at a time against one aggregate root, and not having to worry about a unit of work with multiple entities getting updated at the same time.

By: Carlos Ribas

Carlos Ribas — Fri, 22 Apr 2011 16:41:00 +0000

Sometimes comparing the overhead of the messaging system vs. the work unit cost makes it worthwhile to do in-process concurrency through some flavor of multi-threading. Handling failure does become complex, but often failure is expected to be very rare and thus the performance trade-off against that added complexity becomes worthwhile if the goal is truly focused around performance.

This example case from Ayende (with a whopping 1 second per unit cost) is well-suited for your proposed solution, but certainly folks should not get the impression that using a message/bus architecture is a one-size-fits-all solution to problems in this space.

By: Anonymous

Anonymous — Fri, 22 Apr 2011 15:31:00 +0000

In that case, I’d have a saga coordinating the processing of all of those child operations, with compensating actions in case of failures.

Or, if this is really a set-based operation and not transactional processing, a simple UPDATE statement would work too.

By: Anonymous

Anonymous — Fri, 22 Apr 2011 15:30:00 +0000

So if there are 1M items to process, the last in line gets processed quite slow. Messaging decouples that for me.