How do you handle simple pub-sub, evented architecture in rails apps?

I’ve been asking this question in various forms, via twitter, for a few days now. I’ve received a number of answers from people and have spent some time talking with Jak Charlton about the patterns that I would have used in a .NET Winforms app vs what I should look at in a ruby / rails app. At this point, I’m still confused.

Rather than continue to ask bits and pieces of my questions via twitter, I want to try and get my thoughts out in something coherent… to explain the context of what I’m building and hopefully find some solid advice on how to handle this situation (and the many similar situations that I need to handle).

 

The Context

Joey Beninghove and I are building a medical application. In one part of the system, we are dealing with patients and their current medications. We have to store the medications that they are currently taking. There are multiple places that can modify the medications a patient is taking. For example, the patient profile will have a current medications list and allow that list to be edited right there. In another part of the system, patient medications are selected as part of a treatment plan.

The medications selected in the treatment plan are not immediately applied to the patient, though. The system waits for a doctor to select all of the treatment options for the patient before signing off and saving those options as the patient’s current treatment plan. Once the treatment plan is signed off, the medications that were selected in the plan will get applied to the patient as part of their current medications list.

Since our system deals with medical data for patients, we have to be HIPAA compliant (HIPAA is the “Health Insurance Portability and Accountability Act” in the United States). This compliance requires us to keep audit trails for who is reviewing and modifying patient information (among other things). This means that every time a medication is changed for a patient, we have to log who is making the change, when they are making it, and what it’s changing to.

 

The Idea

I want to keep the various parts of the system decoupled, as much as possible. I don’t want the treatment plan to have to know about the patient profile and current medications list, directly. I don’t want the patient profile or treatment plan to have to know about HIPAA compliance auditing, either. The amount of work that it will take to process either parts of the system, in relation to changing the medications for a patient, would make it very difficult to keep the solution clean and maintainable if I just coded it all directly together (even using separate classes that get called from all these places).

Enter the idea of pub/sub and evented architecture…

In my .NET days, working with Winforms apps for several years, I fell in love with the event aggregator and command patterns. I even wrote up several lengthy blog posts on them and created some sample implementations that I put out on Github. I’ve used that code base many many times, to reduce the coupling in my .NET apps, with great success. I’ve also worked a lot with messaging architectures, using pub/sub, point to point messaging, etc. I’ve written my own service bus for one app. I’ve used MassTransit in another. I’ve played around with NService, too. In the end, the event aggregator and command patterns are in-process versions of the various messaging patterns, so the knowledge transfer between all of this made it easy to understand and cross-polinate ideas.

Given my success with these patterns in .NET, I wanted to look at using them in my ruby / rails app. What I want to do is raise a “MedicationsModified” event / message, and have an arbitrary (unknown from the source that raises the event / message) set of handler receive the message and process it correctly. This is exactly what pub/sub or the event aggregator should do for me.

I know very little about the pub/sub and messaging options in ruby, though. So I’ve been asking on twitter and I’ve been pointed to a number of different options. After some discussion with Jak Charlton, as well, I think the idea of using an EventAggregator may not what I need, either. There aren’t any real implementations of this around the ruby community, which makes Jak think that it’s not a solution that fits the ruby / rails problem space. I haven’t yet decided if I agree with this, but it’s good advice to take in as I’m looking for my solution. In either case, the principles of pub/sub to send an event message out to an unknown number of handlers is what I believe my system needs.

So… what are the pub / sub options that I’ve found, so far?

 

The Problem

The majority of the options that I’ve been pointed at seem to fall into 1 of 3 categories:

  1. far too large and “scalable” for my needs, or
  2. far too small and “do-it-yourself”
  3. delayed execution of existing model / methods

For example, Resque seems to be the hotness right now, for large scale background processing. After all, if github is building it, you know it’s going to be scalable and easy to use. Resque falls squarely in the first category. I don’t need a scalable redis server with a server farm to handle my needs.

On the opposite end of that spectrum, we have solutions like EventMachine and Cool.io. Both of these tools are super simple to get up and running and provide a nice back-end host to offload a job to. However, neither of them provides any kind of message or event handling system. I would have to either find a message handling system to host inside of EventMachine / Cool.io, or build one myself.

I’ve also had Observers and Fibers, from ruby 1.9, suggested. While these are both nice options to know about for the problems that they solve, I’m not sure if they are what I’m looking for. Observers seem to create a little more coupling between the observing object, and observed object. Fibers offer a nice asynchronous in-process solution, but don’t really offer any help with messaging / message handling.

Other systems, such as Delayed-Job and Background-Job seem a little more down to earth and more of my scale. However, they are examples of the third category that make me a bit nervous about coupling. For example, delayed job lets you turn any object’s method call into a background process by calling the object’s method with a .delay call in front of it: some_object.delay.some_method.

I don’t want my handling of the “MedicationsModified” even to be stripped away and boiled down to an implicit method call on the objects that need to be called. That would couple things together more than I want, and create maintenance problems in the future. Even if I created a “MedicationsModified” class and had a method on it that would call all the things I need to call, that class would still be coupled to the things that need to be called. Any time I need to change what has to happen when a medication is changed, I would only have to modify this one class, but I would have to deal with the knowledge of all places in the system that need to be notified of what’s going on.

I also don’t necessarily need my solution to be asynchronous or out of process. I just need it to be a simple pub/sub message handler – in or out of process. Though, as Jak pointed out in our conversations, a web server is already an asynchronous / out of process system (with the UI being the browser on the other end of the internet), so I’m not really opposed to an out of process solution. It just needs to be simple.

 

The Questions

It’s obvious that my brain is still wired for the .NET and Winforms paradigms that I spent the last 3 or 4 years working in. I’m running into some serious mental roadblocks and unable to wrap my head around the available options and solutions for my scenario. So I’m asking you, dear reader:

  • How do you handle simple pub-sub, evented architecture, message handling in rails apps?
  • What options and suggestions do you have for my scenario and my needs?
  • How have you solved this problem, while keeping the complexity and sheer size of the solution within reason?

Any and all suggestions, gems to look at, links to blog posts, articles and presentations, are greatly appreciated.

 

Use My Existing Rails Models

One constraint that I haven’t mentioned yet… I want to use my existing rails models in the message handlers, so that I don’t have to duplicate them in whatever the solution ends up being. Tools like DelayedJob do this for me, and I can even make it work with EventMachine (but I’m not sure of the ramifications of loading up config/environment.rb in my EventMachine server).  I wanted to mention this so that you can keep it in mind when suggesting solutions and answers to my questions.


Post Footer automatically generated by Add Post Footer Plugin for wordpress.

About Derick Bailey

Derick Bailey is an entrepreneur, problem solver (and creator? :P ), software developer, screecaster, writer, blogger, speaker and technology leader in central Texas (north of Austin). He runs SignalLeaf.com - the amazingly awesome podcast audio hosting service that everyone should be using, and WatchMeCode.net where he throws down the JavaScript gauntlets to get you up to speed. He has been a professional software developer since the late 90's, and has been writing code since the late 80's. Find me on twitter: @derickbailey, @mutedsolutions, @backbonejsclass Find me on the web: SignalLeaf, WatchMeCode, Kendo UI blog, MarionetteJS, My Github profile, On Google+.
This entry was posted in Analysis and Design, AppController, Messaging, Principles and Patterns, Ruby. Bookmark the permalink. Follow any comments here with the RSS feed for this post.
  • Anonymous

    Thank you for asking this question! I’ve been trying to wrap my head around this *very issue* while experimenting with RoR for a pet project of mine. If you get answers outside of comments on this post, please let us know here what you find out.

  • http://twitter.com/davidwtbuxton David Buxton

    What you want sounds similar to Django’s signals. Perhaps reading their documentation will help give you ideas how to do it with Rails.

    https://docs.djangoproject.com/en/dev/topics/signals/

  • http://twitter.com/JRGarcia J.R. Garcia

    Have you looked at Faye (https://github.com/jcoglan/faye)? It seems like a simple solution and gets integrated into Rack (from what I can tell).

    @twitter-121771125:disqus That reminds me of how GNOME handles this as well (http://developer.gnome.org/gtk-tutorial/2.90/x159.html). It worked out well for me doing GTK+ programming.

    • http://mutedsolutions.com Derick Bailey

      Faye looks like a web sockets implementation to communicate with the browser. I need a back-end messaging system to run code from my rails models. 

      … the overloading of the term ‘pub/sub’ these days, is very frustrating. i’m not talking about websockets or communicating with a browser. back end code, only, here.

  • http://mutedsolutions.com Derick Bailey

    @MrMcDowall:twitter suggested looking at ActiveSupport::Notifications, via twitter: https://twitter.com/#!/mrmcdowall/status/78669933130686464

    This looks promising… it may be exactly what I need. My initial impression is that it looks like an event aggregator pattern built right into rails: http://api.rubyonrails.org/classes/ActiveSupport/Notifications.html

    I’ll play with this a bit and see how it works out.

    • http://twitter.com/MrMcDowall John McDowall

      Also, Chapter 5 of Crafting Rails Applications – ‘Managing Application Events with Rails Engines’ covers some nifty stuff about Notifications. Don’t let the examples full you though, it’s not just for logging and metrics! 

      Good luck, and I look forward to seeing future posts about your adventures in this area!

      • http://mutedsolutions.com Derick Bailey

        yeah, the example you linked to threw me off for a minute, since it was about logging. but once i looked up the api docs for it, i realized that i was more generic than just logging. i’ll do a follow-up post with whatever solution i end up using.

    • Amiel Martin

      My first thought was ActiveSupport::Notifications. Did you end up trying it? How’d it work out?

  • http://davidrmcclelland.blogspot.com/ David

    I’d be interested to see this functionality in an ASP.NET MVC3 application as well – anyone know of a good reference implementation?

    • http://twitter.com/danielauger danielauger

      I’ve been wondering about this for MVC3 as well. This week, I ended up
      doing a spike that hosts Rhino ESB inside of a MVC3 app for pub / sub
      and queuing. So far, so good. I’ll probably blog about it if it ends up
      working.

  • Geoff Bennett

    I might be missing something here, but can you not simply use the REST API that you’re building to do this very thing? I realise you’re talking about backend processing, but there’s nothing in the “rulebook” that says you can’t do it. JSON your model and dispatch it to the relevant URI.

    • http://mutedsolutions.com Derick Bailey

      I’ve heard of people doing this… I’m not sure I would want to, mostly because I don’t want to open an API that would be available to the general world, to accomplish this. Otherwise, it’s an interesting suggestion… certainly something that would be worth thinking about in another circumstance.

  • http://subdigital.myopenid.com/ Ben Scheirman

    Have you thought of using Rails ActiveRecord observers?  It seems like a decent idea to publish a message (a job) to Resque (or DJ) based on lifecycle events in your model and have it do whatever it needs to do asynchronously.

  • http://www.facebook.com/doronrotem Doron Rotem

    i am dealing with the same issues right now.

    i have already choosed resque as my background queue/worker, but i don’t want to pollute my actions/models and other libs with calling resque. instead i want to trigger/raise/send events that others can subscribe to. an in-process pub/sub. 

    so, before reading about AR notifications, i thought about creating a simple pub/sub class that anyone can use (it will be created in the application controller in a before filter so it would be available to anyone). most of the work subscribers would do is enqueing things for resque.