Starling: A Ruby Persistent Queue Server That Speaks Memcached
It's been around for a while now, but Starling is a "light-weight persistent queue server that speaks the MemCache protocol." Starling makes it ridiculously easy to set up a network-accessible queue (or many queues) for, say, asynchronous job processing between multiple processes and machines. It was developed by Twitter to handle the heavy amount of queueing necessary to keep their service ticking over. Starling is proven in production, with not only Twitter using it in anger, but FiveRuns too. FiveRuns have even created their own fork that, they say, is significantly faster.
Why the sudden interest in Starling? Well, Glenn Gillen has written an excellent introductory guide to setting up Starling over at RubyPond.com. He walks through the process of using Starling (and Workling, a Rails plugin to make using Starling easier) from installation, through to actually adding things to the queue and processing them.
An interesting alternative to Starling is also presented within the comments on Glenn's post - RudeQ. RudeQ uses the same API as Starling but is ActiveRecord / database based, meaning there's no extra process to monitor. I suspect it's nowhere near as fast, but if you'd rather avoid the headache of monitoring another persistent process or don't have the option of having a persistent process at all (shared hosting, perhaps) it's worth checking out.
Post supported by Brightbox: Brightbox is a specialist European Rails hosting company. Each Brightbox server includes an optimised Ruby on Rails stack, SAN storage and access to a managed MySQL database cluster. They also manage dedicated clusters for large scale Rails deployments. Click here to learn more...