Want to stay on top? Ruby Weekly is a once-weekly e-mail newsletter covering the latest Ruby and Rails news.
     Feed Icon

Typhoeus: A High Speed, Parallel HTTP Library for Ruby

By Peter Cooper / May 13, 2009

typhon.jpgPaul Dix, of Feedzirra fame, strikes again! This time with Typhoeus (Github repo), a high-speed, parallel HTTP request library for Ruby. At first glance, you could be forgiven for wondering what the point is when we already have John Nunemaker's awesome HTTParty to build simple HTTP clients, but Typhoeus is, in many ways, like HTTParty on ten cans of Red Bull.

Typhoeus was built because Paul wanted to make numerous simultaneous HTTP requests at a time from his Ruby code. He didn't want any blocking and the whole setup had to be as fast as possible. To do this, he's interfaced with libcurl and libcurl-multi to pull off high speed concurrent HTTP requests and then laid a tidy Ruby API over the top.

Paul has some example code for you to check out, and while it's a little more verbose than HTTParty, the benefits available to you if you need to do high speed, concurrent requests make this a must-have for your Ruby repertoire. An interesting part of Paul's approach is that Typhoeus uses lazy evaluation to pull off the concurrency. When you build a request, the request isn't performed right away, as it would be with, say, Net::HTTP or HTTParty. Instead, a proxy object is maintained that then triggers the request at the very point that data is demanded from the client app.

Installation instructions, potential snafus, and usage information is covered on the official Github repo page.

Note: Paul plans to update Feedzirra to using Typhoeus in due course, but currently they use totally different HTTP request techniques.

Comments

  1. Peter Cooper says:

    Controversy! Matt Aimonetti has put together a benchmark that he claims shows net/http to be faster. Don't be too quick to judge though, there's a bit of a discussion on Twitter going on.. so watch this space :)

  2. Paul Dix says:

    Thanks for the coverage Peter! I've forked Matt's gist to show how it should be done to have Typhoeus make the calls in parallel.

    Also note that the end number output of how much slower or faster one is than the other is off. The 45s sleep throws that number out of wack.

  3. Peter Cooper says:

    Phew, that's good news! I was wondering what was going on for a minute there :)

  4. Paul Dix says:

    There's one other thing I forgot to mention. Typhoeus keeps a pool of libcurl easy handles to run requests. Warming the pool can take a little extra time. In the grand scheme of things this doesn't matter. For a benchmark, however, the pool should really be warmed before starting measurements.

  5. Ashley says:

    This is awesome, I love mythical greek gods with 100 fire breathing serpent heads! And combining that with the mighty Feedzirra will be even better...

    Rock on Paul! :D

  6. мeнимбaлaм says:

    Автор, посты у вас, конечно, очень интересные. Но вы не подумывали заменить дизайн?

Other Posts to Enjoy

Twitter Mentions