Until last week, Mark McGranaghan was a full-time Ruby developer for TechCrunch, the popular Web 2.0 news site. He worked extensively on CrunchBase, a wiki-style database of companies, people, and investors in the world of Web 2.0. His last assignment was the free-to-use CrunchBase API which went live last week.

I caught up with Mark to ask him about his time working at one of the world's most famous blogs with one of the world's most famous bloggers! You might be surprised to learn that head Cruncher Michael Arrington - who is never slow to point out Twitter's downtime - is pretty cool with Rails and what works well for his developers works for him.

Rails Inside: Hi Mark. Thanks for agreeing to this interview. First, how did you get started with Ruby and Rails?

Mark McGranaghan: I started with Ruby at the end of 2006 when I was a student at Bowdoin College. I stumbled on it randomly while I was exploring new programming languages and ended up using it a lot for writing programs for my discrete math class. At the time I also wanted to do some web programming but didn't like the web stacks offered by the other anguages that I knew - Java, Scheme, and Python - so I started looking into Rails. I got the AWDwR book, subscribed to a bunch of Rails blogs, and then just hacked my way from there.

How did you land the job with TechCrunch?

Henry [a co-developer] and I met Michael through a TechCrunch employee that we had known previously. When TechCrunch approached us about building out a structured wiki for the startup community, we were thrilled. I think that just communicating this excitement and showing that we understood why this was an important problem to solve went a long way in convincing Michael that we were right for the job. We also spoke to some seasoned developers that Michael knew; that conversation going well also gave Michael confidence in us. Michael was also very hands-off about our choice of Rails to implement the site - if it worked for us, it worked for him.

What was it like working at TechCrunch?

We had a lot of autonomy working on CrunchBase at TechCrunch. As the only full time developers, we had both the freedom of and responsibility to make all of the technical decisions. That put a lot of pressure on us, but it also allowed us to move quickly. We made a lot of good decisions, and when we didn't we had to quickly learn form our mistakes and recover.

CrunchBase existed for quite some time before you became involved with it, so what base did you have to work from?

CrunchBase previously existed as a PHP application. The site was very simple. There were only pages for companies, with no links to other entities, and search was implemented with MySQL LIKE queries. We scrapped the PHP code, but had to migrate the existing database. This was done with a 1000+LOC Ruby script that pulled everything out the old database, did a series of transformations in memory, and re-saved to a new database. It took us a long time to understand and normalize the data representations used by the old implementation, but it ended up OK. Being able to use ActiveRecord on both the new and the old tables helped a lot.

How is CrunchBase deployed? Does it scale well?

The CrunchBase Rails app is served by a single dedicated machine provided by MediaTemple. We have to contend with several scalability challenges. The first is the general traffic to CrunchBase.com; the site gets a lot of visitors because of its high Google rank and frequent TechCrunch links and the pages are expensive to generate. Fortunately, the site is very cacheable: most requests are served out of a view cache in a few milliseconds. We generally can't use page or action caching because different types of users see different version of each page - instead we use a custom caching mechanism that takes into account the authentication level of the user. Because of the interconnectedness of the CrunchBase data model, any individual piece of data may be cached in multiple webpages, widget objects, and API responses. When such a piece of data changes, all of the corresponding caches have to be expired.

Another major scalability concern is JavaScript and image requests from TechCrunch.com for the CrunchBase widget that we frequently embed on that site. We serve millions of these objects a day. Fortunately, we are able to serve almost all of them statically with lighttpd, which ends up being very cheap.

What was the motivation for the new CrunchBase API you just released?

We created the CrunchBase API because we believe it is in everyone's interest for this data to be free. It lets developers get convenient access to data [on Web 2.0 companies] and build interesting apps. It lets users find the data in new ways through these apps, and it gives us distribution and establishes us as the data hub. Also in the spirit of freedom, we've made the API as brain-dead-simple to use as possible; just visit an API url (example) and get a pretty-printed JSON output [Ed: This JSON pretty-printer has been open sourced by TechCrunch]. We have no registration, no API keys, no requests limits and no throttling.

Is TechCrunch a good place to work? Are other Rails apps in the works and are Ruby / Rails developers being hired?

TechCrunch is a very interesting place to work as a developer since its at the center of the startup/tech industry. A lot of news, ideas, and even people from the tech industry flow through our office, and you can't help but pick up a lot of energy from that. We now have an opportunity to leverage our position as a hub as well as the TechCrunch brand to create some really cool apps. One new Rails app that we just launched is TechCrunch Pitches, which is a site for users to submit and vote on 1-minute elevator pitch videos from startup founders. In order to support our existing properties and develop new ones we are currently hiring Ruby developers.

Now that you've left TechCrunch to return to school, will you continue to contribute to the Ruby community?

I've always liked doing independent work and research while I'm at school. This year I'm going to focus on open source: learning more from the existing projects and starting a few of my own. In particular I'd like to spend some time away from the Rails world - it's getting a little repetitive for me - and try some new things with Ruby and other languages. I've made a few steps in that direction by writing some components for a new Ruby web framework (github.com/mmcgrana). I'd like to bootstrap the framework by developing a few proof-of-concept open source apps against it. Other components of the Ruby/Rails stack that I'm growing frustrated with are RDoc, Rake, and Capistrano; I'll be looking closely at YARD, Thor, and Vlad over the next few months.

Thanks, Mark! Best of luck with your forthcoming ventures. Mark can be followed on Twitter, if you wish to do so. His username is mmcgrana.