In Making CI easier to do than not to with Hudson CI and Vagrant, Dr Nic Williams (of EngineYard) presents a walkthrough of setting up a VM-based continuous integration system in order to reliably test your apps across multiple setups:

It irked me a little that I could develop on one stack (OS X, Rubinius, Sqlite3), run continuous integration (CI) on another stack (Ubuntu, Ruby 1.8.7, Postgresql), and deploy into another stack (Gentoo, Ruby 1.9.2, MySQL). I think what irks and worries me is that there are three sets of differences to be aware of. A bug in production? Was it a missing test scenario or one of the many differences between production and CI environments?

[A solution is to] use a VM that matches the production environment. Each different production environment would mean another VM. If you are managing your own production environment, then all you need is the tools (described in this article) to recreate your production environment in a VM.

Dr Nic

It's typically good stuff from everyone's favorite doctor of Ruby.