Heckle: Tortures Your Tests For Revealing Confessions
Heckle is a great new library from Kevin Clark (though Ryan Davis wrote a proof of concept at RubyConf) that 'torments your tests'. It uses ParseRuby and RubyToRuby to rip your code apart and forces random data into your code (currently it does this for strings, symbols, regexps, ranges, booleans, and numbers) to see how good your tests really are.
Simply, it's fuzz testing for your Ruby tests, and that's a good thing if you want your test suites to be strong and far-reaching.
December 21, 2006 at 3:40 am
Does Heckle have any usefulness if you don't strive for 100% testing coverage?
December 21, 2006 at 5:53 am
I dare say it'd be an interesting test of code robustness and/or ability to sanitize input. This could be important on, say, a library that interacts with remote services.
December 22, 2006 at 6:00 am
If you don't strive for 100% testing coverage, then probably not.
If you even have to think about testing coverage as something you strive for, then probably not.
Heckle is designed for Agile shops, where testing coverage is a by-product of the test-first development process. When TDD gets in your blood, you read comments like the one above and wonder how programmers can be writing code without first writing a test to prove that they need to fix something.
4gauge.blogspot.com
December 28, 2006 at 7:50 pm
Some corrections: It uses ParseTree, not ParseRuby; it is a tool, not a library (you invoke it on the command-line); more important (imo) than mutating literals, it also mutates all branching/looping logic; and while I wrote the proof of concept and Kevin did most of the work since then, I'm still working on it.
December 28, 2006 at 7:51 pm
I should add, YES, heckle is still useful when you're not pushing for 100% test coverage (although, I must ask--why not?). You can invoke heckle on a single method to make sure that it and only it are heckle-proof.
December 29, 2006 at 1:40 am
I brought up the testing coverage question on Aslak Hellesoy's blog as well: http://blog.aslakhellesoy.com/articles/2006/12/19/heckling-with-rspec
The question of 100% test coverage was first brought to my attention by Jonathan Conway http://www.noodlesinmysandals.com/2006/11/13/so-you-think-you-re-agile
But I have to agree, if coding with a BDD or TDD mentality, shouldn't 100% test coverage be built in? Perhaps Jonathan's argument implies that testing coverage drops below 100% after refactoring?