My buddy and sometimes collaborator Ovid has been publishing a series of articles about the test organization. In particular, today's Accidentally duplicating tests with Test::Class. I agree with 90% of what he wrote, and many of the organizations I know of would be better off if they followed his advice. (I won't tell you which 10% I disagree with, because it's subjective.)
It's always helpful to keep in mind someone's experience when judging his or her advice. At the BBC, Ovid worked on a large application with a large test suite which took far too long to run—multiple minutes. On a recent contract, he and I worked with a few other really good developers on a smaller application with a test suite which took about 90 seconds to run.
That's longer than I wanted. 30 seconds is about as much time I want to spend waiting for a parallel test suite to run, but 90 seconds was workable. It wasn't a crisis.
We did things wrong, according to Ovid's article; we used Test::Class with inheritance and we used one driver file per module file being tested. Like most test suites, the organization of our tests grew over time. Unlike many test suites, our organization improved gradually as we spent time refining and revising it.
We had two major causes of slowness in the test suite. One was the use of PostgreSQL transactions and savepoints to ensure data isolation at the database level. Some of the prevailing so-called wisdom in the test community is to use pervasive mock objects to avoid using a real database. That's not just bad advice, it's stupid advice. This application depended on real data with real characteristics and we unashamedly took advantage of real database features like transactions and triggers and foreign keys to ensure consistency in the database. Emulating all of those features in mock objects is silly busywork that does no one any good.
The other cause of slowness was in the application's design. The initial implementation of one critical business feature made one specific database usage pattern very slow. It was only a problem in practice in the test suite, and it wasn't that bad. (The contract ended before it was worth addressing it in general.)
Not every application follows this model. Not every development team has multiple experienced testers who deliberately make time to refine and revise test suites to make them effective and fast. (I always find that there's a balance between making the most beautiful and speedy test suite possible and getting enough coverage of the most important features to allow development to continue at its necessary pace. The article What "Worse is Better vs The Right Thing" is really about is a good summary of my thinking.)
With that all said, I'm not sure Ovid's approach would have helped this application very much. (It would have made a fine experiment for a couple of hours one afternoon, and if it worked, so much the better!)
I like using Test::Class to test reusable OO components. (Test::Routine is another good approach.) I don't always like the syntax, at least in the modern world of Moose, but it does the job. I don't like the explosion of little test runner files.
However, I do like two features of this approach:
- The ability to run an individual class's tests apart from the entire suite
- The knowledge that each test's environment is isolated at the process level
Both are laziness on my part, and that may be where I disagree with Ovid. I'm far too lazy to want to write:
$ prove -l t/test_class_runner.t -- --class=MyApp::Collection::PastryShops... even though it's really not all that much to type.
Second—and this is more important—I like the laziness of knowing that each individual test class I write will run in its own process. No failing test in another class will modify the environment of my test class. No garbage left in the database will be around in my process's view of the database. Maybe that's laziness on my part for not writing copious amounts of cleanup code to recover from possible failures in tests, but it is what it is.
Like I said at the start, however, I agree with Ovid 90% of the time. You'll get a lot of value out of reading and thinking about and adopting what he says. Yet don't assume that the right answer to speeding up your tests is always to switch to a new test runner or test execution model. Sometimes a slow test suite is a sign that you need to profile your application.
Keep in mind that most of my experience writing and maintaining and optimizing test suites has shown that startup time—loading the application and compiling all of the relevant Moose classes—is insignificant to doing real work in the test suites. If you're working with an application that's poorly factored or that's doing a lot more work to start up than it probably should, you'll have different results. (I'm not above preventing the loading of Chart::Clicker, for example, when I don't need it to run any tests.)
Testing is difficult, but worthwhile. Testing in Perl offers you the benefit of great test tools written by a community of smart, experienced people who are willing to experiment to make better tools. Take advantage of that.
