A Test-Infected Culture

Many people consider the CPAN the shining gem of Perl 5. As of this writing, it contains more than 53,000 modules in more than 17,400 distributions with 7316 registered authors. That's a lot of code.

Many people outside of the Perl community don't realize the implications for a centralized infrastructure which allows people to upload, install, read documentation, annotate, report bugs, request features, submit patches, and collaborate on improvements to every one of these distributions. The CPAN isn't just a repository of freely-installable libraries. It's the hub which makes so many of modern Perl's amenities possible.

First, a brief history lesson.

1987

Larry released Perl 1 in 1987. If you download Perl 1 source code (you can build it today), you'll see that it contained a rudimentary but working core test suite. The separation of concerns evident between running tests and interpreting the results is still useful today in the form of the Test Anything Protocol.

1994

Perl versions 2 through 5 continued the language testing culture. The core language had a self-hosted test suite (and still does). Many CPAN modules at that time used a program called h2xs to generate that boilerplate -- and part of the boilerplate was a program called test.pl which contained some "scary black magic -- do not edit" to set up a testing infrastructure.

1998

In 1998, Chris Nandor and Graham Barr created a CPAN Testers project to run the test suites of CPAN distributions and report their results. Remember this.

The 21st Century Testing Revolution

After the Perl 6 announcement in 2000, Michael Schwern agreed to lead the QA working group. He resolved to improve the coverage of the core test suite and the coverage of the core libraries (which was less than optimal). As part of this process, he wrote Test::More and Test::Simple to hide the black magic from users and to make tests easier to write.

This started a revolution, but the world changed when he refused to add features to Test::More, claiming that it was more important for people to write their own testing modules. At his suggestion, I extracted the reusable logic from Test::More and created Test::Builder, which for the first time allowed multiple test modules to run in the same process and collaborate to produce TAP.

The revolution had begun.

Perl QA spent the next couple of years teaching people how to write good and comprehensive and working and useful test suites. We quadrupled the number of tests in the Perl 5 core from the release of Perl 5.6.0 to 5.8.0, and that number has only increased.

Can Your Language Community Do This?

That brings us to the present day. The CPAN Testers project is almost eleven years old. According to the monthly CPAN Testers statistics, there have been over three and a half million reports posted. Activity was low until late 2006, when the number of test reports submitted exploded (see CPAN Testers Statistics Graphs). Though 3.5 million reports over eleven years doesn't sound like a lot, the current testing rate produces well over a million reports a year (perhaps closer to two).

There are even plugins for the CPAN and CPANPLUS distribution installation libraries to report any test failures when you try to install a distribution -- testing those distributions you need the most.

Anyone who uploads a distribution to the CPAN will have it tested on several hardware platforms, on multiple versions of Perl, and on multiple operating systems. These reports are gathered and aggregated and displayed on the CPAN Search page for that distribution. A related project also analyze the probability of a given distribution installing on your platform (taking into account the reports for all of its dependencies, recursively).

Imagine knowing that your code works on platforms you can't access, because people who know those platforms have set them up to give you frequent and accurate feedback.

The Perl 5 language itself has daily smoke reports (though not produced through CPAN Testers), as does the Parrot virtual machine and the Rakudo Perl 6 implementation. All of the infrastructure and experience from testing Perl 5 effectively is going into the specification tests for Perl 6.

Granted, the usefulness of all of this infrastructure depends on people using it and writing good tests, but I know of no other language community which has even attempted a project of this size. Do you?