Organizing Test Suites with Test::Class

When working with large test suites, using procedural tests for object-oriented code becomes clumsy after a while. This is where Test::Class really shines. Unfortunately, many programmers struggle to learn this module or don't use its full power.

Please note that article assumes a basic familiarity with object-oriented Perl and testing. Also, some of these classes are not "proper" by the standards of many OO programmers (your author included), but have been written for clarity rather than purity.

Modules and Their Versions

This article uses the following modules and versions:

• Test::Class version 0.31
• Test::Most version 0.21
• Test::Harness version 3.15
• Moose version 0.7
• Class::Data::Inheritable version 0.08

You may use lower versions of these modules (and write the OO by hand instead of using Moose), but be aware that you may see slightly different behavior.

Notes about the code

Note that Moose packages should generally end with:

  __PACKAGE__->meta->make_immutable;
  no Moose;

I've omitted this from the examples. I've also omitted use strict and use warnings, but assume they are there (they're automatically used when you use Moose). The code will, however, run just fine without this. I did this merely to focus on the core features of the code in question.

Of course, you may need to adjust the shebang line (#!/usr/bin/env perl -T) for your system.

Evolution of a Perl Programmer

There are many paths programmers take in their development, but a typical one seems to be:

1. Start writing simple procedural programs.
2. Start writing modules to reuse code.
3. Start using objects for more powerful abstractions.
4. Start writing tests.

While it would be nice if people started writing tests from day 1, most programmers don't. When they do, they're often straight-forward procedural tests like:

     #!/usr/bin/env perl -T

     use strict;
     use warnings;

     use Test::More tests => 3;

     use_ok 'List::Util', 'sum' or die;

     ok defined &sum, 'sum() should be exported to our namespace';
     is sum(1,2,3), 6, '... and it should sum lists correctly';

There's nothing wrong with procedural tests. They're great for non-OO code. For most projects, they handle everything you need to do. If you download most modules off the CPAN you'll generally find their tests -- if they have them -- procedural in style. However, when you start to work with larger code bases, a t/ directory with 317 test scripts starts to get tedious. Where is the test you need? Trying to memorize all of your test names and grepping through your tests to find out which ones test the code you're working with becomes tedious. That's where Adrian Howard's Test::Class can help.

Using Test::Class

Creating a simple test class

I'm a huge "dive right in" fan, so I'll now skip a lot of the theory and show how things work. Though I often use test-driven development (TDD), I'll reverse the process here to show explicitly what I'm testing. Also, Test::Class has quite a number of different features, not all of which I'm going to explain here. See the documentation for more information.

First, create a very simple Person class. Because I don't like writing out simple methods over and over, I used Moose to automate a lot of the grunt work.

     package Person;

     use Moose;

     has first_name => ( is => 'rw', isa => 'Str' );
     has last_name  => ( is => 'rw', isa => 'Str' );

     sub full_name {
         my $self = shift;
         return $self->first_name . ' ' . $self->last_name;
     }

     1;

This provides a constructor and first_name, last_name, and full_name methods.

Now write a simple Test::Class program for it. The first bit of work is to find a place to put the tests. To avoid namespace collisions, choose your package name carefully. I like prepending my test classes with Test:: to ensure that we have no ambiguity. In this case, I've put my Test::Class tests in t/tests/ and named this first class Test::Person. Assume the directory structure:

     lib/
     lib/Person.pm
     t/
     t/tests/
     t/tests/Test
     t/tests/Test/Person.pm

The actual test class might start out like:

     package Test::Person;

     use Test::Most;
     use base 'Test::Class';

     sub class { 'Person' }

     sub startup : Tests(startup => 1) {
         my $test = shift;
         use_ok $test->class;
     }

     sub constructor : Tests(3) {
         my $test  = shift;
         my $class = $test->class;
         can_ok $class, 'new';
         ok my $person = $class->new,
             '... and the constructor should succeed';
         isa_ok $person, $class, '... and the object it returns';
     }

     1;

Note: this code uses Test::Most instead of Test::More to take advantage of Test::Most features later. Also, those methods should really be ro (read-only) because the code makes it possible to leave the object in an inconsistent state. This is part of what I meant about "proper" OO code, but again, I wrote this code for illustration purposes only.

Before I explain all of that, run this test. Add this program as t/run.t:

     #!/usr/bin/env perl -T

     use lib 't/tests';
     use Test::Person;

     Test::Class->runtests;

This little program sets the path to the test classes, loads them, and runs the tests. Now you can run that with the prove utility:

     $ prove -lv --merge t/run.t

Tip: The --merge tells prove to merge STDOUT and STDERR. This avoids synchronization problems that happen when STDERR is not always output in synchronization with STDOUT. Don't use this unless you're running your tests in verbose mode; it sends failure diagnostics to STDOUT. TAP::Harness discards STDOUT lines beginning with # unless running in verbose mode.

You will see output similar to:

     t/run.t ..
     1..4
     ok 1 - use Person;
     #
     # Test::Person->constructor
     ok 2 - Person->can('new')
     ok 3 - ... and the constructor should succeed
     ok 4 - ... and the object it returns isa Person
     ok
     All tests successful.
     Files=1, Tests=4,  0 wallclock secs ( 0.03 usr  0.00 sys +  0.43 cusr  0.02 csys =  0.48 CPU)
     Result: PASS

Note that the test output (named the "Test Anything Protocol", or "TAP", if you're curious) for the constructor method begins with the diagnostic line:

     # Test::Person->constructor

That occurs before every test method's output and makes it very easy to find which tests failed.

Look more closely at the test file to see what's happening:

     01: package Test::Person;
     02:
     03: use Test::Most;
     04: use base 'Test::Class';
     05:
     06: sub class { 'Person' }
     07:
     08: sub startup : Tests(startup => 1) {
     09:     my $test = shift;
     10:     use_ok $test->class;
     11: }
     12:
     13: sub constructor : Tests(3) {
     14:     my $test  = shift;
     15:     my $class = $test->class;
     16:     can_ok $class, 'new';
     17:     ok my $person = $class->new,
     18:         '... and the constructor should succeed';
     19:     isa_ok $person, $class, '... and the object it returns';
     20: }
     21:
     22: 1;

Lines 1 through 4 are straightforward. Line 4 makes this class inherit from Test::Class; and that's what makes all of this work. Line 6 defines a class method which the tests will use to know which class they're testing. It's very important to do this rather than hard-coding the class name in our test methods. That's good OO practice in general; it will help you later.

The startup method has an attribute, Tests with has the arguments startup and 1. Any method labeled as a startup method will run once before any of the other methods run. The 1 (one) in the attribute says "this method runs one test". If you don't run any tests in your startup method, omit this number:

     sub load_db : Tests(startup) {
         my $test = shift;
         $test->_create_database;
     }

     sub _create_database {
         ...
     }

Tip: as you can see from the code above, you don't need to name the startup method startup. I recommend you give it the same name as the attribute for reasons discussed later.

That will run once and only once for each test class. Because the _create_database method has no have any attributes, you may safely call it and Test::Class will not try to run it as a test.

Of course, there's a corresponding shutdown available:

     sub shutdown_db : Tests(shutdown) {
         my $test = shift;
         $test->_shutdown_database;
     }

These two attributes allow you to set up and tear down a pristine testing environment for every test class without worrying that other test classes will interfere with the current tests. Of course, this means that tests may not be able to run in parallel. Though there are ways around that, they're beyond the scope of this article.

As mentioned, the startup method has a second argument which tells Test::Class that it runs one test. This is strictly optional. Here we use it to safely test that we can load our Person class. As an added feature, if Test::Class detects that the startup test failed (or if it catches an exception), it assumes that there's no point in running the rest of the tests, so it skips the remaining tests for the class.

Tip: Don't run tests in your startup method; I'm doing so only to simplify this example. I'll explain why in a bit. For now, it's better to write:

     sub startup : Tests(startup) {
         my $test  = shift;
         my $class = $test->class;
         eval "use $class";
         die $@ if $@;
     }

Take a closer look at the constructor method.

     13: sub constructor : Tests(3) {
     14:     my $test  = shift;
     15:     my $class = $test->class;
     16:     can_ok $class, 'new';
     17:     ok my $person = $class->new,
     18:         '... and the constructor should succeed';
     19:     isa_ok $person, $class, '... and the object it returns';
     20: }

Tip: I did not name the constructor tests new because that's a Test::Class method and overriding it will cause the tests to break.

The Tests attribute lists the number of tests as 3. If you don't know how many tests you're going to have, use no_plan.

     sub constructor : Tests(no_plan) { ... }

As a short-cut, omitting arguments to the attribute will also mean no_plan:

     sub constructor : Tests { ... }

The my $test = shift line is equivalent to my $self = shift. I've like to rename $self to $test in my test classes, but that's merely a matter of personal preference. The $test object is an empty hashref. This allows you to stash data there, if needed. For example:

     sub startup : Tests(startup) {
         my $test     = shift;
         my $pid      = $test->_start_process
             or die "Could not start process: $?";

         $test->{pid} = $pid;
     }

     sub run : Tests(no_plan) {
         my $test    = shift;
         my $process = $test->_get_process($test->{pid});
         ...
     }

The rest of the test method is self-explanatory if you're familiar with Test::More.

The test class also had first_name, last_name, and full_name, so write those tests. When you're in "development mode", it's safe to leave these tests as no_plan, but don't forget to set the number of tests when you're done.

     sub first_name : Tests {
         my $test   = shift;
         my $person = $test->class->new;

         can_ok $person, 'first_name';
         ok !defined $person->first_name,
           '... and first_name should start out undefined';

         $person->first_name('John');
         is $person->first_name, 'John',
           '... and setting its value should succeed';
     }

     sub last_name : Tests {
         my $test   = shift;
         my $person = $test->class->new;

         can_ok $person, 'last_name';
         ok !defined $person->last_name,
           '... and last_name should start out undefined';

         $person->last_name('Public');
         is $person->last_name, 'Public',
           '... and setting its value should succeed';
     }

     sub full_name : Tests {
         my $test   = shift;
         my $person = $test->class->new;

         can_ok $person, 'full_name';
         ok !defined $person->full_name,
           '... and full_name should start out undefined';

         $person->first_name('John');
         $person->last_name('Public');

         is $person->full_name, 'John Public',
           '... and setting its value should succeed';
     }

Tip: when possible, name your test methods after the method they're testing. This makes finding them much easier. You can even write editor tools to automatically jump to them. Not all test methods will fit this pattern, but many will.

The first_name and last_name tests can probably have common elements factored out, but for now they're fine. Now see what happens when you run this (warnings omitted):

     t/run.t ..
     ok 1 - use Person;
     #
     # Test::Person->constructor
     ok 2 - Person->can('new')
     ok 3 - ... and the constructor should succeed
     ok 4 - ... and the object it returns isa Person
     #
     # Test::Person->first_name
     ok 5 - Person->can('first_name')
     ok 6 - ... and first_name should start out undefined
     ok 7 - ... and setting its value should succeed
     #
     # Test::Person->full_name
     ok 8 - Person->can('full_name')
     not ok 9 - ... and full_name should start out undefined

     #   Failed test '... and full_name should start out undefined'
     #   at t/tests/Test/Person.pm line 48.
     #   (in Test::Person->full_name)
     ok 10 - ... and setting its value should succeed
     #
     # Test::Person->last_name
     ok 11 - Person->can('last_name')
     ok 12 - ... and last_name should start out undefined
     ok 13 - ... and setting its value should succeed
     1..13
     # Looks like you failed 1 test of 13.
     Dubious, test returned 1 (wstat 256, 0x100)
     Failed 1/13 subtests

     Test Summary Report
     -------------------
     t/run.t (Wstat: 256 Tests: 13 Failed: 1)
       Failed test:  9
       Non-zero exit status: 1
     Files=1, Tests=13,  0 wallclock secs ( 0.03 usr  0.00 sys +  0.42 cusr  0.02 csys =  0.47 CPU)
     Result: FAIL

Uh oh. You can see that full_name isn't behaving the way the tests expect. Suppose that you want to croak if either the first or last name is not set. To keep this simple, assume that neither first_name nor last_name may be set to a false value.

     sub full_name {
         my $self = shift;

         unless ( $self->first_name && $self->last_name ) {
             Carp::croak("Both first and last names must be set");
         }

         return $self->first_name . ' ' . $self->last_name;
     }

That should be pretty clear. Look at the new test now. Use the throws_ok test from Test::Exception to test the Carp::croak(). Using Test::Most instead of Test::More makes this test function available without explicitly using Test::Exception.

 sub full_name : Tests(no_plan) {
     my $test   = shift;
     my $person = $test->class->new;
     can_ok $person, 'full_name';

     throws_ok { $person->full_name }
         qr/^Both first and last names must be set/,
         '... and full_name() should croak() if the either name is not set';

     $person->first_name('John');

     throws_ok { $person->full_name }
         qr/^Both first and last names must be set/,
         '... and full_name() should croak() if the either name is not set';

     $person->last_name('Public');
     is $person->full_name, 'John Public',
       '... and setting its value should succeed';
 }

Now all of the tests pass and you can go back and set the test plan numbers, if desired:

 All tests successful.
 Files=1, Tests=14,  0 wallclock secs ( 0.03 usr  0.00 sys +  0.47 cusr  0.02 csys =  0.52 CPU)
 Result: PASS

The next article, Reusing Test Code with Test::Class shows how to inherit from test classes -- and how to refactor test classes!