Why Test::Harness Should Not Enable Global Warnings

Test::Tutorial needed some attention. I've referred to it from various places in the decade-plus of its existence, but when I mentioned it most recently in the new edition of the Modern Perl book, I looked at it again.

The tutorial holds up pretty well for its age, but it didn't mention done_testing, which is a great solution to the "Should my test file declare a plan?" In short, you have three options for figuring out if your test program ran to completion: count all of the tests you expect and predeclare that number, skip it all and hope, or call done_testing at the normal exit point of your program. If done_testing runs, the test run succeeds. If done_testing doesn't run, an exception or other abnormal exit interrupted the test run, and the test run fails.

Along the way, I took the opportunity to clean up a few other things. In particular, I removed the -w flag from the hash-bang lines at the start of the test programs. (A flag enabling global behavior? In a test file? In 2011? More than a decade after the introduction of the perfectly cromulent lexical warnings implementation? DELETE!)

Schwern objected, as is his right, giving four reasons Perl tests should run with -w. In particular, he believes that it is the responsibility of the consumer of a dependency to handle warnings in dependencies.

I can't agree.

Consider the possible cases where a dependency may produce a warning.

package MayWarn
{
    sub no_warning
    {
        my $foo;
        return "<$foo>";
    }

    {
        use warnings;
        sub lexical_warning
        {
            my $foo;
            return "<$foo>";
        }
        {
            no warnings 'uninitialized';

            sub disabled_warning
            {
                my $foo;
                return "<$foo>";
            }
        }
    }
    1;
}

It's pretty clear that interpolating an undefined value into a string will cause a "Use of uninitialized value" warning in two circumstances: where global warnings are enabled and where lexical warnings are enabled.

No Lexical Warnings

Compare calling no_warning with and without -w:

$ perl -Ilib -MMayWarn -e 'MayWarn::no_warning()'

$ perl -Ilib -MMayWarn -w -e 'MayWarn::no_warning()'
Use of uninitialized value $foo in concatenation (.) or string at lib/MayWarn.pm line 6.

In this case, -w affects the behavior of code written without regard for lexical warnings.

Lexical Warnings Enabled

Compare calling lexical_warning with and without the -w flag:

$ perl -Ilib -MMayWarn -e 'MayWarn::lexical_warning()'
Use of uninitialized value $foo in concatenation (.) or string at lib/MayWarn.pm line 14.
$ perl -Ilib -MMayWarn -w -e 'MayWarn::lexical_warning()'
Use of uninitialized value $foo in concatenation (.) or string at lib/MayWarn.pm line 14.

The -w flag is irrelevant to the reporting of warnings, because lexical warnings are always in effect.

Lexical Warnings Enabled, Uninitialized Warnings Disabled

Compare calling disabled_warning with and without the -w flag:

$ perl -Ilib -MMayWarn -e 'MayWarn::disabled_warning()'

$ perl -Ilib -MMayWarn -w -e 'MayWarn::disabled_warning()'

The -w flag is again irrelevant, because lexical behavior overrides global behavior.

-w in Test Suites

I see the point that the rightest thing to do is to track down every CPAN module which isn't warnings-clean and try to convince the authors to Do The Right Thing, but isn't blindly applying global behavior in the hopes of finding and convincing other people to fixing bugs the same argument so many people had against UNIVERSAL::isa? (Just consider how much buggy code is still out there, years into that argument.)

Furthermore, modules which use lexical warnings properly (code written to at least Perl 5.6 standards—a March 2000 level of modernity—receive no benefit from -w. That flag is irrelevant.

Code written to the previous millennium's standards does offer the possibility of benefit from -w applied...

... but I'm not going to be the one arguing that it's sensible for Test::Harness to magically add flags I didn't ask for, and that that obligates other authors to modify their code.

Maybe I'm wrong, and maybe the desire to improve the CPAN trumps my desire to be able to reason about what my code does (and to, you know, use features lexically as I see fit), but it seems to me that the correct place to apply the big hammer of All CPAN Code Should be Free of Warnings, Globally (But Thou Canst Use Lexical Warnings, You Philistine) should be in the CPAN testers, which can test things in and of themselves.

Even so, I can't justify blindly injecting global behavior into what should be protected lexical scopes. Test::Harness doesn't have enough information to decide why dependencies written without use warnings; or no warnings; lacks either pragma. Perhaps the author doesn't know about it. Perhaps the author doesn't care. Perhaps the code is free of warnings, but the author distributes the code without the pragmas enabled to save time or memory (yeah it's probably misguided, but it happens). Further, Test::Harness can't know whether any warnings represent actually dubious code, or whether they're useless. (A handful of experienced and respected and intelligent Perl 5 programmers have argued that the uninitialized value warning does more harm than good, and sometimes I agree.)

What do you think?