Speed up Perlbrew with Test Parallelism

| 2 Comments

Steven Haryanto's Perl First World Problems #1 reminded me of something I've taken for granted lately.

You may have read my Controlling Test Parallelism with prove and Parallelism and Test Suites. I still have Test::Harness parallelism enabled by default on most of the machines where I install my own Perls. While I haven't yet filed tickets and tried to write patches for modules which need a little help to run tests in parallel, I've only found a few lately that need work. That's nice—having a module install through cpanm in five seconds is a lot better than ten seconds or more. (I like cpanm because it's fast and quiet, and part of its speed comes from not printing to the console.)

I like instant feedback.

Like Steven, I noticed quite a while that installing a custom Perl through perlbrew takes a while, but then I remembered that a lot of work went into the Perl 5 test suite to make tests run in parallel. (We did something similar with Parrot several years ago, and it changed the way I work forever.)

To run core tests in parallel, set the environment variable TEST_JOBS=n, where n depends on your computer. I use a value of 9 on a quad-core machine; in practice, that tends to keep the CPU busy while not blocking anything too long on IO. You can set it globally in your shell's configuration file or create an alias or wrapper for perlbrew.

As most of the time spent compiling and installing Perl 5 through perlbrew goes to running the test suite, this has saved me a measurable amount of time.

2 Comments

You can also pass "-j N" to perlbrew directly when building a new perl. This will actually cause it to run make in parallel, which affects more than just tests.

You can also pass "--notest" to both perlbrew and cpanm to skip tests, which is even faster than running them in parallel. :)

Honestly, with the vastness of the CPAN Testers network, it makes no sense for the vast majority of users to run Perl and CPAN tests on their own systems.

The default running of CPAN tests in particular causes great harm by causing failures up the dependency chain, failures that are meaningless to most users yet cause them to back away from installing modules with multiple dependencies such as Moose and Catalyst. It encourages module developers to reduce or eliminate dependencies, aka reinvent the wheel.

--notest is always set on my system, and I wish it was the default for perlbrew and cpanm.

Modern Perl: The Book

cover image for Modern Perl: the book

The best Perl Programmers read Modern Perl: The Book.

sponsored by the How to Make a Smoothie guide

Categories

Pages

About this Entry

This page contains a single entry by chromatic published on January 27, 2012 1:20 PM.

Avoiding The Vendor Perl Fad Diet was the previous entry in this blog.

Why I Run Tests on Install is the next entry in this blog.

Find recent content on the main index or look in the archives to find all content.


Powered by the Perl programming language

what is programming?