What is a "Stable" Core Anyway? The Dual-Lived Problem.

A recent discussion on the p5p mailing list seemed like an opportunity for me to suggest what could help produce regular Perl 5 releases. (Nicholas Clark nearly dared me. He's a bad man.)

As you might expect, the resulting discussion went off in several directions. A few people see little value in regular releases (which means that I should write about that soon). Perhaps the most important point is a question that a couple of pumpkings alluded to. David Golden, ever perceptive, picked up on the real question.

How do you know that Perl 5 is "stable" and ready for release?

Double Lives Take Half as Long

(I addressed this issue from the other side in February in Hanging the Core out to DRY. This is a followup from the "Why does so much time elapse between stable Perl 5 releases?" department.)

One of the biggest time sinks to releasing a new stable version of Perl is chasing down dual-lived core modules. A dual-lived core module is a module in Perl's core library that someone other than p5p maintains outside of the Perl 5 core repository. At various times, someone notices that there's a new release of a core module (usually the maintainer, but...) and suggests that a pumpking merge the new version into the core.

Why are there dual-lived modules?

The most important reason is so that users can upgrade to newer versions of core modules without upgrading the core. Why would they want to do that? Perhaps it's difficult to upgrade core Perl and they don't have permission. Most likely, it's because Perl 5 releases are infrequent. Where a bugfix in a core module may require only a little bit of testing and represent only a little bit of change, releasing a new version of Perl requires a lot of testing and coordination.

Another reason for dual-lived modules is to spread the maintenance burden. While only a few people had commit access to the Perl 5 repository in the past, every maintainer of a dual-lived module can have his or her own repository just for that module. This is less of an issue now that Perl 5 uses git, but old design decisions tend to persist.

A final reason for dual-lived modules is that the modules themselves may be usable on several versions of Perl, even if they're not binary compatible. (The modules may be pure Perl, or their XS may be source compatible with multiple Perl major versions.) If code works correctly and unmodified on Perl 5.6.2 and Perl 5.8.6 - 5.8.9 and Perl 5.10, is there a reason to limit users to waiting for Perl 5.10.1 or 5.12 to use the new version?

Of course, coordinating separate releases from all of these separate repositories and separate authors on a well-known and well-understood time frame and testing them to make sure they work with the core language at that point as well as with each other -- and remember there are dozens (if not more) dual-lived modules spread out among dozens of authors -- is difficult.

As I'm sure you realize by now, this is a tangle.

Cutting the Knot

These separate goals of dual-lived modules are incompatible.

The point of having a standard library full of modules is so that Perl is useful to users by itself, with nothing else installed. This theory suggests that users shouldn't have to install CPAN modules to do useful things.

The point of releasing modules on the CPAN is so that users can upgrade them independently of the core. The CPAN is an integral part of the modern Perl programming experience. (When users complain that they have to install "half of CPAN" to get a modern Perl application to run, part of that complaint is that Perl 5 the language is too minimalistic and flexible and not opinionated in several places to write modern Perl applications on its own. Another part of that complaint is that we don't have CPAN quite right yet.)

I can think of three solutions. They can overlap, but they're also independent.

First, improve the core's automated testing. This helps everyone; it can identify changes in the core code that affect the stability and behavior of the standard library. It can also identify changes in standard library modules which do not work on important platforms.

The sooner you can discover a failure, the easier it is to identify the offending change.

Second, do not automatically merge in all upstream changes to the core. A stable and well-tested version of a dual-lived module is better than the most recent version, at least if you haven't verified it on all interesting platforms. If users want a newer version, they can upgrade themselves. (That's one of the reasons dual-lived modules exist!)

I realize this seems to conflict with my belief that users should upgrade frequently. I still believe they should. I still believe strongly in frequent stable releases -- but users do not have to upgrade. They should have the option to upgrade frequently (and they should know the support and deprecation implications of not upgrading), but they have the choice.

Third, remove dual-lived modules from the core. Maintain and distribute only those modules necessary to install other modules.

This will expose several other problems, however: encouraging Perl distributions to produce stable bundles which represent a usable core, improving configuration and installation support, and overcoming the social inertia toward an unchanging Perl core.

This is probably a historical inevitability, however; the maintenance burden of the Perl 5 core cannot increase indefinitely. Perl 6 has chosen option three to avoid this maintenance burden. (Many design and management decisions in Perl 6 address Perl 5's drawbacks.)