(Or "Why the solution has been obvious from the start.")
If my hypothesis is correct, that Perl 5's current development process is unsustainable, in part because synchronizing dual-lived Perl 5 modules with the core is a fool's game, there's one obvious way to get Perl 5 on regular release schedule and add important features that just about every other serious programming language supports.
Most of the pieces are already in place.
The success of the CPAN and its ecosystem already demonstrates that this strategy can work, as does the presence of virtual dependencies in packaging systems for free operating system distributions.
The idea isn't new at all, but you may not have heard it expressed this way: to solve (some of) Perl 5's maintenance problems with coordinating the stable release schedule of dozens of modules and distributions maintained outside of the core, stop coordinating the stable release schedules of dozens of modules and distributions maintained outside of the core.
Remove dual-lived modules from the core. (Update: Leave those modules necessary to bootstrap a full CPAN installer, of course.)
What replaces them?
If the underling premise of modern Perl and enlightened Perl is correct -- that the world's best and most effective Perl programmers take full advantage of the CPAN to make up for missing language features, to improve their productivity, and because solving a problem once and for all and sharing it is the ultimate expression of laziness, impatience, and hubris -- then the problem in the Perl world is that we don't do this enough.
To bring up a repeated example, releasing software regularly can be difficult. The best way I know to make it easy to release software every month is to release it every month. If it's difficult to release your software every month, you have a strong motivation to make it easy.
The Perl world hasn't made it sufficiently easy to find, install, maintain, and upgrade non-core code. Projects such as local::lib, CP5.6.2AN, and CPAN Testers go a long way. They don't provide everything, but the tools and knowledge are present to add the next few necessary features.
Please note that all of these features depend on CPAN remaining an uploading service and a mirroring/distribution service. CPAN itself doesn't need to change. CPAN works just fine for these purposes: especially as its simplicity has allowed a solar system of related services and tools to form around the shining star that is the CPAN distribution service.
The benefit of removing modules from the core are simple and compelling:
- There is less code for p5p to maintain.
- Releases of modules can occur on their own schedules; releases of the core can occur on its own schedule.
- Bug reports, feature requests, and patches will not as easily get lost in the cracks between the maintainer's preferred communications medium and the p5p list.
- Crufty old modules superseded long ago by something more decent will lose whatever patina they had from their "Of course they're core modules! This means we should use them!" status.
- Places of work which refuse to vet or install non-core modules will be revealed as the soul-sucking portals of hate that they are and all right-minded people will ignore them.
(The final reason is conjecture, but very satisfying to imagine.)
For this to work, the Perl community must:
- Produce a graph of modules and their dependencies in a form consumable by other services. David Cantrell's CPAN Deps (link not provided in deference to his bandwidth and CPU resources) is a great start.
- Exploit the CPAN testing reports to identify potential changes in leaf nodes of the graph due to upedge changes.
- Revise the testing strategy (and its reporting) to alert upedge authors of the effects of changes on downedge authors and vice versa.
- Publish metadata on the best-possible recommendation of versions for modules along a graph edge. That is, if you want to install Catalyst on FreeBSD 7.0, you may want Scalar::Util x.y, Test::Simple x.y, and Moose x.y.
- Given this metadata, provide a single, installable (source) tarball which bundles all of the nodes of a graph edge.
This is all feasible, if the Perl community has the will to do so.
Some may object that the final two steps may occasionally fail. That's right. You can't solve that problem -- not completely. You can try, but you will fail. That doesn't mean you shouldn't try. It only means that you're a fool for designing a process that can't cope with the occasional upgrade glitch.
Others may object that this does not address the education or the advocacy problem of teaching Perl novices to look for these bundles or system administrations that they're hateful little trolls unless they support these bundles on their system. Nor does it address the technical problem that occasionally plagues installing CPAN modules. Those are solutions for a different post.
One final objection is "What's the point?" or "The Perl 5 release schedule works for me!" or "But you ought to support these bundles for eight to ten years." Those are fine opinions, if wrong, for one simple reason.
The right way to write better software is never to go more slowly.
I do love this whole concept and I do wish it would happen. What are the chances of it really happening though? I fear the "will" isn't there. I hope I am wrong.
"Remove dual-lived modules from the core."
That's a nice idea, but it is completely infeasible. Removing dual-lived modules means you cannot build and test Perl. It also means you do not have the toolchain installed to install modules from CPAN which makes the premise of pulling in everything users need from CPAN difficult, if not impossible.
The thought that dual-lived modules adds to our maintenance is also is a fallacy. Being the primary fool that's played the "fool's game" over past several years, I can tell you that spending time in the Perl bug queue and reviewing and applying patches takes a whole lot more time than dealing with the dual-lived modules. What actually takes up most of my time is testing Perl with an ever expanding roster of CPAN modules.
Another way of looking at this is that p5p indirectly supports all of CPAN with every release. This is a responsibility that is not taken lightly. We've caused breakages to DBI, modperl, and a whole host of modules, but most people wouldn't know this because this hasn't happened with a Perl release. Breaking these modules or any other commonly used modules with a Perl release would be seen as a decrease in stability and is not acceptable. Speeding up releases and allowing them to break CPAN is not an allowable option.
What Perl needs more than more releases is more people doing useful things. Windows programming experts are always helpful. Fixing bugs and tackling perltodo items would be even better. Specifically, http://rt.perl.org/rt3/Ticket/Display.html?id=66092 has a list of bugs blocking for 5.10.1. Fixing any of those bugs would be greatly appreciated.Smoke testing on OS's outside of the ordinary for p5p (NetBSD, QNX, and MVS to name a few) would be helpful too.
Thanks for responding Steve, but I can't believe your claim that dual-lived modules do not add to maintenance, especially given that Nicholas Clark and Dave Mitchell claim that they do. In particular, they've said that trying to get smokes clean on all relevant platforms for all point releases of dual-lived modules is exceedingly difficult.
Take almost anything on VMS, for example.
Obviously a smaller core would include just those modules necessary to download and install everything else.
I also don't believe that Perl 5 will ever gain more developers by going even slower -- unless you believe it's acceptable that it's okay that the horrible performance regression in Perl 5.10 is still present in the newest major release of Perl some 18 months later (unless you or your distributor want to apply patches).
Getting all of Perl's tests to pass on all environments is a difficult task regardless of dual-lived modules or not. I remember when I added some seemingly simple tests for POSIX to test the assorted time functions. It was an absolute disaster. It seems that almost every OS has their own special interpretation on how the standard C time functions are supposed to work. Another time, a new OpenBSD release added reentrant functions and required the structs to be bzero()'d, something we weren't doing. This killed the ability to build a threaded Perl a week or so before a planned Perl release. I've also spent more than a couple evenings fixing up bits that worked fine on several OS's but caused the Win32 smokes to go black. We fairly regularly get bugs opened from people that cannot build Perl on AIX because for some crazy reason libm isn't installed by default. Actually, a dual-lived module is causing Win32 builds to fail on bleadperl right now. It's a bit of a dual edged sword though since ExtUtils-Makemaker is required for Perl to build. And don't get me started on atan2(0), negative zeros, or using exit() vs. POSIX::_exit() in a child process.
Notice that in none of the above did I mention VMS.
Also, I don't believe I ever said to go slow. I would be exceedingly happy to go faster. However, it takes more time and energy than the small but dedicated group of volunteers always has. As I said before, we need more people to do useful things. Coding and testing speaks much more loudly than words. Again, see http://rt.perl.org/rt3/Ticket/Display.html?id=66092 for ways to help.