How Forking Perl 5 Could Work

Stevan Little (the man behind Moose) gave a talk at the Orlando Perl Workshop called Perl is not Dead, it is a Dead End. The talk culminated with an announcement of an experimental reimplementation of the useful parts of Perl 5 in Scala, a project called Moe.

This is not the first fork or pseudo-fork of Perl 5. The Topaz project eventually begat Perl 6, which begat Parrot (the way I understand Parrot today is that it's what you get if you try to write a virtual machine in C to run Perl 5.006 effectively, and then check in a lot of suboptimal code hoping that magical elves will somehow coalesce out of the luminescent aether to fix your mess). Later on came Kurila which threw out some of the awkward parts of Perl 5 (and some of the useful parts) in an attempt to gain more speed. Lately, something called Perl 11 looks like an attempt funded by CPanel to add optional optimization levels to a pre-5.10 Perls. (I suspect the people behind Perl 11 will object to this characterization, but I find the lack of specificity frustrating.)

Now comes Moe.

Will it succeed? I don't know. Before anyone can address that question responsibly, you must understand what a fork can or should fix.

What's Wrong with Perl 5?

Perl 5 has two main problems: its implementation and its implementation. Any time you think you've found another problem, look deeper and you'll discover it's one of those two problems (and very likely "its implementation" is the culprit").

The Perl 5 VM is a big wad of accreted code that, in some places, traces its lineage back to Perl 1. It's exactly what you'd expect from code written in the era where "make it fast" meant "Write it in C, come hell or high water" and it shows. Where a Smalltalk might be implemented in something like Slang or PyPy in RPython, Perl 5 doesn't do that.

That choice almost certainly made sense in 1993. By the time of Topaz in 1999, C made less sense. By the time of Parrot in 2001, C made little sense. In 2013, C makes almost no sense.

Bear in mind that my argument is "Writing the whole thing in C makes little sense for a language larger than Lua".

Why moe Might Fail

"Almost as good as Perl 5" isn't that compelling. If I wanted to use a faster and better language with worse deployment and fewer libraries, I'd write more Haskell.

Perl 5 has no language designer with singular vision and the time and taste and experience to shape the language in a cohesive gestalt. (That's probably the worst part about Perl 6—it took Larry's attention away from a working product to something which so far has not delivered anything usable in and of itself.)

Technical reasons, like "Wow, the JVM isn't really a good platform for a Perl!" or "The subset of Perl 5 that's practical to implement is basically Groovy and that already exists." I'm not predicting these specific cases, mind you. I offer them as examples of technical reasons which may exist. (Even though I suspect the JVM really isn't a good platform for a Perl.)

Social reasons, like "This is more work than we thought, and it depends entirely on volunteer labor." (That excuse worked for Perl 6 for a while between the time TPF stopped giving $50k grants and the Ian Hague "Get this in a usable state in the next couple of years to attract more grant money!" grants didn't achieve their goals.)

Can moe Deliver?

It's possible moe can work. It has to avoid two traps:

• Falling into a local maxima because of the limitations of the underlying technology. (It would be mean of me to call this the Rakudo-on-other-VMs Trap, so I won't.) For example, it is so exceedingly difficult to implement a language with decent performance when the semantics of how you use memory and where you get that memory and how you release it and when you release it are different from the assumptions that the VM and its optimizer and any JIT and tooling and extensions expect that you would have to be the combination of Michael Abrash, Mike Pall, John Carmack, Cliff Click, and quite possibly Pablo Picasso to make it work well across multiple VMs.
• Ossifying into something that can't change before it produces sufficient utility. (It would be mean of me to call this the Parrot Support Policy trap, but at someone who argued both sides of that support policy at various times, it's one of the most important reasons why Parrot and Rakudo locked into their whirlpool of mutually irrelevant destruction.) The best general use projects I've seen have found themselves extracted from specific situations only at the point where the specific project can support the necessarily generalization and where there's enough external knowledge about the needs of the extracted process that such generalization is possible.

In other words, it's a mistake to commit to supporting internal details until you're certain that those internal details will remain in place without reducing or removing your ability to make necessary changes for future improvements. Both Perl 5 and Parrot fell into this trap.

What moe Could Produce

If I were to implement a language now, I'd write a very minimal core suitable for bootstrapping. (Yes, I know that in theory this is what NQP or whatever it's called these days in Rakudo is supposed to provide, but unless NQP has changed dramatically, that's not what it is.) Think of a handful of ops. Think very low level. (Think something a little higher than the universal Turing machine and the lambda calculus and maybe a little bit more VMmy than a good Forth implementation, and you have it.)

If you've come up with something that can replace XS, stop. You're there. Do not continue. That's what you need.

Then I'd figure out some sort of intermediate tree- or graph-based structure suitable to represent the language as I'd like to implement it. (Rakudo has a decent implementation here. If it had gone into Parrot five or six years ago, the world would be a different place.)

Then I'd produce a metaprogramming system, something of a cross between Moose's MOP and the Perl 6 metamodel. (Rakudo gets this very right. Credit where it's due. If Parrot had managed to adopt this in 2011... well, that's a long rant for another time.)

With those three things, I believe it's possible to build all of the language you need. If you're clever—if you're careful—you can even produce a system where it's possible to create and modify a sublanguage through metaprogramming but limit the scope of those sublanguages to specific lexical scopes in your system. (That idea is probably the idea that Perl 6 the language gets most correct. It's also very Lispy, in the sense that it's tractable in Lisp, but fortunately for the rest of us programmers, Perl actually has syntax.)

Figuring out a bytecode system is mostly irrelevant. (Freezing bytecode too early cost Parrot a lot of time and energy.) Figuring out a replacement for XS is essential (everyone says that Perl 5's parser is the biggest barrier to alternate implementations, but the utter dependence of the CPAN on XS and the haunted Jenga horrors of XS and the Perl 5 internals is the biggest barrier to adoption of alternate implementations).

Breaking the dependence of the CPAN on the Perl 5 internals—even in a small way—while allowing the evolution of the language and the annealing of the implementation over the specification toward completeness (annealing in the AI sense, not necessarily metallurgy) may be a viable path to producing a Perl 5.20 which allows optional backwards compatibility if you want it and usable new features if you need them.

Notice that this plan ties the implementation of an alternate Perl 5 to no one specific backend. I suspect that RPython will demonstrate that it's workable, while I'm tempted to suggest that LuaJIT has possibilities. (Again, I think it's 70% likely that the JVM and the CLR will prove themselves workable for the first half of implementation and completely bizarro-land useless for the second 80%, but Rakudo will demonstrate that soon enough.) I don't know about JavaScript VMs.

Is this worth doing?

Hey, I've paid my dues trying to implement a modern VM and modern Perl implementation.

The real question is whether an alternate implementation of Perl 5 can demonstrate its value for real programs before Booking.com's money runs out keeping Perl 5 on life support. It's clear that the current Perl 5 implementation will never get the kind of attention it needs to introduce it to the 21st century, and it's pretty clear that no Perl 6 implementation right now is anything other than years away from being useful, let alone of interoperating with a Perl 5 implementation.

I don't think Perl 5 is flirting with irrelevance. I do think that every year that goes by with Perl 5 ossifying further in implementation makes it less likely that the necessary changes will happen. (The code doesn't get much cleaner, the likely implementers get busier and less interested, and the fashion-driven Silicon Valley marketing machine keeps vomiting out new trends you absolutely must adopt right now, you creeping dinosaur, which is a distraction of sorts.)

If the CPAN has proven anything, it's that one size doesn't always fit every program. Maybe the p5p model of trying to please everyone (and generally only pleasing sysadmins with Perl 4 code they reluctantly last updated in 1993 only because they started it in 1987) doesn't fit all either, and maybe an alternate implementation of Perl 5 will produce a viable model to reinvent Perl 5's internals.