The Implementation of Perl 5 versus Perl 6

A friend asked me a really confusing question today:

Do you think that Perl 6 is a well-designed language?

(Okay, that's not the confusing part.) I told him that I believe so, for the most part. Perl 6 has a coherence and a consistency that Perl 5 lacks. When people call Perl 5 "a cleaned up single dialect of the wonderful mess that is Unix", it's true. It's just never quite been cleaned up enough.

For a good example of this, look at the inconsistency of regular expressions and PCRE and the other Perlish extensions to regular expressions. That's definitely a source of the complaints of line noise. (I say this as someone who's written far too many gentle introductions to regular expressions.)

For other examples of Perl 6's consistency and coherence, look at the small unifications that make sense: an improved type system, a unified object system with an intelligent metamodel, metaoperators, and a clearer system of context which provides for such niceties as pervasive laziness.

In our further discussion, I said "It's a difficult language to implement." (I've patched at least four implementations of Perl 6 over the nine years I worked on the language, and that number might be low.)

Then I confused myself by saying "I think it's easier to implement than Perl 5 though."

In one example, Perl 6 has a well-defined grammar which requires only single-pass parsing. By contrast, the Perl 5 parser weaves in and out of parsing and lexing and even actively running code to determine how to parse various constructs. Most of that is practically unnecessary, but it happens anyhow.

(Of course, Perl 6 also allows users to override the grammar even to defining what the language considers whitespace, so you don't have much chance of writing a fast or efficient Perl 6 parser because you can't even assume that you can skip over whitespace in fast code; you always have to look up the current rule for what whitespace is. A really good specializing JIT could probably inline some of these rules and assumptions, but someone would have to write that, and there's no indication that that will happen for Perl 6 any time soon. If you've ever asked "Why is the Perl 6 parser so slow?" or "Why does Rakudo use so much memory?" or "Why does Rakudo take so long to start?", now you know the answer: you're paying for flexibility in the parser that almost no one ever, nowhere, never is going to use.)

As another example, the optional type system of Perl 6 allows improvements in optimization that are difficult to implement in Perl 5. Knowing that certain arguments can only ever be read—never written—means that an implementation can cheat on the question of whether it passes arguments by reference or by value. Multidispatch as a fundamental strategy—resolved at the point of compilation—reduces runtime branching and can allow aggressive inlining. So can the decision to mark a class or module as closed to further modification.

What confused me?

Why does Perl 5 exist? Why has Perl 6 been a dead end for so many years, and why doesn't it look like that will change any time soon?

It's easy to say Worse is Better or Second System Effect, but I'm not sure either one really applies fully.

As I see it, Perl 6 suffered for its first few years with the lack of any usable implementation at all. (Parrot had a couple of half-hearted implementations from almost its start, but the less said about those, the better.) Only when Pugs appeared did the language designers have any chance at all of testing their ideas with even a modicum of working code. The best thing Pugs ever did was spur the development of a test suite.

Unfortunately, Parrot had already become a tangled mess of poor code written to the wrong design under the aegis of scope creep by that point. We had the opportunity to reduce some of that wrongness, but it never happened. (Yes, I'm responsible for part of that mess in design, implementation, and even management.)

Pugs imploded shortly after anyhow. (Heroics aren't sustainable.)

One of the drawbacks of an independent test suite is that anyone can write an implementation to that test suite. It's tempting to throw out a big mess of working code and start over, figuring that at least you can salvage your tests. (I've never seen a project of any size succeed by throwing out everything but its tests, but I've seen several projects improve by committing to and delivering a relentless series of improvements.)

I've claimed on multiple occasions that the current trajectory of Perl 6 could best be characterized by a desire to reinvent the world every twelve to eighteen months. (I also get a lot of feedback that my claims are misleading and wrong and mean-spirited, but at this point there's a lot of history to support my claims and not a lot of success to counter them, so there's that.)

At that point in the conversation, my buddy asked about Parrot's current use case. "What's it for?" he asked. "What does it do well? What is it supposed to do?"

As far as I can tell, Parrot is on life support. It exists primarily to ensure that something can run Rakudo until the Rakudo developers finish porting Rakudo to another VM. (The current target I've seen is the JVM, which as everyone knows supports Perl semantics about as well as a universal Turing machine supports the lambda calculus.)

In other words, if you tied your hopes of having a usable and useful Perl 6 implementation any time soon to Rakudo running on NQP running on Parrot, I'm sorry.

If Perl 5 is a worse language with a worse implementation, how did it succeed? It was usable from almost the start. Perl 4.0.36 came out in February of 1993 and Perl 5.0 appeared in October of 1994. That's 20 months—a long time in Internet time, but a blip in Perl 6 history.

Perl 6 is a lot more ambitious than Perl 5 to be sure, but is it seven or eight times more ambitious? Ten? If Perl 6 is generally usable by February 2017, that's 200 months. Is that an appropriate length of time for an order of magnitude increase in ambition? I don't know.

Could things change? Sure. It's just code. Parrot could have a sudden infusion of energetic developers who take great pleasure in removing awful code, revisiting inconsistent interfaces, improving poor designs, and generally adopting the useful semantics that a modern VM needs to express to have any chance of running dynamic code effectively and efficiently while not ruling out extension or optimization. (The lessons of Smalltalk, Dis, and Lua are relevant, even though the world seems to have a strange hangup on LLVM's unfulfilled promises in this realm—not to say anything bad about LLVM and Clang for what they do well, but optimizing C++ is very, very different from optimizing everything else. So there's also that.) Unfortunately, Parrot won't change. Rakudo's seen to that; by spending years complaining about Parrot, sabotaging Parrot, and driving off Parrot's developers, Rakudo's all but guaranteed that.

Could Rakudo change? I doubt it; the rifts between Rakudo and Parrot have their roots back as far as 2001, when Parrot's design took the road marked on the map as "A better VM for Perl 5.6". If, in the 2008 time period, Rakudo and Parrot had mended things such that the features Rakudo currently has bolted on to NQP to support Rakudo properly had made their way into Parrot proper, things might have turned out differently. (Parrot's current object model was designed to the satisfaction of absolutely no one—everyone involved in its design looked at it and thought "Wow, that's wrong in so many ways" but we could never communicate well enough to figure out something more correct. I suppose it's darkly amusing that the person who implemented Parrot's object model went on to spend several years alternately revising an object model for Perl 6 and telling Parrot developers not to change Parrot's object model.)

The design flaws in Parrot will never get addressed. Then again, porting NQP to other VMs will demonstrate that Parrot was still the easiest target for NQP, warts and all. Rakudo in December 2013 will look a lot like Rakudo in December 2012: modest improvements to be sure, but incomplete and generally unusable for most purposes. The state of Perl 6 in 2013 will be overpromises, underdelivery, and disappointment—just as the state of Perl 6 has been since about 2003.

Despite all that, I do believe that the design of Perl 6 as a language is solid. How unfortunate that the task of implementing the language has suffered as much as it has.

Creating yet another half-hearted (choose whichever body part you find most appropriate) implementation won't gain you much momentum, and you'll spend a lot of time reimplementing basic features that aren't all that interesting before you reach any point where you do something useful. Parrot's all but dead, and NQP is probably the next piece to suffer the inexorable rot of "Hey, this big wad of code doesn't do the right thing, so let's add yet another layer of abstraction so we don't have to touch it" because abstractions leak and leak and leak like batteries bulging in a toy left outside for the winter. I suppose it's possible that a huge failure to get NQP and Rakudo working anywhere but Parrot will cause people to look at Parrot again, but that presumes that Parrot will survive until then and that the NQP retargeting will have dramatic and obvious failures rather than modest but subtly lingering time-sucking flaws. Alternately, I could be wrong and NQP on the JVM or the CLR or the LuaVM or some JavaScript engine could work wonderfully, and building up all of the infrastructure Perl 6 needs could go very quickly and it won't suffer from major problems like the impedance mismatch of an alien memory model. I give that a 30% possibility.

Perl 5 has huge flaws in implementation, but it delivered working code in a reasonable time period, and it's not going away. In the two and a half years since the first release of Rakudo Star, it's perpetually overpromised and underdelivered. Nothing I've seen since then has convinced me it'll cease to be anything but a toy for the foreseeable future. In other words, Rakudo Perl 6 has failed and will continue to fail unless it gets some adult supervision.