Goodnight, Parrot

I stopped working on Parrot in January 2011. I'd started sometime in late 2001, probably around September, around the same time I submitted my first patches to p5p. Nine years is a long time.

I didn't have the fortune to attend the infamous mug-throwing meeting which launched P6 in 2000, but by 2002 I found myself attending P6 design meetings and taking copious notes and even contributing ideas here and there. (It's not easy to speak up when Larry and Damian get going. Throw in people like Allison Randal and Dan Sugalski, and it can be downright intimidating.)

I started working on Parrot because it sounded like an interesting project, because I wanted to play around and learn, and ultimately because I wanted to contribute to something that other people would use. That was the same combination of feelings I get when I extracted Test::Builder and made it a real thing. Countless people use that code millions of times every day, and it makes their jobs easier and it makes the experiences of immeasurably more people better.

That is a powerful thing.

Of course, I'm a pragmatist at heart. Software has to be used. That's why making Test::Builder felt more satisfying than getting a C patch in the Perl core, because the former gets used all the time and the latter fixed a very simple bug in a very unlikely code path in a Unicode system that's since been rewritten and very few people ever ran the code. (I have a few patches in the Linux kernel too, which ought to feel really cool, except that they're not very interesting patches in a not very interesting driver for a piece of very uninteresting hardware that not many people have. Still, both owners and the business manager of Onyx Neon have code in the Linux kernel, so that's something.)

I kept working on Parrot because I believed it would be the official-ish, primary-ish platform for P6. (It took quite a while before Larry gently convinced me that having multiple P6 implementations is a benefit, in some ways. I mostly believe him now, though that's a complicated thought for another time. I'll leave it at this: developer interest is not fungible. You can make suggestions and you can offer game theory and economic analyses to suggest that people focus one direction or another, but volunteers will do what they want and won't do what they don't want and you'll have a better time in life if you can accept that, or at least not waste your time fighting that, because you will lose that battle every time.)

Parrot had its problems from the start. You can explain away many of those problems as "People don't always make the right choices" and you can explain some of those problems as "People want different things and don't always communicate effectively with each other" and you can explain still more as "A little forgiveness goes a long way, but a lack of forgiveness stings for even longer."

I won't address the personal side. (I made plenty of mistakes. They're fairly well documented.)

On the technical side, Parrot suffered from the start from its initial design. I'm not the only one who's referred to it as "A great VM to run Perl 5.6." If you took what Perl 5.6 needed and cleaned up the internals, you'd end up with something that looks a fair amount like Parrot at the high level. That made a lot of sense in 2000, when Perl 5.6 was new and there would be no distinct 5.10 because it would have become a P6 slang and both would interoperate on Parrot and then everyone would either keep their old 5.6 or slowly migrate to P6.

Things didn't work that way. (In retrospect, that would be easy to predict, but remember yourself a decade ago and see how well your predictions turned out. Then expand that to include dozens of other people. It's enough to make you a quantum physicist.)

Parrot also suffered a lot from the idea of merging experimental code and having to maintain it. (In retrospect, that describes a lot of the mess around the Perl implementation, from XS to the bytecode backend, to undocumented behavior ossified thanks to the CPAN and the DarkPAN. The Perl world isn't always great about enforcing boundaries and it's definitely not great about sticking to the consequences of violating those boundaries.) I don't remember the exact details, but I do remember at one point there were at least three or four competing implementations of at least one subsystem, all at various stages of incompleteness, all messy code.

Everyone's least favorite part of Parrot—the IMCC system which implements the PIR language—was one of those experiments crammed into the core system and left to stagnate and spread its rot over the objections of its developer, whose reaction as I recall was both "Wait, that was just an experiment! I didn't intend for this to happen!" and "I was going to clean up a lot of stuff. It's not ready!"

After the great shakeup in 2007, the pendulum went the other way: a rush to finish specifications and implement them to the letter without a practical vetting of whether those features actually worked either in theory or in practice for users of Parrot.

That was also the time the deprecation policy (the other most hated part of Parrot) came about. The motivation seemed sensible, from one point of view. Parrot was always intended to be more than just "The virtual machine for Perl 6". Even as far back as the Programming Parrot April Fools hoax, the notion of cross-language interoperability was on the minds of the developers. Forcing Parrot to act like a mature project which didn't pull the rug out from under the feet of languages developed atop it would make the platform more attractive to developers of other languages.

That was the hope.

As things turned out, this wasn't enough to attract other language developers. Rather, it wasn't sufficient in practice when combined with the state of Parrot to attract and retain developers of other languages. Parrot had never offered great tools to build good languages, and Parrot's features didn't quite work well enough together for most purposes.

The Rakudo developers to their credit did build decent tools, but two problems worked against that. First, they did iterate on those tools several times, but those changes weren't always backwards compatible. In other words, to use those tools to build your own language on top of Parrot, you had to keep up with the needs of Rakudo as well as changes in Parrot. Second, because Parrot had its deprecation policy in place, Parrot had to maintain a stable enough interface that Rakudo would keep working as much as possible, which meant providing a stable interface for Rakudo's compiler tools, which meant not updating those compiler tools regularly in the face of breaking changes.

Granted, in the desire to rebrand Parrot as something not completely tied to the fortunes of a P6 (which by then had seemed to have been in development forever and delivered very little and not even Pugs had saved), Parrot had strongly suggested that the Rakudo project find its own repository, thank you very much, without letting the door hit it in its little backside on the way out. (The door did hit it in the backside on the way out.)

Prior to that point, any language within the Parrot repository which had a working test suite would automagically get updated for any Parrot incompatibility which occurred. That is to say, if I added a new parameter to a C function (often for performance or correctness reasons), or if I renamed a function (for cleanliness reasons), I could change the code in Rakudo itself to match that change and no one would notice that any incompatibility had occurred. These deprecations and changes were atomic. In one commit you had one version of Rakudo and Parrot which worked together nicely and in the next commit you had another version of Rakudo and Parrot which worked together nicely.

Hooray!

Because volunteer time and interest and skills are not fungible, some of the people working Parrot had goals very different from mine. I wanted a useful and usable P6 which allowed me to use (for example) PyGame and NLTK from Python and (if it had existed at the time) a fast CSS traversal engine from a JavaScript implementation. Other people wanted other things which had nothing to do with P6.

I won't speak for anyone else, but I suspect that the combination of a deliberate distancing of Parrot from P6, including separate repositories, the arm's length of a six month deprecation policy, and an attempt to broaden Parrot's focus beyond just Rakudo created rifts that have only widened by now.

I offered repeatedly to perform optimizations for Rakudo, to add features specific to Rakudo's needs, and to fix bugs that were holding back Rakudo. For the most part, these offers went unheeded. (I can think of a few cases where I was told repeatedly "Rakudo doesn't need that feature"—see Patrick Michaud's admission that Rakudo had told me not to make several improvements to Parrot—and then after I stopped working on Parrot, someone would sigh loudly and publicly, write a flurry of patches to Parrot to implement exactly what I had offered, and then someone would post a congratulatory blog suggesting that memory use or speed had improved, and isn't it all wonderful. Goodness knows I spent a lot of time fixing bugs in Rakudo's own C code which used Parrot underneath.)

Why didn't I fix those myself? Two reasons. First, the idea of opportunity cost. I didn't want to spend time writing code that Rakudo didn't need in the hope that it would eventually be useful, because I wanted to stay available to fix immediate problems. Second, because many of the things I would have had to do to make improvements would have violated the deprecation policy, if not by the letter of the policy in its spirit. Do that a couple of times and get yelled at and called a horrible person and you move on to other things.

My primary goal was always to get a usable and useful P6 implementation as soon as possible.

By the end of 2010, the Rakudo developers had decided to redesign their compiler tools again. I implied earlier that PIR wasn't a great language in which to write a compiler. Let me state that more strongly. PIR is a mostly terrible language in which to write a compiler. It's better than C in many ways. That's not high praise in the 21st century.

Almost no one in the Parrot world was satisfied with PIR then either, but it was a language under Parrot's control (and under Parrot's deprecation policy) and not a language developed somewhere else under separate design goals for separate purposes. Parrot did include a copy of Rakudo's compiler tools for historical reasons, but Rakudo used its own copy because Parrot would only use old versions due to deprecation policy and... oh, you see the madness.

Anyhow, in late 2010, Rakudo's developers decided to work on another version of the compiler tools. (The third version? Fourth? I forget—at least the third version.) This version would have the new goal of eventually being completely agnostic about the underlying virtual machine.

I thought this was a goal not worth pursuing at the time. My goal in working on Parrot was always to help realize a useful and usable P6 implementation as soon as possible.

My reaction at the time was a combination of "You're rewriting your compiler tools yet again?" and "That's going to take a lot of time that you could otherwise spend making Rakudo more usable and useful now" and "There's no benefit to Parrot in supporting those tools as the primary language for developing compilers on Parrot." I may have been a little more blunt in expressing my opinion then, and certainly reasonable people could have disagreed about the second point (but the so-called "nom rewrite" of 2011 didn't take only a couple of months as promised—it took most of a year before Rakudo had reached feature parity from before the rewrite).

I decided I wasn't having fun and silently stopped contributing to Parrot and P6 by the end of January. (Okay, someone had removed my P6 commit access too, but that's a different story.)

(I've mentioned before that my company was developing a product to sell to actual customers—a product based on Rakudo Star. The intent was to have it ready by April of 2010, which was Rakudo Star's initial release date, but that pushed back to the official release of Rakudo Star. Unfortunately, Rakudo wasn't stable enough that I could in good conscience ship a product based on it, so I waited until January 2011 for the project to mature. Then I decided to re-evaluate through the year to see if the nom rewrite would help things. By January 2012, I scuttled the project altogether. Trying to keep it up to date was no longer worth the costs.)

After YAPC 2011, I did spend a little time working on a prototype of a smaller, faster core for Parrot, but that went nowhere. I couldn't do the project on my own and had zero help, so I dropped it after a couple of days. Parrot limped along for a while, slowly losing contributors. (I don't mean to suggest that because I left, everyone else did, but it certainly seems like Parrot stopped being fun for everyone around the same time period. Funny how being told that everything you're doing is wrong and that you'd better not fix things is demotivating.)

Over three years ago I suggested that building on a foundation under corporate control makes you a sharecropper. You see it with web services—do you really want Twitter or Google or Facebook to have the power to pull the plug on your business? I know that's not an argument which convinces everyone, though.

I still have grave technical concerns about the direction of Rakudo, but I've expressed them often enough by now.

By the end of 2010, the Rakudo developers had decided that Parrot wasn't going to meet their needs such that any eventually useful and usable implementation of P6 called Rakudo would have to run on some other VM. Three years later, they're making steps toward seeing what's technically possible, and there's apparently an explosion of interest in porting Rakudo's compiler tools to all sorts of backends.

Of course, if you want to run the most mature and most useful Rakudo implementation, you're still tied to Parrot for the foreseeable future...

... which is kind of a problem, because there's an active question as to whether anyone's even interested in maintaining Parrot as a project anymore. The Parrot Foundation will shut down and transfer what it holds (copyright, trademark, legal stuff) to an interested entity, if one exists, and then... well, as Allison says "[We] had a good run."

The obvious idea ("Wait, because Parrot's still the foundation for the only Rakudo implementation even close to anything useful and usable, why not let Rakudo take it over and mold it to Rakudo's needs?") didn't convince Rakudo hackers Moritz Lenz and Carl Mäsak. They decided to lecture everyone who was still interested in Parrot that Rakudo doesn't want to take on Parrot until Parrot can demonstrate that it's not a dead project and that it has real benefits for Rakudo.

(After years of Rakudo developers telling Parrot developers not to work on performance improvements, that scolding scoured away any interest I might have in contributing to P6 ever again. Open source development—the meritocracy always works so much better with passive aggression.)

I know volunteer time and interest and talent isn't fungible, but what I wanted was a P6 implementation that was useful and usable sooner rather than later. Because I thought I wouldn't get that in the foreseeable future, in 2011 I stopped contributing to P6 and Parrot altogether in favor of things I enjoyed more. Everything I've seen from P6 since then has demonstrated that that decision was correct.

Parrot 5.0 came out last month, and there'll probably be a Parrot 5.1, but there's an irony to realizing that Parrot will peter out as of Parrot 5 and that there will probably never be a Parrot 6.

Update: Okay, I was wrong. Parrot 6.0.0 came out.