No Policy Can Save Wrong Code

I've written before that software projects need sane, published deprecation policies, and I still believe that...

... but listen to a tale of two intertwined projects with a seemingly-sane and published deprecation policy that sounds great but doesn't actually work.

Parrot and Rakudo used to be much closer together. The Parrot repository had a subdirectory languages/ which held several language implementations running on Parrot. This was well and good (How did a lion get rich? It was the olden days!)—whenever Parrot underwent an API change, a simple ack of the source tree could find most places that needed to change, and running the test suites of the projects under languages/ could find the rest.

(Getting people to run the full test suite is a different story, but in that case at least guilt and a good bisection tool work post hoc magic.)

Then one day, Parrot left the Perl-centric world of perl.org and kicked languages/ out of the nest. languages/perl6 went to Rakudo.org and the troubles began.

You see, back in the olden days when a lion could get rich and the Perl 6 VM was like the Perl VM but a little bit better, the way to extend Parrot was, obviously, write a whole wad of C code. Want to add a new data type? Write a wad of C code. Want to change how a core wad of C code works? Write a whole wad of C code. If hacking on Perl had taught anyone anything (and I can't believe I can write this with a straight face), it's that finding hackers ready, willing, and able to write big wads of C code to work around or customize or extend other big wads of C code so that someone else doesn't have to write C code is a really easy prospect.

Anyhow, the Parrot execution and extension and embedding model has always assumed, as does Perl, that the VM should expose a big wad of C functions that extenders and embedders should use to manipulate big wads of C structs and other data types defined in C.

On one hand, customization and flexibility is good. On the other hand, people like me who know both C and Perl write in Perl when we can and C only when we have to for several very, very good reasons.

Imagine a ventriloquist with a Parrot puppet, except he doesn't stick his hand in the puppet to pull levers and flip switches and wiggle his fingers. He has a robot hand, operated by a joystick, and that robot hand is half robot and half organic and it tickles actual organs inside the puppet and the bird goes off and does its thing. Oh, and he's operating that joystick with his toes. Plus he's underwater, wearing a blindfold.

In other words, Rakudo has an intimate knowledge of the guts of Parrot and does some strange and bizarre things. (If you look through the revision history of those parts of Rakudo, you'll see I've fixed some of those strange and bizarre things and I've perpetuated others.)

When Rakudo and other languages left the nest, a deprecation policy came about. Imagine this vaudeville act:

HLL Developer: "Hey, you changed this API!"

Parrot Developer: "You weren't supposed to use it."

HLL Developer: "No one told me that, but regardless, you broke my project."

Parrot Developer: "Oh, sorry. Well here's how to fix it."

HLL Developer: "Wish you'd told me that months ago."

Parrot Developer: "Yeah, we'll do better next time. By the way, what in the world were you doing with that?"

HLL Developer: "No one told me to do it differently, so I just threw together something that works."

You can see where this is going.

Thus began the great Parrot version numbering wars. The end result was that Parrot would have two supported releases every year, one coming every six months. The initial deprecation policy promised no backwards incompatible changes without notification in at least one supported release.

In other words, if a project which was previously in languages/ had moved somewhere else where a well-meaning Parrot developer didn't notice that a simple change broke backwards compatibility, that project had the ability to request a reversion of the change, a documentation of the change as a compatibility change, and a delay of up to six months before making the change...

... depending on when the project noticed.

I've mentioned the idea of technical friction before, but get out your calendar.

This policy didn't only protect languages from unintended changes, it protected languages from desired changes. For example, if Rakudo wanted a new feature in Parrot, and if the best way to add that feature in Parrot meant breaking backwards compatibility (changing the order of search paths, for example), both Rakudo and Parrot developers had to mark their calendars.

The deprecation policy period is now three months, for hopefully obvious reasons.

Even so, now there are at least two projects on separate time tables with separate goals for features and improvements and deprecations. When Rakudo wants a feature from Parrot that Parrot doesn't yet support (or Rakudo developers believe it doesn't support fully or at all or in the precise way Rakudo wants), Rakudo tends to toe the joystick. When Parrot makes a change that exposes the fragility of tickling bird gullets with robot fingers, Rakudo gets to complain. The deprecation policy followed to the letter prevents Parrot from adding the features Rakudo demands at the same time that it prevents Parrot from improving the features that Rakudo uses.

You might read this and think that the problem is the deprecation policy. It's not—at least, that's not the root cause.

I hate the polite fiction calorie-free slogan that "Perl 6 is a research project", because it's a polite way to suggest that Perl 6 is an embarrassing pipe dream no true Perl hacker takes seriously, but implementing Perl 6 has required a lot of research and fits and false starts. (Before you complain next that it's taking a while, you write a performant grammar engine which allows lexical overriding of any grammatical rule or category.) When Parrot began, no one knew exactly how a Perl 6 VM machine should implement several important features. That was also true in 2005. That was also true of Parrot's 1.0 release in March 2009.

Making an effective Perl 6 means continually thinking and rethinking and simplifying and inventing new things.

Now tie an anchor around the feet of a live bird and pretend you're a ventriloquist.

Granted, Parrot's also long had the goal of running other dynamic languages efficiently and effectively. Its endgame is more than merely Perl 6 (and Perl 5). Yet a Parrot that won't run Perl 6 well has failed at its primary goal, and subsequent goals are less interesting.

The right solution is to invent a time machine and not kick Rakudo out of the nest. (The rightest solution is to invent a time machine and start implementing Perl 6 on Parrot from the first day as a first-class citizen and not a rat's nest of Perl parsing madness which had, to my count, at least two replacements before Rakudo.)

Rakudo's developers—who had commit access to Parrot, of course, so they could fix things if they wanted—declared that the only real solution would be to add another layer, another project external to both Parrot and Rakudo, which will sit in between the two and abstract away the Parrot specific parts in the hope that someday someone will add another VM backend so that Rakudo doesn't have to be tied to Parrot forever. In other words, throw away all of the work that went into Parrot, start over from scratch, and hope that this rewrite will be the one that sticks.

That went over about as well as you'd expect. (Chasing away Parrot's developers set Rakudo back by another couple of years.)

If the notion of working around a broken deprecation policy—broken because of too little coupling between two projects so intimately intertwined—is to add another layer of coupling to separate those two projects seems a little strange to you, you're not alone. By my count, that's three projects in a driver's seat built for one. If you can figure out which project is upstream and which project is downstream and how to debug with reasonable each which change where broke which test and who gets to fix it, you're a far, far better developer than I am.