Clean Your Room

Like many of you reading this, much of my programming work requires maintaining software, and much of that software has significant amounts of code written by other people. Like most of the software projects in the world, much of that maintenance comes when stakeholders realize that the business could make more money or spend less money if the software did something new, or at least different. Other maintenance comes from a developer realizing that the code would be easier to write or faster or simpler or more secure or more correct if we could do things a little bit differently.

One of these projects is experiencing a great deal of growth right now. We're fortunate to have a small team of very good developers, a single customer/stakeholder/business unit, an ever-improving test suite, the technical freedom to make our own decisions, and a little bit of slack time every milestone with which we can make improvements as we go along.

In fact, one of the core abstractions of the system came about because we noticed a repeated pattern in our code and figured out a way to encapsulate that pattern in a single named entity. That's happened a few other times in the system, and it tends to happen to me a lot.

Because we have good developers, a useful test suite, and slack time, we can take advantage of the single deployment target to move things around as necessary. That means changing APIs. That means splitting apart classes or moving functions or renaming things or even deleting big pieces of code as we replace them with rewritten components.

You can make a lot of progress toward the right design with that freedom.

If you make a habit of improving your design, and if you're diligent about finding the right design, you'll discover that the places where you have the wrong design become more obvious. You'll also discover that the things you want to do tend to become easier over time.

If you discover that you've somehow finished your software or found a design that you'll never have to change again, stop. Then accept your Fields medal.

If, for whatever reason, you can't make improvements, or you can make improvements only very slowly, you're in trouble. There's a strong correlation between having the right design and being able to deliver working software. If your design is wrong, you'll spend more resources trying to achieve the same goal than if your design is right, or at least more right.

If you're running a volunteer project, you'll churn through more volunteers, unless there's an exceedingly attractive reason for people to stick around through that frustration.

All of this came to mind when I read David Golden's Is Perl 6 pointless, hopeless or just not done?. (Spoiler alert: David's article is really about Perl 5.)

The semi-annual sound-and-fury parade about renaming Perl 5 or Perl 6 sometimes sparks the somewhat useful discussion about whether Perl 5 can ever "break backwards compatibility". I take some responsibility for fueling the silliness of that discussion; if you have more free time than I care to devote to the task, you can look back in the archives here for some posts which rather overstated the problem.

Regardless of if or when Perl 6 is ever an option for your projects, Perl 5's future seems to me to walk the knife's edge of one sad fact: the Perl 5 implementation's source code compatibility with itself is the most difficult part of maintaining Perl 5. In other words, it's never had a really good room cleaning. Maybe it's never had the volunteer effort needed to make that happen, or maybe it's never attracted the kind of people with the skill and time to make it happen, or maybe it's never had the kind of project slack necessary to make it happen, or maybe it's all of those and more, but the real problem isn't that there's no will or ability to make changes incompatible with Perl 4 or Perl 5 before 5.10 or 5.12. It's that it's really difficult to do so within the Perl core itself, and that's even before you realize that some of the most innocent looking changes could render significant percentages of the CPAN unusable.

For a long time I believed that Perl might be the second example to contradict Joel Spolsky on whole-cloth rewrites, but I lost the will to work on Perl 6 and someone else can take my place there. If Perl 5 is difficult enough to maintain as it is, what chance is there that a complete rewrite will reach the point of utility before it burns out volunteers?

Even Python 3—the comparatively modest changes involved there—has taken several years before it's a serious target for migration of existing Python projects.

I suspect that the best hope of having a vibrant and vital Perl 5 which supports its full ecosystem is somehow figuring out how to redesign its implementation in place to support both new features and better infrastructure (cheaper Unicode, better concurrency, a core MOP, even macros and JIT).

That will be neither cheap nor easy, and it probably requires recruiting and training at least another couple of full-time developers. Hands wave at this point. The CPAN testers shudder. A dozen projects have come and gone and made little visible progress. A little voice is far too infrequently heard asking "Who's going to do the work? Who's going to maintain it?"

Yet the alternative is that every year Nick and Dave do a great job of keeping things running and making improvements here and there as they have spare time, and pumpkings like Jesse and Ric make wise decisions to cut through technical disgreements, and the thankfully too many to count now release managers take some of the administrative burden off of Nick and Dave and Jesse and Ric so that there are fewer single points of failure, but there just aren't enough resources to make the kind of dramatic improvements that a focused effort on a redesign in place might produce. To be excessively clear: you should blame no one for the Perl 5 internals being what they are. There's been a shortage of resources to make improvements for far too long.

I know it'll be expensive and difficult and painful, and I probably undercut my argument by admitting that I'm still burned out on volunteer projects, but if there were somehow to appear people willing to take on this task, I would donate some time to help and my business would donate some money to TPF to help make it happen.