Vision and the Perl 5 Ecosystem

"A project without a strong sense of a vision"... vision is a dangerous thing. Clarity of vision got us Lisp, Forth, and Java -- all wonderful languages in their own way, but each also severely confined. It's human nature to know, or to feel like we know. We want to know what problems we're facing, konw what solutions there are, and know which is the best. But we can't. We don't know what the real problems are -- only hindsight will reveal that. We especially don't know what the problems are going to be a year or two down the road. The best solution won't occur to us until later. And there probably isn't even a single best but instead a complex web of compromises which would remind us what we have now and where we are now but in a temporally updated form. I'm not saying that lack of vision is a good thing; but lack of a single, unified vision certainly may well be. Perhaps the odd assortment of visions, most of them small, incremental ones, are the best thing. Compared to Java, Python and other language efforts, Perl probably makes its programmers feel the least like they're being drug along with someone else's "vision", though I think Ruby is doing a good job there too.

#corehackers is brilliant. I wish I could take advantage of that right now but I'm dealing with some crap. Actually, this reminds me of when Nickolas Clark was briefly working at the same company I was -- he was micromanaged and his skills squandered. The company was obsessed with the extremely short term as they had been for ten years. Part of my recent job ranting is based on this story. Not to compare myself to Nickolas Clark, but I think a lot of us Perl programmers are stuck in a vicious cycle -- we're over-worked and micromanaged. Even if our companies wanted to invest back into Perl, they wouldn't do it by being at all generous with our time. They're not even generous with our work/life balance. They're employing Perl programmers because they work hard for what they're paid and do what they're told. This makes it hard for anyone who wants to give back to Perl to do so. Too many people in the Perl community also spend too much time between jobs. Besides being disruptive, that's unnerving and it makes it hard to concentrate on Perl itself when you do have work. We're all going to find our selves in a position pretty soon here where we're working on Java by day and Perl by night. I think this is a case of a problem got bad before people were aware of it -- the community needed the image overhaul long ago and now we're getting the bulk of the punishment for the lack of it now.

But I think there is shared vision -- p5p, XS programmers, and old fashioned lovers of Perl 5 want the core cleaned up and refactored. As for what will be done with the newly cleaned up core no one can predict, but I have faith in the distributed instinct of smart people.

You've been beating a drum to try to motivate people. You remind me of myself when I was in charge of Phoenix Perl Mongers. My message of "come to meetings, present, actually present when you say you're going to, c'mon, we can do this but I can't do it myself" sounded good on the surface but people only got the subtext of "I'm tired of people checking meetings out and I really only want regulars; there's a lack of presentations and it's your fault; I don't like giving presentations; the whole thing kind of sucks anyway". You, like (I think) myself are a lot stronger on the technical elements than the social elements. I think, despite our intentions, we're going to step on a lot of toes unless we can find more constructive ways to beat this drum. Perhaps there are other ways that involve less blood, sweat and tears, but being an example and just being helpful to others is the best approach I've seen actually work.

Yes, the community has grown insular. I've been saying that for a long time. Friends have independently come to me, in one case after having spent time in the Ruby camp, and said the same thing to me. The reaction to typesafety then later autobox was an eye-opener here -- people kept commenting that "we don't need that!" and "that's not the Perl way!". Some times typesafety is a godsend -- when you're working on a large project where interfaces have to change often you get built-in tests that are more thorough than manually written ones. Yet the community has grown better at arguing against having features that they don't have than at evaluating ideas for their actual merits.

All I can say is that Perl's hope is just a seed right now, if it doesn't want to become, for 99% of people out there, a badly conceptualized stereotype overriding any realities and scaring off any fresh blood once and forever.

Any language that has been around for a while goes through terrible growing pains. Python lost a lot of people with py3k and in the load road leading up to it. Ruby's inability to get a fast and backwards compat new version out has damaged the perception of Ruby as being worthy to deploy for any real purpose. Java's hacknayed attempts to bolt on autoboxing and generics have tarnished its lustre as a simple, clean C-like language. It's remarkable -- nay, amazing -- that Perl has had so few growing pains and has held up so well for so long. It's over due for maturing, no question, but I think it's important to give it credit and especially not to be condescending about what it didn't do well enough in getting to where it's at. This remarkable accomplishment has to be weighed against the damage done by the messiness of core and the over-reaching backwards compatibility. Remember that even if Perl weren't "messy", the langauge would still have a stimga just because of the code written in it. There's now a lot of really bad Java in the world due to colleges teaching it as a first language and the radical growth of its use in India where novices are learning it cold and going out into the field and programming it. Any language that makes a lot of money for people experiences this influx of ineptitude. Ruby is winning over Java in the our-programmers-are-smart-and-write-clean-code department. Perceptions of bad code in Perl are only partially the fault of Perl's impressive backwards compat but I suspect more largely the fault of Perl's success in bringing in new programmers. Trying too hard to "clean up Perl" won't have the desired result.

All of this stuff is complex, non-linear, involved, and we lack the perspective on it to make the judgment calls. All we can do is effectively deal in large quantities of uncertainty. Part of that is not beating any drum too loudly, embracing the chaos, and encouraging and helping every programmer interested in core without politics about directions people want to take perl getting involved.

-scott

"Better is Better"... yes, but there are problems with putting things in core. Not everyone thinks Module::Install is a good idea. The fragility and lack of maintainership are overriding problems and it might be that core cleanup has to happen before user-visible goodness can be deployed. Yes, CPAN is a good place to experiment with language extensions and I love it when those are able to make it into core, eventually. They converse into proposals for extensions to Perl 5 mostly by debate on p5p about whether the feature is interesting enough that tired bodies can be bothered to risk having to take over maintaining the module.

"Does the CPAN help or hinder the desire for stability of the once-a-decade upgraders?" I could argue both ways on this but I think I'd be missing the point that PHP ate our lunch here. PHP's victory in the "install it and it works" space demonstrates Worse is Better better than anything else I can think of. We need PPI or something similar to it to be adopted as the way to distribute Perl apps. MakeMaker optimizes for the worst case -- running on an Apollo or something. That has merit and I don't want to throw out that baby with its bathwater but how can we make it so that people can install perl, download a script, run the script, and have it work, and preferably have the answer be less doomed in the long term than PHP's?

CPAN meets the needs of Perl programmers. Perl programmers seldom write code for anyone other than their employer or other Perl programmers. This is a problem. I've written about this before. Ruby programmers write code for *people*. PHP programmers almost never write code for each other. Code re-use is good, but it's a tool and it needs to be evaluated critically.

Not wanting to upgrade all of the CPAN modules and risk breaking something -- more likely than perl itself breaking -- keeps people from upgrading perl, I think. Very few people know about autobundle. I've had embarrassing outages due to perl upgrades and module problems.

I wrote before about Motorola's hundreds of installs various versions of perl in people's home directories on the mot.com/motorola.com server... that kind of follows what Flash does with upgrades. Each version of the player ships with each previous version of the player, it seems like. Perhaps it's time to fork Perl 5, so to speak, with deprecations. People aren't afraid to run old versions; they are afraid to upgrade. Capitalize on this.

Sorry, long winded because "I lack the time to make it short".

The various groups are at odds; the clash between the camps of "code re-use is good" and "I don't want to upgrade because things might break/installing Perl programs is hard!" stands out the most.

So far, CPAN authors have been doing the only thing they know how to make CPAN play nice with upgrades, and that's to be backwards compat with their APIs or else bump the major version number. That's besides fielding bug and smoke reports, specifying which version of perl the thing works with, and so on. The simple fact is that the more code you change, the greater the chance something is going to break, and an autobundle update and a new version of perl changes a heck of a lot of code. People are right to be nervous.

The whole CPAN problem really isn't my bag. I probably don't have any real insights for lack of thought. My knee jerk reaction is for module authors to please be more aware of what actually happens when lay people try to use their modules in the case where their module becomes a popular dep. In that case, "code re-use is good!" becomes more of a gray area. Also, we out-clever ourselves. I keep finding that I have to convert modules to MakeMaker before they're install because someone is using some new fangled build system that doesn't bloody work.

It may be that we have to decide that CPAN is for programmers and make something else for Perl app bundles. Perhaps the bundles need to get smoked. Perhaps it should smoke bundles that get posted other places if they're popular enough -- such as the blog code that's floating around, and AppEngine, and those things. Perhaps there should be some under the hood interaction between the two but from the user's point of view, they should know if upgrading perl is going to break any application in the registry of Perl apps.

-scott

I find this post confusing. The bullet points about CPAN near the end appear to come from out of nowhere. I don't see the connection between them and the alleged insularity problem you discuss earlier in the post.