June 2013 Archives

If you read blogs.perl.org you've probably seen a debate over whether some of the libraries often lumped into "Modern Perl" are necessary.

You may also have read Perrin Harkins's Fond Farewell to CGI.pm, which does an admirable job of sending off that venerable piece of code many of us remember far too well. Back in the olden days, a lot of us spent a lot of time converting a lot of standalone CGI Perl programs to use CGI.pm instead of a hand-rolled parser copied from too many places on the Internet that didn't have a clue about security or performance or correctness. If we were lucky, those scripts were Perl 4 and used something called cgi-lib.pl, which at least made an attempt to be useful in multiple places.

The CPAN was a revelation. It was an oasis. It was, all of a sudden, the solution to problems we didn't know we had.

I find no small irony in a few posts asking whether there's any reason to use Moose or Mouse or Moo when you can write your own object accessors by hand (I wrote my own templating system by hand too. No more.) juxtaposed with a farewell to the library that convinced me that the CPAN was here to stay.

Sure, I wrote a handful of very small programs that didn't use CGI.pm or anything like that because they didn't even need that much. (I wrote my own web server in the late '90s, because it seemed interesting.) Yet every time one of those programs grew a little bit larger, a little bit more of what CGI.pm provided became more useful. Cookies? Handled. Persistent form values? Handled. (Oh, what a glorious thing that was!) A new RFC I didn't know existed (if I even knew at all what an RFC was)? Supported!

I kinda feel the same way about some of the pieces of what I call Modern Perl.

For example, I know I could build something like XML::Rabbit on my own, if I had to. I could also parse XML by hand, or with a regex tokenizer, or with XML DOM methods.

I don't have to.

I know Robin could have written XML::Rabbit without Moose. Sure, its interface would be a lot clunkier. It would have taken him a lot longer. It might not even exist without Moose.

Thankfully, the Perl world has Moose and things like XML::Rabbit built on top of Moose because Moose exists and is stable and is usable and gets used.

Again, there are people parsing XML out there the Perl 4 way, and if they're getting their jobs done, then thumbs up for pragmatism. If all you need from one document is just a tiny little piece of information, by all means use the simplest thing that could possibly work, for whatever definition of "work" makes sense in your context.

Yet when I look at the code I wrote back in 1998 and compare it to the code I wrote in 2008 or just today, I see how much more I can do (okay, a decade and a half of experience helps a little) with less, because the language and tools and libraries have improved that much. By all means, mist up when you think about CGI.pm, but to claim that everything since then is unnecessary bloatware that doesn't do anything you couldn't do by hand?

Sure, maybe you could do everything by hand. If that's your idea of fun, more power to you. If that makes the most sense for your business rules, good for you for being pragmatic and letting something other than shiny programmer magpie syndrome inform your decisions. Yet you'll not often catch me shunning improvements to productivity and capability. I'm too lazy not to work hard to keep up with new ideas, because some of them (Moose, testing, higher-order programming, Unicode, Perl itself) change the game so much they define the game.

You can't afford modern Perl in your environment? That's fine. That's the case for many uses. I can't afford not to have modern Perl in mine.

A few days ago, one of Perl's most distinguished free-floating agents of chaos posted a patch for discussion. dots.pm changes the Perl 5 dereference arrow to a dot. This lexically scoped pragma also changes the concatenation operator from dot to tilde, which is the current syntax in Perl 6.

p5p reception tended to be positive. Discussion elsewhere tended to be negative. After reviewing the proposal and reception, Perl 5 pumpking rjbs rejected the patch for 5.20. His reasoning is straightforward. While fans of dots.pm may feel a legitimate disappointment at the current rejection of the patch, it's worth praising rjbs for an evenhanded evaluation of the intent and implementation of the feature. (If that evaluation had gone into the smartmatch operator or automatic dereferencing of references to aggregates with each, for example, Perl 5 would be in better shape.)

My initial reaction to the patch was mild interest. It didn't immediately grab me as an idea that Perl needs. The more I thought about it, the less I liked it, for four reasons.

It's too invasive technically for my taste

It's a small patch to the parser, but it repeats a pattern which I've never liked in feature.pm, which is to say that it adds branches to the Perl parser/tokenizer/lexer which apply to the current compilation unit based on hints provided to the lexical scoping.

To some degree this is a limitation of how a Perl program gets parsed, but the more optional features and branches in the parser, the more difficult it is to maintain the parser. I know this sounds like a slippery slope argument, and a pernicious one. To some extent it is, but the parser and tokenizer and lexer are already a big ball of mud. Making that more so worries me.

With that said, the patch itself is as clean as you can reasonably expect. This is no criticism of Chip's skills.

The patch does too much

I like the idea of changing method calls from $invocant->method to $invocant.method. I'd like to experiment with that in my code for a while. (Back when I wrote P6 code, that syntax was easy to use and easy to read.)

I'm ambivalent about changing the concatenation operator from dot to tilde. If there's a way to keep concatenation as it is, so much the better—but that probably means requiring significant whitespace around the concatenation operator and forbidding whitespace around the method invocation operator. The latter is troublesome; it would be a shame to borrow the problematic "unspace" concept from P6.

I'm not thrilled at all about using dot as a generic dereference operator, turning $href->{key} into $href.{key}. That seems to borrow trouble; think about bugs waiting to happen with that code.

It might encourage fragmentation

While some parser changes merely add new keywords (say, defined-or) or make code that was previously a syntax error work (package BLOCK), this changes the meaning of two (maybe three) operators which have worked this way for almost 20 years.

Yes, the effect of the dots pragma is local, as it should be, but the effect also creates divergent dialects of Perl. Rather than having to learn the meaning of new terms when the say feature is in effect, dots means having to learn the new meaning of a term when it is in effect.

That cognitive burden seems higher.

It's like updating the style guide on a well established project. Previously it recommended always quoting hash keys. Now it recommends the opposite. You'll spend time working with both styles until you scrub the old version out of your code and tests and support files and everywhere it's reached.

Yes, the lexical scoping of dots.pm helps, but do you really want to sprinkle use dots; and no dots; throughout existing code while you're making that transition?

If the use of dots.pm spread to the CPAN, could it ever be contained? Modern::Perl was never supposed to be a dependency of other CPAN modules, but it is. So is common::sense and, most unfortunately, strictures.pm. (The latter is worse because it behaves differently when you run it from what appear to be revision controlled directories. If you like action at a distance, HEY LOOK OVER THERE!)

It exposes but does not improve a problem of the Perl 5 implementation

This argument is subtle and fuzzy.

Patching the Perl 5 parser is relatively easy. (I've done it a few times. When I added the ... operator, I had to work around a few issues like this.)

Creating a local dialect of Perl should be possible. Every time you create a function (or import one), you're doing that. Every time you use a prototype, you're doing that.

Wouldn't it be nice if, Lisp-like, it were possible to create your own dialect of Perl in which you can replace operators, add operators, or even remove operators? (I won't miss reset, and I would love to fix the unperlish contextual behavior of the x operator.)

Making that happen is a huge amount of work. It may never come to pass. Yet adding more branches and conditionals and special cases to the unholy union of the Perl lexer/parser/tokenizer seems to me to make that work even more difficult and even less likely.

I don't blame people for patching the parser to make these syntactic changes happen. It's much, much easier than fixing the parser to make these changes easier without patching the parser. Yet the latter is the right technical choice, even though it's more expensive now.

Yet that runs into the same problem as always. Perl needs to allow experimentation, but it shouldn't rush to ossify experments as core features merely because it's most expedient to perform these experiments in the core.

The other day, I found myself with the perfect example to explain continuations to some web programmers I know. (If you didn't already know how continuations can make you sandwiches without you leaving your office, they're powerful things.

Imagine you have a controller for a web application. Imagine one of the methods is edit_profile. You have an access control mechanism that requires authentication before someone can edit a profile. Your code might look something like:

sub edit_profile {
    my ($self, $request) = @_;

    return $self->redirect_to_login_and_return( $request )
        unless $request->is_authenticated;

    # actually edit the profile here
    ...
}

Now HTTP is stateless. Barring some persistent connection through websockets or the like, the server must redirect the client to an authorization page, receive and verify another request from the client, and, if that succeeds, redirect to the action shown here to complete the user's request. If there's state built up to make this request (a user ID, form parameters, the contents of an active shopping cart, whatever), something needs to manage that data through all of these redirects.

There are plenty of ways to handle this; you've probably implemented at least one. It's a common pattern.

If you're like me, you'd like to be able to set aside the current request temporarily, manage authentication, and then resume the request just after the point of checking for authentication.

In other words, once your web framework has handled the HTTP response to the point of figuring out what the client wants to do (edit a profile), restore the state of the web application and start over as if the client request had been authenticated from the start.

Again, you can do this all manually, but if your language or framework supported continuations, you could have a mechanism to capture all (or some—that's a delimited continuation) of the control flow and request state through your application at the point of checking for authentication and restore the state of that control flow.

There are nuances here. You probably also need to serialize the state of the continuation and HTTP request and store that on your server (never trust the client). You need to collect stale continuations (clients can abandon requests) but not too frequently (you probably have tabs open in a browser window that's been open for days, too). You have to go to some lengths to avoid circular references, and it's not always easy to serialize information like open sockets, open files, and active database connections.

... but these problems are solveable, and if your language and/or web framework supports this (if someone has solved them once and for all so that the average programmer doesn't have to), then you have a powerful tool for managing state atop a stateless protocol.

Note that a web framework could provide this feature even when its host language doesn't... but that's an idea for another time.

If programmers could learn one thing from successful businesspeople, they should learn about the idea of opportunity costs. Sure, it's fun to throw away a lot of code and rewrite it from nothing, but in the years you're waiting for that mystical magical super sixy project to get usable, you could have been making money with working code, even if it's a little shabby around the edges.

Sometimes opportunity cost works the other way, too. If you can get a 1% return by putting your money in a CD for 12 months or you have the chance to get a 10% return if you can buy the right stock sometime in the next three months, hold out for the 10% return. You might get it. You might not. Yet the reward is greater than the risk.

So it goes with programming.

When I wear my programmer hat, I want to write the best code imaginable. I want to find the right abstractions. I want to discover the most elegant design. I want to put in the least effort. The risk of getting it wrong and having to do more work is greater than the risk of missing a deadline.

When I wear my business hat, I want the most valuable features as soon as possible. The risk of missing out on business value (greater revenue, lesser costs, greater productivity) is greater than the risk of increased future maintenance costs. After all, it should be possible to measure those increased costs and deal with them when it makes the most sense from the business point of view.

Project management includes the art of navigating between the business desire to have working software sooner and the programmer desire to have elegant software. That's not easy, but there are ways to give both groups some of what they need.

Sadly, community-driven development of the free and open source software worlds often lacks this management. We don't lack the tension though. Consider, for example, the debate over whether it's acceptable to release software without documentation. The business argument is "It can provide value to people." The developer argument is "It's not finished without documentation."

This tendency matters less in the F/OSS world as in the business world where your paycheck depends on your ability to deliver working software. What's the worst that can happen? People will move on to a competing project which better meets their needs. (And you thought F/OSS people didn't understand capitalism.) So you annoy your users and drive them away; you're a volunteer, and there are always plenty of volunteers.

... until there aren't.

Sure, that's an extreme position. Though it's easy to trawl through GitHub (and before that, SourceForge) to find the abandoned carcasses of projects which never delivered anything of value to anyone and consequently never attracted sustainable development beyond the whims of the originators, I suspect that it's more interesting to consider the quiet desperation of active projects stuck in not-invented-here rewrite limbo which struggle to achieve usefulness.

What if they spent more time focusing on the value their code should provide to potential users and less time constructing elegant, airy edifices?

It's important to write clean and maintainable code. It's important to focus on quality and craft. Yes, please let us do that. Yet if you want to have real users, shouldn't you also consider how your choices affect them and what that costs them?