February 2011 Archives

While I like Cameila (especially as a humorless-stick-in-the-mud-detector), I really like the Perl 5 velociraptor.

While I don't necessarily recommend the book in general, butterflies can be scary, so what appears to be a harmless lepidoptera might eat your brain. The bright colors are warnings against potential predators. Tread carefully. For all of the beauty, you have no idea what the butterfly might do to you.

Of course, Perl 5 is smarter and more dangerous than you think. If you fail to respect it, it will eat the competition.

Imagine if they teamed up, perhaps with a Perl 6 with access to most of the CPAN. Talk about a velociraptor with butterfly wings, or at least warning spots.

Take off your "My primary role as a software developer is the creation of free software for others to use" hat. Replace it for a few moments with your hat of "My primary role as a software developer is the creation of software which provides a good or service to customers." (I assume you don't sell or license this software; the software is a mechanism of producing your product or service.)

Suppose your business operates under a few assumptions:

• Deploying software sooner allows you to generate revenue more quickly
• Your technology choices (language, libraries, and tools) are sufficient to solve your business problems
• Your technical staff has the knowledge, culture, and discipline to write maintainable, deployable code

Does your technology choice matter?

Yes—but what matters more? Code you can deploy today—assuming you can generate revenue from it—is more valuable than code you can deploy next week. Code that exists today—and does what it needs to do—is more valuable than code which may exist.

With that said, quality is important. Code you can deploy today is valuable for today, but if you can't maintain and modify and deploy a new version next week, it's less valuable than code that you can maintain and modify and deploy again and again and again. I've known projects which started out in one language because of the ease of prototyping, then switched from language to language as the project's scope expanded.

A good team with good programmers and good discipline and the freedom and flexibility to refactor and revise the design of the project to make the current version sufficiently easy to maintain and modify while allowing future expansion is in a great position to continue to deliver business value.

The technical requirements are a language with libraries and tools of sufficient quality to allow development, maintainability, and deployment.

"Sufficient", of course, is a management weasel word which hides a great deal of complexity and other assumptions—but that complexity and those assumptions depend on your business needs and your available programmers and....

Even so, some of us have a secret weapon. Given an additional resource, such as the ability to manage deployment yourself (without the limitation of, for example, a $3.95/month shared hosting plan, an internal IT system which mandates the use of WebSphere, or the Silicon Valley requirement that all new startups must make use of monkeypatching), sometimes you can find a dramatic improvement in quality or deployability or deliverability or ease of implementation or clarity of code or not having to write much code at all.

For me, that's Perl. That's why I care about improvements to Perl 5. That's why I care about a Perl 6 in which I can write and with which I can deploy great applications.

Perl is my secret weapon.

Perl 5 has two separate poles around which we users orbit when considering the historical stages of Perl 5 adoption: system administrators and developers.

While explaining the meaning of the phrase "Modern Perl" the other day, I realized that there are two categories of users of Perl 5 when we consider the use of Perl 5.

One category uses Perl 5 because it's ubiquitous. It's already installed and configured (whether as part of the default OS installation or a shared hosting package or a helpful system administrator). It's easy to begin. Copious tutorials abound, the core libraries are sufficient, and Perl's lack of ceremony means that they can write baby Perl and get their jobs done and that's as far as it goes.

You're in the other category, almost assuredly. I am. We use the CPAN. We look forward to upgrading to Perl 5.14 and Perl 5.16. We see something like MooseX::Declare or Plack::Middleware::Auth::Basic and can't wait to replace custom code in our applications with a component other people have written, documented, tested, and maintained. We're comfortable with a community built around the simple structures that have arisen from community consensus regarding the CPAN.

Modern Perl, as I see it, is an invitation for the people in the first group to join us in the second group.

With that said, we must keep in mind the concerns of the first group:

• I'm not necessarily a programmer.
• I'm not necessarily a system administrator.
• I don't understand how to manage dependencies effectively.
• I don't know which modules to use.
• I'm stuck with an ancient version of Perl 5.
• I don't know which book or tutorial to read.

... while removing technical and social barriers to encourage them to join the second:

• Improving the development process and release schedule of Perl 5 to help supported (read "Modern") versions of Perl 5 spread
• Making CPAN distributions available without requiring root access
• Making CPAN configuration and use simpler and faster
• Improving documentation beyond API references
• Replacing XS (and the need for a compiler) with Ctypes for Perl
• Improving the introductory material for Perl beginners

... and more.

Not everyone in the first group will migrate to the second, but given that the evolution of Perl 5 is primarily in the hands of the CPAN these days, we can only do novices a service by making the CPAN ecosystem—code and tribal knowledge—available.

The most recent time I did any significant GUI programming in Java was over a decade ago. Even though Swing had just come out, I used AWT because Swing wasn't available for the GNU/Linux platform to which I needed to deploy my code.

Since those days, Java's had a few releases and a few updates and a couple of owners, and if I were doing GUI development in Java right now, I'd at least consider whatever toolkit Eclipse uses—but my opinion on GUI development in Java is completely out of date and largely irrelevant. If you were to start a new GUI project in Java right now, you would do well not to listen to me for specific details. I may or may not be able even to give a general overfiew of the field because my knowledge of the state of the art is far, far out of date.

Unlike a fine wine, my technical knowledge of Java GUI programming has not improved over time.

I could say the same thing about PHP or Rails or ASP.

Perhaps if we were better at acknowledging the fact that our experience bitrots as the state of the art changes we could have better discussions of the quality and utility of technologies.

(In other words, your experience installing Red Hat 5.2 and using it for 20 minutes is rather irrelevant these days, as is your use of Perl 4 in 1993 or, these days, even Pugs.)

C. S. Lewis wrote something in the Screwtape Letters to the effect that every vice is a virtue misapplied. Doesn't that describe software development?

CPAN is valuable because of the semi-formal guidelines which have grown up around it:

• a culture of sharing
• a first-come, first-served namespace policy
• a culture of testing and quality
• good tools to produce, bundle, install, and verify distributions
• optional but free bug reporting
• a standard documentation format and structure

The common Perl 5 module documentation structure—a name, a synopsis with code examples, a description, and then an API listing—is good for describing a single, standalone module. You can extend it to explain the philosophy behind your code and its use, as something like Test::More does, or you can provide a very simple explanation of a very simple API, such as strict does.

Like many of the good features of Perl 5, the CPAN documentation format borrows liberally from a feature of Unix: specifically, the format of man pages.

Like many of the not-quite-perfect features of Perl 5, features borrowed heavily from Unix show their limitations. In particular, the biggest flaw of the API-heavy documentation structure is that it's only good at documenting APIs and its passable at best at describing the important relationships between components of the distribution which provide APIs.

In other words, if you want to understand DBIC and use it effectively, you must understand the relationship between result objects and resultsets as well as the separation of responsibilities between them. (DBIC deserves credit for recognizing and addressing this documentation gap by providing multiple pieces of documentation beyond API explanations, but the problem is difficult to solve.)

You can see the problem more dramatically by exploring something like HTTP::Response, which is part of a well-factored and robust distribution but which has the relevant information spread through several separate pages of documentation because the structure of the documentation reflects the inheritance structure of the code.

Again, you can't fault Gisle for factoring his code well and documenting it effectively in the dominant style of the time (and Gisle deserves much credit for helping to spread the dominant style of documentation)—but isn't it ironic, a little bit, that where objects should be well-encapsulated black boxes that their internal design decisions are so apparent in the documentation?

Some will suggest that we need better ways to explore the documentation, but I believe there's a bigger question we need to answer. When I'm flipping through all of the documentation for DBIC to figure out the best place to put validation code to ensure that my objects always get created in a known-valid state, I need something far different than a list of methods used to search for objects that already exist. I know what I want to do, but I don't know which module's documentation holds the right answer.

I don't know what the solution is.

Update: Your author disclaims the contents of this post and leaves it here only as a historical curiosity.

Last July, I wrote An Accurate Comparison of Perl 5 and Rakudo Star. Despite almost seventeen years of tuning and optimization, there's no huge difference between Perl 5's startup time and memory consumption when compared feature-by-feature to Rakudo Star.

The same holds true today. In fact, Rakudo's startup time in my most recent measurements is better than Perl 5.12.2—and bigger performance improvements are coming to Parrot in the next few months.

Perl 5.12.2's startup time, loading all of those modules except Parrot::Embed is 0.627 seconds at best. Rakudo's startup time is 0.441 seconds at best. That means Rakudo starts 30% faster than a similarly configured Perl 5.

Rakudo's memory use is still higher; from the REPL, it uses 176,780/130,968 KB virtual and resident memory. Perl 5's usage is 141,204/47,884 at the debugger. Memory parsimony is probably the next area of optimization for Parrot and Rakudo.

Of course the Rakudo and Parrot developers can't optimize for your needs without knowing your needs. Improving synthetic benchmarks may or may not improve real programs. You can help these efforts by writing real programs, by testing real programs, and by finding areas where real programs seem slower or more memory hungry than they should. In particular, I believe the parsing component of the Rakudo ecosystem has several undiscovered bottlenecks which could yield notable improvements.

Please see #perl6 on freenode and #parrot on irc.perl.org for more details.

You asked, and we (eventually) delivered.

Modern Perl, the book, is now available in ePub format as well as A4- and letter-sized PDFs.

There is no DRM and there are no special licensing terms. Please redistribute this book far and wide and tell everyone that they too can write great Perl 5 code that takes advantage of all of the hard-earned wisdom and effort we've put into improving the language in the past several years.

Of course, you can always tell people to buy the printed copy—I'm a fan of paper—but the important part is spreading the word, whether through reviews on Perl Monger lists, web sites like Slashdot and Ars Technica and LWN, and of course booksellers. You're also very welcome to upload and seed torrents, post it on ebook sites, or just hand around USB sticks whenever you get together.

Bilingual Perl hackers have already begun translations into other languages, and we're very happy to see this and help in any way possible.

If we're going to show off how great Perl programming can be, let's make sure everyone has a chance to read a good book about it.

What's the future of Perl? Let's look ahead five years to the possibilities.

(Note: like all intelligent, attractive, and good-hearted people, I use the term "Perl" to refer to the entire Perl ecosystem and all versions of Perl. If you want to argue that "Perl" has, now and forever, always meant only "Perl 5", please set your time machine to late 2000 when that argument was relevant and, albeit barely, worth having. If you don't have a time machine, consider that a decade is long enough to have learned some coping skills and rethink your priorities.)

Who Uses What

• No one uses Perl, any version, for new projects.
• Some people use Perl 5 and some people use Perl 6.
• Most people use Perl 5 and almost no one uses Perl 6.
• Most people use Perl 6 and almost no one uses Perl 5.

What do they use it for?

• New development.
• Maintenance of existing projects.
• Porting from other languages.

What's the relationship of Perl 5 to Perl 6?

• One has subsumed the other.
• One has failed and the other has survived.
• Perl 5 is becoming Perl 6 as much as possible.
• Perl 6 allows you to use Perl 5 in process.

What New Features Does Perl 5 Have?

• A foreign function interface
• A real object system with syntax and everything
• An improved exception system
• An improved concurrency system
• Function signatures
• Grammars
• Junctions
• Strictures by default
• None of these
• All of these

Do You Still Use Perl?

• Yes
• No

Is Perl Healthier Then Than Now?

• Yes
• No

With all of your answers, you have a decision to make. Do you like this future? If so, great!

If not (and here I admit to some concern myself), how can we make the future more appealing?

Are you a good Perl 5 hacker?

Do you know how to access the symbol table yourself, without resorting to string eval? Do you know how to find all of the variables within a package or all functions within a package?

This metaprogramming—or reflection—is a fundamental part of higher programming languages. While all symbolic languages intend to operate on parametric symbols instead of raw data (that's why you can write equations in programming languages instead of using your computer as a glorified adding machine), higher languages operate on language constructs as well.

You can see this in lower languages such as Java, where the only good way to make flexible and extensible programs is to generate big wads of XML which control your AbstractFactoryInterface instantiations such that the classloader can inject the right code to make the language do what you need without forcing you to write even bigger wads of repetitive boilerplate Java code.

(During a consulting job several years ago I had to generate 60+ unique reports with Crystal Reports. The reports differed by way of two column names, but Crystal Reports wasn't flexible enough to let me define one report once, then pass in the two parameters I needed. This is why 4GLs failed.)

Metaprogramming and parameterization and flexibility beyond "This is how you write programs in this language, darn it, and you shall never think of doing anything different that the language designer didn't have in mind." may be difficult to implement well and it may not completely satisfy everyone, but it's useful and necessary.

Consider Package::Stash. If you couldn't remember all of the answers to all of the questions in the first paragraph, you'll probably find P::S much easier. (Certainly it's less buggy and much less annoying than munging the symbol table by hand.) Use it when you need it. It's nice. It's Perlish. It doesn't look like the dog's breakfast of random punctuation that you need to manipulate symbol tables (which are even uglier than the standard dereferencing syntax, because they're special hashes).

Now that I've explaind a bit of programming language design philosophy, explained a dead end in programming language design, and pointed you to a useful tool from the CPAN, let me get to the point:

There's no excuse for something like Package::Stash not being core behavior of Perl 5.

Think about that for a moment.

How can you defend Perl 5 as a practical language, one which makes easy and hard things possible, when standard higher language behaviors such as symbol manipulation are so eye-gougingly ugly and esoteric by syntax that they are out of reach of even adept Perl hackers? These are not black magic concepts (else MIT wouldn't have taught freshmen Scheme as a first language). They are fundamental to a proper understanding of symbolic computing and practical programming.

I know the arguments against subsuming something like P::S in the core though, and they're all nonsense.

• You can already do all of these things with the existing syntax. You can also store relational data in flat files instead of an RDBMS. That doesn't justify anything.
• Not everyone will like this syntax. The core should come up with something everybody like. Nonsense; the core should provide one really good way to do something (and the "They're just hash references with special keys and some other bits glued on!" approach isn't it) without precluding alternate approaches. Refusing to do something without complete consensus is as much a recipe for failure as adding every feature proposed without considering its necessity, utility, implementation, and coherence in the gestalt.
• What about all of the existing code that doesn't use it? Irrelevant; it still works. That code's already written. The cost of writing that code is a sunk cost; you'll never get it back, no matter how much you penalize code under current maintenance and new code written from this point forward.

There are plenty of dependencies of popular CPAN distributions which do not represent failures of the Perl 5 core to provide useful features in a usable way, but consider this: every time you have to install or load one of several competing exception or function signature or list handling or library management or build systems or FFI workalikes or object system modules in a large chain of dependencies (of which a transient bug in any costs you debugging time), you're paying a tax the core developers have levied on all users in lieu of subsuming useful and necessary language features into the core of Perl 5 itself.

Things have improved in the past ten years (and especially the last 18 months), but this is why non-Perl programmers look at Perl 5 and say "Modern? That's what you call modern?" All of the rounded corners and hip slogans and fauxhawks and body spray in the world won't change that.

Of course, it's difficult to criticize p5p for its conservatism: no one has stepped forward with a coherent and singular vision for the continued and managed improvement of the design and implementation of Perl 5 as Perl 5—nor is it clear that p5p would accept anyone other than Larry doing so.

Is there a solution?

Suppose you want to see how a hypothetical Perl 5.16 with a method keyword might work. Reading the patch to add method to Perl 5 isn't as fun as using it yourself.

If you're an old hand at patching and building Perl 5, you already know what to do. That leaves about a million of the rest of us.

First, check out a recent bleadperl from Git. Within that directory, check out revision 8bdb331 (as it's the basis of the patch). You can use later revisions, but you may have to massage the patch yourself:

$ git checkoout 8bdb331

Now download the method patch and apply it:

$ patch -p1 < 0001-Added-use-feature-method-and-tests.patch

Everything should apply cleanly. Now you need to configure Perl:

$ sh Configure -de -Dusedevel -Dprefix=$PERLBREW_ROOT/perls/perl-5.13-method

If you don't use perlbrew, you should. With this configuration you can install bleadperl with this patch as if it were a released Perl 5, at least as perlbrew sees it. Change the value of -Dprefix to match your preferred installation directory. After a few moments, you can regenerate the parser and other generated files, then build, test, and install the new Perl yourself:

$ make regen regen_perly
$ make test
$ make install

If you have trouble building or installing bleadperl, read the included INSTALL file. I cannot provide technical support here; the usual Perl 5 support channels apply (especially if you also have trouble building, testing, or installing bleadperl without this patch).

Then you can use perlbrew to switch to the new Perl 5 (perlbrew use perl-5.13-method or perlbrew switch perl-5.13-method) and test it with your favorite code.

Update: improved instructions per oylenshpeegul's comments.

Thanks to IBM dW and Seda Özses for fixing the code to which I linked in CGI is Okay, but Bad Code is Irresponsible. To their credit, the security flaw and the Perl 4 style code in the article have become much, much better—now it's good example code, which is what everyone in this process wants.

This sort of thing will happen again. Perhaps we can learn from this experience and handle it better next time.

What should you do the next time you see example code with a security flaw and/or clunky style?

• Contact the author directly and privately.
• Explain the security flaw and demonstrate how to fix it and it alone.
• Expand further on stylistic issues (explaining that, although there is not complete agreement in the entire Perl 5 community over some stylistic issues, the dominant trends are fairly widely understood as something we can all call modern or enlightened Perl), revising the code as necessary.
• Suggest that the Perl 5 community has many resources such as PerlMonks, Perl Mongers, and mailing lists full of people who can and will perform technical review on code.
• Be polite.
• ... if that does not work, publish an explanation and corrected code elsewhere.

Where possible, stick with the constraints of the article. For example, Ms. Özses's code has good reasons to use CGI instead of a Plack-capable module. Respect that. (Though where it's obvious that a CPAN module is far superior solution, such as in the case of CSV parsing, do recommend the module.)

We share a goal, that useful and usable and correct and secure and sufficiently elegant code is available. We do not have to write the same code nor the best possible code to achieve that goal, but...

... people who publish code examples have a responsibility to write good code.

... people who review and comment on code have a responsibility to be both accurate and polite.

To that end, I owe Ms. Özses an apology. However I intend neither personal attack nor insult in my writings here, I failed to live up to my personal standards of kindness and politeness in this case. She deserves much credit for taking the suggestions of various Perl 5 programmers despite unwarranted criticisms.