The Replaceable Batteries of Your Standard Library

How much should a language's core library contain?

Perl 5 has included the CGI module as long as I can remember. (Module::CoreList suggests a first release in Perl 5.004, released in May 1997.) It has support for the API of a Perl 4 library called cgi-lib.pl, which came around even longer.

You can count on having the CGI module installed on any complete Perl 5 installation deployed in the past decade.

(Even so, many novices asking questions on sites such as PerlMonks still show off their own copied and pasted CGI parsing routines which have somewhere between four and ten bugs, misfeatures, and security holes.)

You can argue that a CGI parameter and HTML generation module was essential throughout the late '90s and early 2000s, but look at the growth of PHP (with much easier deployment than Perl 5) and the popularity of frameworks such as Ruby on Rails (which lets you go from nothing to a database-backed CRUD application in ten minutes) and tell me that CGI.pm as a core library meets the needs of the dominant set of likely users.

I don't mean that people don't use CGI or shouldn't, but that it's worth considering if its presence in the core meets the current needs of the current Perl 5 users and likely Perl 5 users. You can make the argument that there's little cost to including this module for novices and expecting adepts to be able to install something else, but every additional core library adds a maintenance cost.

The thesis behind this site--behind the idea of "Modern Perl"--is that writing Perl today the same way you did in 1997 misses most of the power of the language. Yet if you look at the internals of the CGI.pm module, you see that it reads like the author doesn't know how to write modern Perl.

There's a good reason for this; it's stayed essentially the same in design and implementation since its earliest conception. Fifteen years ago, no one knew the best way to use objects in Perl 5. No one knew the right way to delay loading frequently-unused code to speed up compilation time. It wasn't staggeringly obvious that separating the concerns of HTML generation and argument processing was necessary or good. Perl 5 developers hadn't gone through the pain of implementing, understanding, using, and maintaining an API that wants to be stateful and procedural and OO while reusing almost all of the same code.

This is not a slight on Lincoln or Perl 5 developers in the '90s or anyone. We (and notice that this is a collective noun) didn't know what code we would wish we had written back then. We know better now, and hopefully code we write now will survive better the next decade.

This task of evolution could have been easier, but tradition impedes all but the most serious changes to APIs shipped with Perl 5. Backwards compatibility means never having to say your first attempt wasn't perfect.

It's a tricky situation. A library that was the obvious best option at the time makes sense in a the core library. As time goes on, it becomes less obvious, but it's difficult to change because people are using it and it becomes difficult to remove because people are using it, and new people start using it because there's little friction to start using it, but the people in the know don't use it because it's not obviously the best choice... and you have a world in which people in the know can write really great Perl 5 code and people who start using Perl 5 write clunky Perl 5 code because the whole process makes it easiest for them to write clunky code.

Assume that decent programmers in the know will always be able to improve their ability to write great code. What if we focused on encouraging novices to write great code? (They won't write great code immediately, but we can encourage them to write less clunky code by discouraging them in ways they won't even notice from writing clunky code.)

How do you avoid the problem of effectively freezing good libraries forever by putting them in the core where they're easy for novices to use and too tempting for adepts to ignore?

I think you subvert the whole process and get rid of core libraries.

Next up, how this can work in practice. (For fun, ponder the existence of Perl 6 and its answer to this problem at the language level.)