April 2009 Archives

If you've followed Ovid's use Perl journal recently (and you should), you've seen a lot of practical discussion over the use of Perl roles in his work at the Beeb. There's also plenty of theoretical discussion.

I joined the Perl 6 design team in 2003. I spent a lot of time thinking about problems in every object system I'd ever seen. When we released Apocalypse 12 in 2004, we included a new feature called roles. The design borrowed from a Smalltalk idea called Traits, but (as usual) we Perl people have our own take on things.

Rakudo Perl 6 provides a very usable implementation of roles, and the Moose object system for Perl 5 allows you to use them as well.

Roles are different from so-called traditional class-based or prototype-based or even multidispatch object systems in several important ways.

Role Goals

In my mind, object orientation provides two essential features: encapsulation and polymorphism. A well-designed object oriented system models domain concepts along well-defined boundaries, hiding internal details from the outside world and treating like concepts as like concepts because they share the same interfaces.

Careful readers will note that the word "inheritance" does not appear in that paragraph. That is no accident.

Polymorphism is an interesting concept. I see it as providing genericity and extensibility. I may not know all of the potential uses for an object when I create it, but I need to have an accurate idea of how the object behaves. I need to be able to name that collection of behaviors. If I've created the object well, the collection of behaviors the object provides has a meaningful and understandable name.

A role is a name for a discrete collection of behaviors.

When you want to perform operations on an object, you need to know what kinds of behaviors that object supports. Is it a GUI widget? Does it represent a customer? Can you introspect it? Can you serialize it? Does it know how to log debugging information?

In effect, you're asking the object "What do you do? Do you perform this role?"

Once you start asking that question, you can (and should) stop caring about how the object performs its role. I've said nothing about inheritance, or delegation, or composition, or allomorphism. Those are mechanisms. Those mechanisms should be well-encapsulated inside the object where you can't poke or prod at them because they're none of your business.

The important question is not "Do you have a method called log_debug_output?" or "Do you inherit from DebugLogger?" but "Do you perform the DebugLogging role?"

That's subtle, so let me repeat it a different way. If you write code mindful of roles and you don't know the specific class of an object you receive but you want to call a method called log_debug_output on that object and have it behave as you expect, you want to check that that object performs the DebugLogging role. It doesn't matter how the object has that method. It could inherit it from a superclass. It could mix it in from a collection of unbound accessory methods. It could delegate it to a contained object or proxy it to a remote object. It could reimplement the method directly. It could compose it in from a role. It doesn't matter how the object has that method -- that's none of your business outside of the object -- only that it does have the method, and that it understands that method in terms of the DebugLogging role.

That last part is also subtle. The duck typing hypothesis suggests that method names alone are suitable to determine appropriate behavior. Roles avoids problems there by requiring disambiguation through naming. A TreeLike role's bark method has an obviously different context from a DogLike role's bark method.

Roles allow you to express (or require) context-specific semantics, especially when combined with your type system. A role-aware type system allows you to express yourself with better genericity: as long as you hew to a well-defined interface and do not violate the encapsulation of objects, you can enforce well-typed programs based on the specific behavior of objects regardless of their implementation.

This is very theoretical. Don't worry; I'll show specific examples in future entries.

Role Features

Roles are more than just tagged collections of behavior. You can think of them as partial, uninstantiable classes.

Roles can provide default implementations of behavior they require. By composing a role into a class, you can import its methods (and other attributes) into the class directly at compile time. There are rules for disambiguation and conflict resolution and requirements and provisions, but you get the rough idea. A role also provides a mechanism for code reuse.

Parametric roles take the concept even further by customizing their composable behavior based on arguments provided at composition time. They're intensely powerful.

The final -- and perhaps most subtle -- feature of roles comes from building them into your type system. Every class implies the existence of a role. If you declare a DebugLogging class, other code can access the DebugLogging role. They may not be able to compose in behavior from that class -- unless you write the class to make that possible -- but they can say that they perform the DebugLogging role, with all of the concomitant role composition checks to enforce this, to produce an object which may successfully substitute for a DebugLogging object anywhere that expects a DebugLogging object -- even though there's no formal relationship between the original class and the class which performs its role.

As I said, this is powerful but theoretical. Tomorrow I'll discuss Perl roles versus inheritance.

In A Lesson in the Importance of Language Design, Zbigniew Lukasiak raised the question "How do we argue that this API is more elegant than that?"

It's a good question. There's more art to it than science. Yet good art always includes craft, so I believe that it's possible to identify several principles which contribute to elegance. The first is conciseness: saying a lot with a few words.

The Inelegant Way

This code for declaring a class which inherits from a parent and overrides methods in Perl 5 is idiomatic and familiar to many Perl programmers:

package Some::Class;

use Modern::Perl;

use base 'Parent::Class';

# or

@Some::Class::ISA = 'Parent::Class';

# or if you're paranoid

BEGIN { @Some::Class::ISA = 'Parent::Class' };

# or

use vars 'ISA';
@ISA = 'Parent::Class';

# or

our @ISA = 'Parent::Class';

sub foo { ... }
sub bar { ... }

1;

I've written before about how many Perl specific concepts you need to understand to write or comprehend this code. There are several -- a package is a class, the package global @ISA contains all parent classes in method resolution order, the base pragma has certain advantages and disadvantages, methods are subs with an invocant, etc. You don't have to know all of these to write modern Perl, but you should be familiar with the underlying mechanisms to some degree. What's atop is just syntax.

(Careful readers might note that parent may supplant base as the preferred inheritance pragma, for various subtle reasons.)

Would you call this code elegant? There's a simplicity to it, but a lack of uniformity. There's an orthogonal minimalism to other parts of Perl 5. Is it elegant?

Consider an alternative.

The Concise Way

use MooseX::Declare;
use Modern::Perl;

class Some::Class extends Parent::Class {
    method foo { ... }
    method bar { ... }
}

Can anyone deny that this example is more concise than the first example? Reading it is straightforward, even though the underlying mechanism is the same. The differences are merely syntax.

The second example has little or no superfluous concepts. It does not expose the underlying mechanism. It does not (ab)use other Perl concepts (such as the import() mechanism of use) visibly. It reads declaratively. Is there anything in the code that's unnecessary?

It's concise.

Not all elegant code needs special syntax, but sometimes a little bit of syntax can make a lot of inelegance just go away.

First, read David McLaughlin's Ugly Perl: A lesson in the importance of API design, and the comments. The piece itself isn't short, but it's worthwhile. The most important high level concept is near the start:

What we took away from that evaluation process is that, more than choice of programming language, more than choice of development paradigm and more than choice of development methodology, good API design is the single most important factor in how developers perceive the quality of code. Introducing the code review process to our team only confirmed this: almost every review I've taken part in has revolved around naming conventions and style rather than performance or implementation.

David's choice of jQuery as an example illustrates that point nicely. Solving a problem in an elegant way is sometimes compelling enough that people will switch technologies, even if there are other drawbacks. (I admire how little effort it takes to deploy a PHP application, and how little code you have to write to create a CRUD application in Ruby on Rails. There may be big flaws in both approaches beyond that point, but each technology solves one problem in an elegant way.)

The opposite argument also applies. A solution which looks clunky may turn off people:

package MyDatabase::Main;
use base qw/DBIx::Class::Schema/;
__PACKAGE__->load_namespaces;

1;

One of my favorite modules is CLASS. It's really simple and really stupid, but look how much prettier that code sample is:

package MyDatabase::Main;
use CLASS;

use base 'DBIx::Class::Schema';
CLASS->load_namespaces;

1;

Okay, I also replaced the bletcherous qw// syntax with something less syntactically noisy, but isn't CLASS easier to read than __PACKAGE__? What if my class patch had been accepted?

class MyDatabase::Main is DBIx::Class::Schema
{
    use CLASS;
    CLASS->load_namespaces;
}

None of this is rocket science. It's barely any work modifying the Perl 5 parser (at least once you accept that sometimes adding features is worth the pain of telling people with 15 year old code that they have to add a single line to make it forward compatible). The changes themselves are very simple. They're surface changes. They're just syntax. The guts of Perl don't have to change at all.

Yet what a difference a little bit of syntax makes. As the first commenter on Unshortening URLs with Modern Perl wrote, "Modern Perl looks sexy!"

David's absolutely right that beauty and elegance are important to APIs. How much more important are they to languages. In my mind, that's what makes iterating languages so important: how else can you help people write beautiful code?

I spent a lot of time in 2003 and 2004 thinking about object orientation. Specifically, I wanted to find a better solution to duck typing, fragile base classes, deep inheritance hierarchies, cross-cutting concerns, and code reuse.

A talk by Dr. Andrew Black led me to a paper which described actual results of what I'd been designing in my head. After some discussion with the Perl 6 design team, we developed the concept of Perl 6 Roles.

In those days there was no Moose. I and several other people wrote proof-of-concept modules to demonstrate how to use roles in Perl 5. Now Moose is the state of the art in object systems, and it's definitely the way to use objects in modern Perl.

... with one exception.

Never Wrong By Design

My colleague and co-author Curtis "Ovid" Poe has written about his concerns about "silent" overriding of role methods with declared class methods. (If you haven't read my explanation of roles already, you're already lost. You just won't know it until the next paragraph.)

Ovid's code has several roles, which each provide several named methods. This is rather the point of roles: they're named collections of useful behavior and attributes. They don't stand on their own. You build up classes from collections of roles. The only relationship between roles is when you apply a role to a class (or another role, but if you haven't used roles before, ignore this parenthetical note).

Ovid declared a class and applied several roles to that class. That's normal. Then he declared several methods within that class. That's also normal. Note that, at this point, the available methods of the class come from multiple places. Some of them come from the class declaration. Others come from the applied roles. If the class inherits from other classes, other methods may come from its parent classes.

This is all normal. This is all by design. The Perl 6 Roles and Parametric Types specification says:

A class's explicit method definition hides any role definition of the same name. A role method in turn hides any methods inherited from other classes.

The method resolution is:

1. Methods defined explicitly in the class.
2. Methods defined in roles composed into the class.
3. Methods defined in superclasses.

Role composition is somewhat special, at least when compared to inheritance. Because composition is declarative, the compiler can resolve dispatch to methods at compilation time. That is, if you have two roles which declare methods of the same name, the compiler will produce an error message: you must resolve the conflict explicitly, by writing your own redispatch mechanism, excluding one variant method, or providing your own implementation of the method.

That last possibility gives away the punchline, if you read it closely.

Ovid's code composes several roles into a class which declares its own methods. At one point, someone added or renamed a method in a role or added or renamed a method in a class such that the class method took priority over the role method he expected.

At that point, he argued (successfully, it seems) that Moose should give a warning when role application behaves as specified in Perl 6.

When Warnings Attack

I don't want to argue the utility of the design, and I'm not going to discuss why adding a warning is wrong (it is, but I won't argue it directly).

It's more important to discuss good and bad examples of warnings and error messages.

Consider the (sadly optional) error checking when perl says "You used strict, but you never declared this variable!" This is useful because it tells you something about your scoping, hints that you may have made a typo in a name, and tells you where the problem was.

Consider a hypothetical warning that you've used a declared variable only once; perhaps its a parameter to a function which you increment, or decrement, or read -- but in all of its scope, you only access it once. Is this warning useful? Perhaps it indicates vestigial code that you can safely delete. Perhaps not, though: what if someone has passed a tied or overloaded object which does something useful when you read a value or increment or decrement it. That's likely bad design with all of its implications for weird action at a distance, but the compiler can't know that statically. Is the potential value for warning about vestigial code which a decent compiler can optimize away worth the potential cost of warning in the wrong circumstances?

Consider the "use of uninitialized value" warning, enhanced in Perl 5.10 with the name of the variable containing an uninitialized value. The warning suddenly became an order of magnitude more useful, as now you can check your assumptions without performing lengthy debugging sessions to determine which variable combined with which set of input produces the unexpected condition.

Of course, I know of several codebases which disable this warning. Sometimes explicitly initializing several strings to empty (not undef, but '') is too troublesome -- Scalar::Util's reftype() function is one of my least favorite offenders here. Then again, in a previous job where I worked with Ovid, I spent refactoring time over several days removing all potential uninitialized value warnings from a piece of code so that we could see actual problems in our code.

Is that warning valuable? Sometimes.

Consider the warning against using symbolic references in Perl 5. It's optional, but unless you're Damian Conway, Abigail, or Tom Christiansen you'll hear a lot of complaints for not using the strict pragma. For the most part, this is good advice -- it's all to easy to write code that misbehaves in difficult to debug ways without a little help from the compiler, especially when you're a novice programmer.

However, Perl 5's metaprogramming support usually requires symbolic references (but thank goodness for Moose, which alleviates this). In other words, the compiler can only determine whether you intended the behavior by the presence of the declaration of use strict; or use strict 'refs';.

The question is whether the warning is useful enough for novices and the warned-against behavior is rare enough even for experts that the cost of disabling the warning lexically is small enough to justify the warning. Besides that, it's not enabled by default. The feature -- symbolic references -- work as designed and specified, but in general they're useful only to people who know how to disable the warning.

(One flaw in my argument here is that symbolic references are also available to everyone not yet clueful enough to ask perl to detect potentially dangerous behavior -- but turn the idea around and ask why, in 2009, this should still be.)

Role On New Warnings

When analyzing the benefits and drawbacks of new warnings, I believe it's important to evaluate at least three criteria implied earlier:

• How valuable and accurate is the warning?
• How common is the warned behavior actually dangerous?
• How onerous is working around the warning?

In the first case, perl cannot determine the source of a collision between a method declared in a role and a method declared in a class. Is it a typo? A renaming gone bad? A false cognate? A true cognate? Deliberate behavior?

One proposal is not even to try, as if "Hey, something went wrong!" is a useful warning message. If that were the case, adding the name of the variable containing an undefined value to the "Use of uninitialized value" warning would have been of low utility.

In the second case, perl can only guess whether a locally declared method exists for disambiguating colliding methods from multiple composed roles or does something else entirely. The names of roles are as meaningless to the compiler as are the names of methods. It doesn't care. The intent of code is useless to a Universal Turing Machine Simulator. It's only useful to people -- so useful that only crazy people with something to prove write directly to UTMs.

The real question is "Why is there a collision?" Did someone misread or misunderstand the documentation for a role? Did someone make a typo? Did someone upgrade part of the system? None of this context is available to the compiler. It can only guess -- and for it to guess correctly, someone needs a statistical model of the world which demonstrates that people rarely override composed methods with locally declared methods.

You must actively debug the problem to find the source of the conflict. Which methods did each role provide? Where did all of the methods come from? What changed recently? Compilers are notoriously bad at history and change management. Fortunately, there's a perfectly good field of study which deals with change management; it's called change management.

The third case is the least important. Then again, it might be the most important. How often is this a problem? Typos in variable names are often a problem, and the compiler can detect them trivially. Metaprogramming is rare enough that enabling symbolic references in a small scope is rarely onerous (though to be fair, it's all too easy to disable them in a greater scope than you anticipated, if you assign an anonymous function to a typeglob and don't re-enable strictures within the function).

The best reason against this warning is that it thinks of roles as mixins instead of composable types, but that's a very different discussion. A class which composes several roles to take on the types of those roles instead of to reuse code -- that is, allomorphism -- may produce many, many of these warnings. Another proposal is to allow explicit exclusion of lists of methods when composing a role, as if explicit declaration of methods in two places (first when composing the role and second when declaring the methods in the class) were an improvement.

Optional or Not

One facet of this discussion rarely seen is whether a warning should be optional or mandatory. The cost/benefit tradeoffs change dramatically when you flip this switch.

Optional typo and symbolic reference checking in Perl 5 has helped a lot of bad code spread -- even the proliferation of global variables is a net drawback. It's also cost a lot of people a lot of debugging time. Perhaps it's helped more people write one-liners, but when was the last time you spent hours debugging a one-liner?

Compare that to optional Perl::Critic warnings. If your project decides that a method length of more than 24 lines is a code smell, you can determine that yourself. If you decide that a specified, implemented, and tested feature of Perl 5 is troublesome (tie), you can disallow it yourself locally.

You have the information. You know your history. You can improve the heuristic.

The first solution to bridging the gap between The Two Worlds of Perl Deployment is to segregate system Perl paths and application paths. As Zbigniew Łukasiak mentioned, one fine way to do this on your own is with the local::lib CPAN module.

The second part of the solution addresses a more subtle problem.

Language Feature Freeze

What would happen if you added a new method to Perl's UNIVERSAL package? I've run into that. I added the DOES() method, which went into Perl 5.10. Though isa() and can() already existed, with lower-case names, the pumpking argued that the possibility of a collision user-code which itself defined a does() method anywhere was too great to ignore. Thus the official way to check that a class or object performs a named role in Perl 5.10 is $invocant->DOES( 'role name' );.

What would happen if you added a new keyword to Perl? I proposed a patch to do just that, adding a class keyword to Perl 5. The patch is in limbo, even though it has tests and adds a nice new feature backported from Perl 6 and does not interfere with the existing test suite.

Of course, even if the patch ever were applied, you'd still have to enable the feature explicitly with use feature 'class'; or use 5.012; or even my own use Modern::Perl;. That's right: you don't get simple, declarative, compile-time class declarations in Perl 5 by default. You must explicitly request that the language help you.

That doesn't seem very Perlish to me, either.

Why is this a problem? What if someone from the "Must Never Change, Darnit!" camp wrote a Perl program in 1993 that needs to run, unmodified, today on Perl 5.10 (or 5.11 or 5.12)? What if that code defined a class function? (I hate writing parsers, but I'm decent enough at them that it only matters if that function has a prototype which makes it take a single function reference.) The stars just might align such that a sixteen year old program might behave differently under a modern version of Perl.

In other words, a modern version of Perl now, by default, behaves syntax-wise like a version of Perl released fifteen years ago.

I exaggerate slightly -- the our keyword is relatively new (only nine years old).

Old By Default is Wrong

The feature pragma exists so that you can enable new features in code which uses them. This is great if you don't want to touch code you wrote in 1993, but it can be a little tedious if you want the compiler to warn you about typos in variable names (a feature from around 1994), enable warnings about dubious constructs in specific lexical scopes (a feature from 2000), and so on.

The problem with the feature pragma is that it's exactly backwards. It freezes Perl's feature set to that of around July 2002 (give or take). Anything added in the past nearly seven years gets lumped into an alphabet soup of features you must explicitly enable. If you don't know that magic incantation, you don't get that feature. If you want to explain how to use modern Perl to a Perl novice, you have to deal with a big block of magic code they won't understand yet. So much for only essential complexity.

Remember that the goal is to ensure that code written before a feature was available even when running on a version of Perl which includes that feature.

A better alternative is to have feature limit the features you use. That is, a program written in 2002 for Perl 5.8.0 should specify that it uses only those features available in 5.8.0. That is, any code explicitly, declaratively, and lexically identifies the specific set of behavior it expects Perl to provide. Any code without such a declaration uses the default set of behaviors from the running version of Perl: by default, everything new.

I realize that this may require editing old code, but the problem's not nearly as bad as people make it sound. It won't affect most programs. The feature pragma currently enables only a handful of features. I can't search the DarkPAN to verify this, but I believe the chance of collisions is small. It also likely only affects programs running on Perl 5.12, as that's the first Perl version likely to include this.

The benefits are, to me, compelling. Couple this with library path segregation, and Perl 5 becomes much more robust in the face of CPAN installations and OS upgrades. Modern features are available to everyone, and not just the elite who understand how to enable them. Backwards compatiblity and future proofing becomes a declarative exercise in defensive programming. What's not to like?

I promised to offer two partial solutions to the gap between "Stable at all costs" Perl and "Please join us in the 20th century" Perl in The Two Worlds of Perl Deployment. The first solution already exists, technically. I haven't see anyone take advantage of it yet.

First, please read Jess Robinson's The two aspects of Perl for a different explanation of the same problem. (I should have linked this from the first article, because it helped me crystallize my thoughts.)

Localizing Change

One of the most persistent lessons of language design I've learned from Perl 5 is to avoid change-at-a-distance. Magical global variables are troublesome in Perl 5 -- not primarily because they can be difficult to read and remember unless you use them all the time, but because they're global to your process. One of my favorite features of Perl 6 that people will notice only in the absence of problems is that many of the lovely and useful features of Perl 5 are still present but now have proper lexical scoping. Autoflush is now a property of output filehandles. So is the input record separator.

The principle of localizing change applies also to Perl installations. The Mac OS X Perl upgrade problem comes from two different packaging systems trying to manage the same files. Similar problems come from upgrading other core modules, such as Scalar::Util.

Localizing Libraries

While the right long-term solution is Hanging the Core out to DRY and removing everything from the core, a simpler short-term solution exists and works today.

Perl 5 allows you to specify alternate installation paths for Perl libraries -- including core libraries. This is useful for installing CPAN modules locally when you don't have root permissions on a box. You can specify additional directories on the command line with the -I flag, add directories to the PERL5LIB environment variable, add code to the sitecustomize.pl flag (see the sitelib member of the Config module, or add use lib 'foo'; lines to a program.

OS vendors who package and distribute Perl and use it for core utilities should install their core libraries in a specific directory accessible only by their core utilities and they should configure Perl to install modules from the CPAN in another directory where their utilities do not look.

That is, if you distribute Perl as part of an OS or distribution, and you distribute Perl utilities which the system needs to function, you must pay attention to specific library versions and ensure that the proper library versions are available, even if users install newer versions from the CPAN.

Allowing users to install new versions which overwrite old version which an OS version might downgrade or sidegrade invites the type of trouble Apple users have seen lately.

This is merely a configuration problem, though the Perl 5 Porters should provide a simple guide for packagers which explains the solutions. (Note that pumpking Nicholas Clark has worked on reordering Perl 5's library search order to ameliorate some problems here.)

Similarly, admins and other Perl users who wish to enforce stability should consider segregating library paths for individual applications as necessary. A little bit of disk space is a cheap way to reduce unintentional collisions.

Perl's history shows two spikes in adoption rates. The first came early in its life, when system administrators realized that Perl could replace homegrown and ad hoc polyglot scripts written in shell and AWK and Unix command line utilities. The second came several years later, after the WWW and CGI revolutions, when programmers recognized that Perl made a great language for dynamic web sites.

Perl has expanded into other application domains, but it's still a bicameral (not sorry for the pun) language. I've pitched this tent clearly in one camp, but I want write clearly about the other camp and why it doesn't concern me.

Stability and Predictability

Many Perl-using system administrators fit into this camp. They have a lot of mottos, such as "You can't break existing code!" and "Slow and steady change is fine!" Upgrades should be infrequent and predictable. Some of these people have code they wrote a decade or more ago that runs, unchanged, on the most recent version of Perl.

The Perl 5 development process has catered to this group since its inception.

The goal of this group is to ensure that old code -- never touched -- always works the same way. It's easy to mock this position, as some of its adherents are truly crazy in expecting that people running a decade-old release of Perl should expect that a CPAN module written in 2009 will install and run successfully, but there's some validity to this position.

Take, for example, the all-too-common example of the Mac OS X update breaks Perl story. Perl is useful. Apple distributes several utilities which rely on Perl. When Perl-savvy developers make routine changes and perform routine upgrades, the OS gets caught off guard. Something changed. Please note that this problem is not particular to Apple. It's affected many other operating systems and distributions, both proprietary and free.

The problem with this approach is that it's fundamentally incompatible with the other approach. Expecting that nothing will change when you change something is a good way to discover that making a change may, in fact, change something.

Again, I'm mocking an extreme position. I do find it reasonable to preserve some degree of backward compatibility for a reasonable period of time, but that line of thinking leads to the question "How do you solve this problem?", which I'd like to delay until the next entry. The real point of contention between these two camps is on how to deal with change.

(This question is at the heart of Masterminds of Programming -- Powell's affiliate link -- which asks it of the designers of several influential programming languages.)

Improvement and Abstraction

The other school of thought believes that change is necessary to improve the language. The Perl 5 development process has made some concessions to this view over time as well. The module system and the CPAN itself demonstrate the value of a loosely-organized and self-organizing language extension system.

The goal of this group is to improve the language and its ecosystem based on feedback from real-world usage. What's good? What's difficult? Which changes would make the language and its libraries easier or faster or more pleasant or more powerful? Which new niches can and should the language target to improve the lives of programmers or gain new ideas?

Another good question is "Are programs getting easier to write, over time?" (Try Moose for an obvious improvement over Perl 5's default OO.) A scary question is "Are the internals getting better over time?"

There are very good reasons Perl 5 has trouble attracting new developers, and they're the same reasons that Perl 5 doesn't have a JIT or an advanced garbage collector or ports to other VMs -- not that I'm bitter. (I have patched the Perl 5 parser, however. If that doesn't give my authority some appeal, I'm not sure what would.)

The Real Question

No one sits completely in one camp or the other. They're poles. This is a spectrum. No one seriously argues against fixing any bugs because people might be relying on them. (Some people seriously argue against fixing some bugs, because someone might be relying on them, however. Drives me crazy.)

My stance should be obvious. I believe that it's possible to design and build a working, reliable system which undergoes constant improvements at regular intervals such that upgrading frequently is boring and predictable. Other projects have done this. Other projects in the Perl ecosystem have done and continue to do this.

The most important question is not "Should change occur?" but "How can change occur to minimize disruption to the first group and meet some of the goals of the second group?" I have two practical, technical details to explain in my next two entries.

I normally talk about Perl 5, but many of the ideas and discussions here will apply equally well to Perl 6. (I hope we'll throw out the bad ideas, embrace the good ideas, and make some new mistakes no one has ever made before.)

I have a belief in relentless progress, where a process of thoughtful reflection with frequent feedback and rapid iteration can produce amazing results in a reasonably short period of time. FLOSS projects -- community-driven FLOSS projects, at least -- have a strong advantage in that their progress is public. Sometimes, that progress is very public.

Rakudo's status page features a prominant graph. Near-daily updates of this graph show the relentless progression of Rakudo to passing the complete Perl 6 specification test suite. If you look today (or view a stunning commit to the status graph data, you'll see that Rakudo has jumped from 8444 passing spectests to 10224, out of 15627 spectests. That's a 21% improvement from one day to another.

If the chart isn't reference enough, Rakudo surpassed 7000 (7007, to be precide) passing spectests (out of 15240) on 25 February. That's 3217 passing spectests in six weeks -- new passes 76.5 a day.

Perl 6 has been a project in process -- design and implementations -- for a long time, but as I've said before, I know of no better way to produce great software than in rapid, well-considered iterations with plenty of feedback.

Many people consider the CPAN the shining gem of Perl 5. As of this writing, it contains more than 53,000 modules in more than 17,400 distributions with 7316 registered authors. That's a lot of code.

Many people outside of the Perl community don't realize the implications for a centralized infrastructure which allows people to upload, install, read documentation, annotate, report bugs, request features, submit patches, and collaborate on improvements to every one of these distributions. The CPAN isn't just a repository of freely-installable libraries. It's the hub which makes so many of modern Perl's amenities possible.

First, a brief history lesson.

1987

Larry released Perl 1 in 1987. If you download Perl 1 source code (you can build it today), you'll see that it contained a rudimentary but working core test suite. The separation of concerns evident between running tests and interpreting the results is still useful today in the form of the Test Anything Protocol.

1994

Perl versions 2 through 5 continued the language testing culture. The core language had a self-hosted test suite (and still does). Many CPAN modules at that time used a program called h2xs to generate that boilerplate -- and part of the boilerplate was a program called test.pl which contained some "scary black magic -- do not edit" to set up a testing infrastructure.

1998

In 1998, Chris Nandor and Graham Barr created a CPAN Testers project to run the test suites of CPAN distributions and report their results. Remember this.

The 21st Century Testing Revolution

After the Perl 6 announcement in 2000, Michael Schwern agreed to lead the QA working group. He resolved to improve the coverage of the core test suite and the coverage of the core libraries (which was less than optimal). As part of this process, he wrote Test::More and Test::Simple to hide the black magic from users and to make tests easier to write.

This started a revolution, but the world changed when he refused to add features to Test::More, claiming that it was more important for people to write their own testing modules. At his suggestion, I extracted the reusable logic from Test::More and created Test::Builder, which for the first time allowed multiple test modules to run in the same process and collaborate to produce TAP.

The revolution had begun.

Perl QA spent the next couple of years teaching people how to write good and comprehensive and working and useful test suites. We quadrupled the number of tests in the Perl 5 core from the release of Perl 5.6.0 to 5.8.0, and that number has only increased.

Can Your Language Community Do This?

That brings us to the present day. The CPAN Testers project is almost eleven years old. According to the monthly CPAN Testers statistics, there have been over three and a half million reports posted. Activity was low until late 2006, when the number of test reports submitted exploded (see CPAN Testers Statistics Graphs). Though 3.5 million reports over eleven years doesn't sound like a lot, the current testing rate produces well over a million reports a year (perhaps closer to two).

There are even plugins for the CPAN and CPANPLUS distribution installation libraries to report any test failures when you try to install a distribution -- testing those distributions you need the most.

Anyone who uploads a distribution to the CPAN will have it tested on several hardware platforms, on multiple versions of Perl, and on multiple operating systems. These reports are gathered and aggregated and displayed on the CPAN Search page for that distribution. A related project also analyze the probability of a given distribution installing on your platform (taking into account the reports for all of its dependencies, recursively).

Imagine knowing that your code works on platforms you can't access, because people who know those platforms have set them up to give you frequent and accurate feedback.

The Perl 5 language itself has daily smoke reports (though not produced through CPAN Testers), as does the Parrot virtual machine and the Rakudo Perl 6 implementation. All of the infrastructure and experience from testing Perl 5 effectively is going into the specification tests for Perl 6.

Granted, the usefulness of all of this infrastructure depends on people using it and writing good tests, but I know of no other language community which has even attempted a project of this size. Do you?

Several responses to How to Ruin Your Ability to Release Software asked "How does the company mentioned in the final paragraph do it?" The failed release management strategy of the first company is common. If that's so obviously wrong, how can anyone else do it right?

The secret is simple. Projects with frequent software releases can release their software frequently because they release their software frequently.

I know that sounds flippant. Before you write angry comments, let me digress for a moment.

Making the Impossible Possible

One of my favorite stories of last year is Booting Linux in Five Seconds. I'd interviewed Arjan van de Ven the month before on a related topic, and he let slip that he and Auke Kok had eliminated most of the cold boot time for GNU/Linux operating systems.

Other people had tried several approaches to reduce boot time. Most seemed to believe that starting services in parallel was the right approach, but few systems cracked the minute mark.

Arjan and Auke took a different approach. They set a goal: a five-second boot time. They relentlessly set priorities and threw out everything that stood in the way. The Linux kernel is endlessly configurable, but the configuration of any particular machine is reasonably static. Why probe for new hardware on every boot? Throw it out. Why recompile keyboard maps for every launch of X.org? Throw that out.

I don't mean to trivialize the process. They overcame some interesting technical challenges ("How do you use available CPU and IO resources to their limit?"). Yet their approach was deceptively simple. They removed everything that stood in the way of achieving their goals.

How to Release Software Frequently

The only realistic way I know of to release software frequently is to start to release software frequently. You will probably fail the first few times you try. If your release process is painful, it will continue to be painful until you identify and eliminate the causes of the pain.

First, your team needs to agree to make changes. They may be radical and painful. Without a collective agreement to try the new approach, you will likely fail. Don't bother trying without this.

(By "collective agreement" I mean that everyone involved in actively producing a release must agree. There's no reason you have to ship a new CD to customers every week or every month. That may be a good thing, but project-only weekly releases are a good thing in and of themselves. I'd use them even if I worked on a shrinkwrapped application with an 18 month sales cycle.)

Second, you need to establish the duration of your release cycle (See Iteration Planning). I prefer weekly releases for full-time teams, but two weeks is fine and a month is workable. The most important attribute of this process is a single, very subtle change in your mindset: tying the release date to the calendar, not to any particular feature.

Before you hit that comment box again, please take a moment to consider the implications, both good and bad. Ready? Good.

By tying a release date to a calendar ("We release a stable, usable version of our software every Wednesday morning at 10 am"), you have two options. Either your software is usable and releasable every Wednesday morning at 10 am, or it isn't. If you want to meet the commitment you've made, you need to do something radically different.

Obviously one of those changes is to reduce the amount of time and work it takes to produce a release. That must be a quick process. It must be an automated and repeatable process. It must not block on the knowledge or activity or presence of any one single person. This is not easy to achieve, but you get a chance to improve it once a week, every week. This will be painful. It will improve over time. (One of the best places to start is to improve the speed of your test suite.)

Done Means Done

The biggest change is that trunk must always be stable. You can work on whatever features you like, as long as you keep the trunk of the project in a releasable state. You must be relentless.

Part of this stability process is never merging features until they are completely and utterly finished. The agile world calls this Done Done. In contrast to my colleague's project, Wednesday morning at 9 am is not the time to check in a feature that you just made compile and are pretty sure that QA won't find any bugs, at least you hope so.

If a feature isn't done -- verified usable as intended -- you can't merge it to the trunk. It won't be part of this week's release. That's not so bad, however. There's always next week's release.

What You Can't Do

Of course, there are more implications. The Iteration Planning link suggests that you need to identify week-sized units of progress and prioritize them relentlessly. I agree, but that won't help much until you can release software frequently.

Note that there are several things you can't do anymore:

• Hold feature freezes. The release process is too frequent for that. Good. Keep your code Done Done instead.
• Play games with your bug tracker.
• Hold back releases more than a couple of hours. If a release slips even by a couple of hours, something has gone wrong. Identify it and figure out how to fix it, right away.
• Segregate QA and development. The days in which you can throw code over the wall and hope someone else will tell you if it's Done Done are gone. You can't afford to wait to find out. Get feedback immediately!
• Maintain long-lived branches in your SCM, hoping to land them eventually. You don't have time for the divergence from trunk. Keep your branches small and short and your features small.

Is this realistic? I believe so. We've done it on Parrot for almost two and a half years now with volunteer labor. We're not perfect by any means, but we're predictable and we're improving.

None of this is out of reach of any decent team which wants to improve its processes. It's not easy, but it's possible -- and I believe it's better than every other alternative in software development.

(or Always a Release Candidate, Never a Release)

In How to Ruin Your Ability to Release Software I described the release process a colleague's workplace uses. That discussion started with a complaint about a subject near and dear to my heart: release candidate cycles.

The Semantic Dance

A release candidate is a build of the software that everyone hopes has no bugs users will care about. In truth, everyone knows it has bugs, and so it's actually a release made to prove that you intend to release software sometime, and if no one reports any bugs in it, you can tell them it's their fault that, when you finally get around to releasing software, the really-released-this-time-no-foolin' has bugs, even if you knew about them before you released the release candidate.

My colleague's project has one very simple metric for when they can produce a release candidate: the project has no bugs labeled "unhandled" in the bug tracking system. In other words, the difference between a build of the software suitable for end users is whether someone has changed the label of every new bug in the bug tracking system. (While you can use the number of bugs as an indicator of some information about the quality of the software and the rate of change in the number of bugs as a derivative of the quality of the software, keeping in mind popularity and availability, I'm not sure the text of particular UI widgets in the bug tracking system even has the whiff of a usuable metric. Talk about data-driven programming!)

Of course, that doesn't work, and (if anyone actually tries the release candidate -- they're universally buggy, so why would you?) people report bugs, and all the project wanted to do was finally release a new version of the software, and so it's time to stop doing everything else and fix all of the bugs without making any other changes to the system.

If you're very unlucky, or if your project manager is actively hostile or didn't understand the sentence in Dr. Royce's paper which says "This does not work", this is the first time a separate QA group has seen the software.

Preventing Change at All Costs

The goal of a release candidate is to be completely boring. Nothing should happen. Users and testers should discover no bugs. Nothing should go wrong. If anything, you should lose weight, look ten years younger, and drive a nicer car.

The point of having release candidates at all is so that the final, eventual, we-really-did-it release will be that boring. Rumor has it that China will hand-deliver a lovely fruit basket to the Dalai Lama if this ever happens.

Projects with release candidates often create a branch in their source code repositories to represent a stable point for development. For "stable" read "nothing can ever change, unless it's to fix a bug, and then it's only the most minor change possible." In other words, stability (does the software build, does it meet customer needs, does it work?) is less a goal for day-to-day development than it is for the once-in-a-blue-moon crunch time when someone realizes that software that you never release to customers is worthless.

The reason for creating a stability branch is so that developers can continue to develop software without worrying about those pesky concerns... yet. There's always time later to fix things. You can see this attitude in bug triaging and bug fixing. "This bug isn't very important. We can downgrade its severity. This bug is too hard to fix the right way. Let's just hack around it."

Of course, developers have to wait to hear back from QA -- and in organizations with a strong barrier between dedicated QA and developers, you won't see developers looking for their own bugs. Developers will go off and build the next big wad of code to cram into a pending release right before the next release candidate branch branches.

Imagine the tangle of merging from branch to branch. Imagine the work involved in unraveling minimally intrusive hacks and fixing bugs the right way. Imagine the arguments from a developer who wants to run off and write new code and hates to hear that code he wrote six to eighteen months ago could never have worked, and a QA person who knows that he'll lose this argument and get chewed out for the shoddy quality of the release.

At just the time when a project's quality needs to increase, the management structure of the project acts to depress its quality by hiding bugs, splitting development efforts, and actively preventing feedback on efficacy and suitability.

My favorite bad example comes from Perl 5, where all development takes place on a branch called bleadperl. Another branch represents code which will become Perl 5.10.1, and likewise Perl 5.8.10 or Perl 5.6.3 or whatever (Perl 5.8.10 is unlikely, but this development antipattern held for the 5.8.x series). The person in charge of releasing a new stable version of Perl spends a day or so every week merging changes from bleadperl to maintperl. A significant percentage of the work invested in a new stable release of Perl is manually merging patches already committed between branches.

If this person falls behind or goes on vacation, the differences between bleadperl and maintperl increase, and the time to produce a new stable version of Perl increases.

(Now imagine if all development took place on branches and only merged to trunk when they were stable; suddenly there's an extra day a week available for Perl 5 development.)

Backwards Day

If you'd never seen software development before, you might think that the normal rules of life do not apply here. Unfortunately, they do -- it's the results of this software process that go wrong.

If you don't know if your software works, why are you releasing it?

If you're not sure if your software meets customer needs, why are you releasing it?

If your developers can't keep the software stable, why should you believe that they'll stabilize it later?

If you play games in your bug tracker, why should anyone trust it?

If you can only guess if a release is worth using, why would anyone use it?

Why are so few people asking these questions about their projects?

In Shooting Yourself in the Foot with Customer Branches, I described a colleague's current work project -- specifically, how they multiplied the amount of code they had to maintain for every new customer profile.

That's not their only problem. His current frustration stemmed from inadequate testing (and very poor QA management) due to poor handling of code freezes.

A popular (but wrong) theory of software development argues that development progresses in linear and predictable cycles. You gather the specifications. You design the subsystems. You code features. You stabilize a release branch and close it to all but bugfixes (this is the feature freeze). Then you perform detailed QA to try to find and diagnose all of the bugs possible before the release.

That's a very popular theory -- it's almost as popular as it is completely and unarguably wrong.

Management has realized that eventually they have to ship software. There are always features to add and bugs to fix and subsystems to refactor, but the only way to release software is to draw a line on the calendar and say "We're shipping whatever's ready then." So far, so good.

Of course, you can't ship without this cool feature. You can't ship when the long-promised feature isn't ready. You can't ship with this many bugs. You can't ship because there's always something someone wants, and if they have to wait another 18 months for the next version, they'll take their money and find someone else to pay for the software. There's always a rush and a crunch at the end of a release cycle when everyone suddenly realizes that this time, they're not kidding about the release date.

Try being in QA when this happens. All of a sudden, the day before the code freeze, a dozen feature branches that weren't ready land on trunk. QA hasn't seen them. QA didn't even know they built. (Some of them don't.)

Now QA has the unenviable task of working around the damage of features that may not even meet the basic needs of customers but absolutely can't slip because the next release is months or years away. QA gets in arguments about features, because they don't work. Developers don't want to hear that. Management doesn't want to hear that. Change is bad during a freeze. Bugs in design or implementation or requirements take too long to fix, and the relentless drums of "But we promised to release this month!" keep counting down the days.

Eventually, a release candidate escapes, as an admission that no one knows for sure if the software works, and wouldn't it be swell if someone might try to test it, because everyone thinks it probably works, because the project manager has spent the last several weeknights in the bug tracker, reclassifying bug severities and reassigning them to later milestones because one of the criteria for a release candidate is "Zero blocking bugs" and another is "Zero critical severity bugs" and if the difference between getting a bonus this year or making the quarterly numbers means that customers will file bugs in a few months that everyone argued about fixing this week, well, there's a chance customers won't notice. All software has bugs anyway.

Bugs linger in the bug tracker. Their priorities and severities mysteriously change. Duplicate bugs pile up. The software gets released and everyone takes a long weekend....

... and on Tuesday morning, they're back at the grind, preparing for the next patch release cycle to fix all of the problems they knew about before but couldn't fix, because you can't make changes during a code freeze, because no one knows if the code works, because you can't create a release candidate with that bug count, because releasing software is a very big deal, because it's just too darn hard to prepare a release with all of this ceremony. At least they have daily status meetings, though.

Meanwhile, another company down the street knows what "done" means. They keep their bug count low. They can release a new version of their software stable enough for customers to use every day, if they want. Releases aren't a big deal. They're boring and common place. (How do they do it? See The Rapid Release Tautology.)

If you're the first company, the second company has already eaten your lunch, and they'll continue to do so every week, until you fail or until you figure out how to release software.