Perl 5 and Binary Compatibility

One of the issues under consideration in the Perl 5 support policy is binary compatibility.

Binary compatibility is the likelihood that binaries compiled against a previous release will work with a newer release. These binaries are most likely modules with XS components, though they can also be programs which embed Perl. This is very different from source compatibility (Unix's traditional compatibility guarantee), where the syntax of a program doesn't change in incompatible ways. You may not be able to take advantage of new features, but old features continue to work as you expect.

An XS of Pain

If you've used the CPAN much, you know that XS modules can be more difficult to configure, build, and install than pure Perl modules. (If you haven't used the CPAN much, know that there's a convention of providing pure Perl versions of certain XS modules. They may be slower and less efficient, but they can be easier to install and debug.)

Why is XS so troublesome?

Windows and Mac OS X users have noticed that installing XS modules requires a working development environment, including the Perl headers, a decent compiler, and a passable make utility. (To be fair, even Unix users can have trouble, especially those on platforms with horrible C compiler support. The C99 standard is a decade old. If you're not busy, would you mind implementing some of its features? Thanks!) Strawberry Perl is a great Perl distribution which includes a preconfigured development environment suitable for building XS modules.

Part of the problem is that XS is difficult to use correctly. The Perl 5 core has far too little encapsulation; XS exposes many intimate details of its internals. This allows a lot of power, but it has other implications which I'll discuss in a future entry. The problem here relates to binary compatibility.

Duck Sequencing

The silly example of duck typing suggests that anything that looks like a duck, walks like a duck, and quacks like a duck is obviously a duck. That's fine in a lot of languages. It's not (usually) fine in C (though like most things in C, you can fake it if you're exceedingly clever and disciplined and very good at lying to yourself).

Suppose you have a Duck struct:

struct Duck
{
    quack_func_t *quack;
    walk_func_t  *walk;
    unsigned int  num_ducklings;
    bool          has_feathers;
}

If you don't know C, that's fine. Think of this like a hash or a dictionary where the order of the keys really, really matters and the size of the values really, really matters. In C terms, this describes a blob of memory. The compiler carves out some 16 bytes of memory (four bytes per pointer, four bytes for an unsigned integer, and four bytes for the boolean -- wasting plenty of bits to make the struct size a power of two). Anytime in the source code you refer to a Duck, the compiler knows that the first four bytes refer to the function the duck uses to quack.

At least, the compiler believes you when you tell it that. The compiled binary doesn't check. That information isn't there. All the binary knows is that it has a chunk of memory and that to quack, it grabs the first four bytes from that chunk and uses that to look up a function to call.

All is fine and good if you've typed your program appropriately. Imagine that someone comes along and says that ducks must also swim. This means that the Duck struct also needs a swim member. There are several ways to handle this. One is to put this member where it makes the most sense:

struct Duck
{
    quack_func_t *quack;
    swim_func_t  *swim;
    walk_func_t  *walk;
    unsigned int  num_ducklings;
    bool          has_feathers;
}

This version of the struct has functions at the top in alphabetical order. That's nice for maintainers; it has a well-defined structure. Unfortunately, any binary which used a Duck before is now broken until you recompile it. Remember, the binary doesn't check that the struct's layout has changed. The binary doesn't know about the struct layout. It just knows to look in a specific spot for a specific amount of memory which it can treat in a specific way.

Code that tried to walk before will now swim. If those functions have different signatures, expect a crash. Code that checked the number of ducklings will now show a very, very fertile duck when the code treats the walk pointer as an integer value.

When Ducks Cry

The right way to maintain backwards compatibility is to put the swim function at the end of the struct declaration. Code compiled against the previous Duck won't be able to swim because it doesn't know anything about that struct member, but at least it can do everything the previous Duck did...

... unless it did anything advanced with Duck structs, like constructing its own or relying on a particular layout for reflection purposes.

At this point you should be able to imagine the chaos if you want to remove a struct member.

While there are some benefits to C's compact representation of data, the drawbacks are serious. C's type system is a thin veneer over memory layout which goes away at compile time.

Another approach is to hide the details of a Duck behind an interface of functions. To create a duck, call a function. To manipulate a duck's member variable, call a function. (You can also use macros.) Adding a layer of abstraction gives you the ability to hide a duck's intimate details behind an interface that won't change as much.

This often works, but those functions can change too. As I alluded before, changing a function signature or name can make existing binaries crash too. You can keep old functions around as a compatibility layer, revising arguments and delegating to the new versions....

Less Code is Easier to Maintain

... but the more code you have, the more difficult it is to maintain.

This argument applies to the users as well as the developers of Perl 5. The cost of sorting through an API and documentation is important to consider. Pawing through lists of deprecations and backwards compatibility concerns is not free. In the Internet age, when Perl 4-style tutorials have long outlived their usefulness, it's easy to find a tutorial that explains the old and broken way to do something. If that code's still around, it's not obvious that the new way works, or what that new way might be.

Sometimes the best way to implement a new feature is to remove old code; sometimes the only way to fix a bug is to remove old code -- especially when you develop iteratively toward the optimal potential solution.

If Perl 5 had a well-defined and well-enforced boundary between perl internals and Perl extensions, p5p would have an easier time rearranging the internals to support new features, remove bugs, and to improve the code to make maintenance easier. The current situation allows extensions to poke at Perl's guts. As I've said, we can't even tell if or when this happens.

Improving Perl 5 is unnecessarily difficult -- but we need the courage and the freedom to break binary compatibility when the gains outweigh the costs of upgrading and changing. That's a calculation we perform too infrequently.

The Current Policy

The current binary compatibility policy is that all minor releases in a major release series maintain binary compatibility. 5.8.0 established a binary compatibility level in July 2002. The nine subsequent releases over the next six and a half years modified neither existing function signatures nor exposed struct layouts.

Of course, without a well-defined and regular release schedule, users can't predict when they'll have to recompile their XS extensions.

Without a well-defined extension API, XS developers may have to support multiple major versions of Perl. Without a formal end-of-life policy for Perl releases, XS developers have to make their own decisions about what they'll support -- and which APIs work where.

(XS itself is often unnecessary, but that's a different problem altogether.)

Don't misunderstand. Binary compatibility within releases in a major family is usually very good. Perl 5's reliance on the CPAN is wonderful, but reinstalling dozens of modules after upgrading to a newer minor version would be problematic. (This may argue for better CPAN distribution management.)

Yet drawing out binary compatibility for unknown periods of time -- like drawing out the lifespans of releases for unknown periods of time -- leads to unpredictability, not just for developers but for users. Reliable, constant improvement requires not just the will but the ability to make changes. Sometimes those changes are incompatible with what's come before. (If we could predict the future reliably, they wouldn't be -- but the best way we can learn is through experience.)

Allowing ourselves the ability to make changes -- and remove vestigial code that we'd otherwise have to support for indeterminate periods -- allows us to improve. Tying those periods to well-known calendar dates actually increases the predictability of our systems and processes.

Users don't have to upgrade, of course. Users can choose to stick with the best Perl 1999 had to offer. It's free. They get the same amount of support they would back then.

Yet if we can make improvements -- if we have the will to unshackle our future from our past -- we may be able to offer them something far better than we could have imagined in 1999. That's my goal, anyway.