The Mid-Career Crisis of the Perl Programmer

HN tl;dr: innovation in the 2010s means getting your anonymized social network for marmots acquired so that big SV can put ads on it or datamine your users

proggit tl;dr: you should have spent your time learning how to write a CRUD app and a mediocre template system, dependency injection framework, and ORM in Haskell Rails Erlang Scala Rails Python Node Clojure Julia instead of mastering one tool and learning how to solve problems.

It Begins

I shut down my music career in 1998. "Career" is too generous a word. I'd made money as a musician, but not enough to call myself a professional and certainly not enough to pay the rent and buy food. I talked my way into the Color LaserJet group at HP.

I wrote some code for the printer group, just because I could. I had a little AWT app (even though Swing had just come out, it wouldn't run on Linux for quite a while) for customer support techs to record data about customer problems. Then the networking group asked me to write a program to email customers whenever the web site had a new technical article published. The ten line shell script (written in whatever HP-UX 9.x supported out of the box) was orders of magnitude less than the Java version I could never convince myself to finish.

When I realized that I enjoyed playing with HP-UX and the Linux box under my desk far more than I appreciated learning all of the ways that paper could jam or PostScript drivers could render things wrong, I switched to system administration. (Programming was in a different state altogether, and my music degree hardly compared to the CS degree they wanted.)

After fighting fires for three months and automating away the remaining problems, I had a lot of time on my hands. The web was suddenly very interesting. mod_perl had just come out and applets were obviously not working. I picked up Perl.

I started a couple of small free software projects on my own. They never went anywhere. (Everyone used to write a template system in those days. Then I found the source code to Everything 2's predecessor, sent in a couple of patches, and (after Carly Fiorina decided to save money by not paying system administrators) eventually started working for the dot com that produced Perl Monks and promptly imploded.

Then I wrote a book. It had a cool cover but you probably didn't read it. (It was only the first book in English about blogging, but the title was wrong anyhow. This was the last gasp of the first dot com era, when everything seemed like a good idea in a certain context, almost including the TSA.)

Writing a book is equal parts frantic research and tedium, and in the first of many moves designed to convince myself that my actions had a positive effect on a cruel and uncaring universe, I took up Michael Schwern's call to unify the internals of Test::Simple and Test::More. The result is Test::Builder.

This was a revolution for everyone on the perl-qa list and it was a revolution for Perl in general, but it's the kind of revolution that you look back on now and think "Wow, that was obvious!" and "That was a revolution?" because it seems like such widespread common sense, like "It's good to drink water!" and "Maybe smoking has deleterious health effects."

It was a revolution in that it let a thousand blossoms of test modules bloom (so people could solve their own specific testing problems and share those solutions) while providing the soil in which all of those blossoms could bloom (so they all worked together with almost no effort on the module writer's part). This is an important principle of API design: handle the difficult, tedious, and tricky parts at the low levels so that everyone gets that right while not ruling out people doing their own things at higher levels. If that makes you think of layering, you're right.

I did some consulting work for a while, based on the strength of my free software work with Perl. (Making a name for yourself answering questions on a site does help get your name in the community and build some networking connections. Getting a patch or two in a dominant project helps as well.)

Then my career took a lurch sideways, and I spent some time as an editor concentrating on free software and technology.

Free Software as a Hobby

It was important to keep my programming skills fresh, so I increased the amount of time spent contributing to free software projects (in and around writing books). Parrot sucked up a lot of my efforts.

(More than one Parrot/Rakudo/P6 developer said to me in separate conversations as far back as 2007 that "Sure, it's a long slog now and no one's getting compensated much for it, but when it's done in the next 18 months, there'll be a lot of interest in books/training/consulting, and we'll be in a good position then!")

This was a rough time to be in Perl. Having helped push the Ruby snowball back down the hill with Rolling with Ruby on Rails (just over nine years ago now!) and with Perl stuck on 5.8 for another almost three years, the hope was still that a new major version would turn the language's fortunes around. Meanwhile, the testing culture had infected the core language as well as the CPAN like an unseen brigade of healthy gut fauna (don't leave the dinner table without it). There was growth and roots and foundations, but the effects would take some time.

Pugs helped, but Pugs has been a historical curiosity for a long time now.

The Resurgence of Perl that Nobody Noticed

While the world stopped waiting for a new major version of Perl, Perl 5 came out with several new major versions and the CPAN continued to be the reason that I stuck with Perl. I'm not alone in this. Moose is probably the singular reason there are any significant new projects in Perl in 2014. This by no means should take credit away from other projects such as DBIx::Class, Mojolicious, Catalyst, and other projects well worth mentioning, but Moose was a revolution in the noisy, flashy, overthrow the dominant paradigm sense. It was Fidel to the rest of the CPAN's Che—the one that gets the credit, unless you're some sort of weird countercultural rebel who likes giving oligopolic corporations money to wear t-shirts with your icon's image on them. (That metaphor rode away from my point on a motorcycle tour of South America. Sorry.)

That revolution needed a name.

Modern Perl

JavaScript had its resurgence, going from "that awful language that everyone has to use and nobody likes" to "Sure, if you squint hard enough and ignore all of this awfulness over here and are very very disciplined, you can write code that's not too bad". (If that apologia sounds familiar and you feel bitter, keep in mind that, unlike Perl, you have no choice whether to use JavaScript on the client side.)

I figured this was a great time to leave the increasingly depressing world of corporate technical book publishing to work on my own technical book publisher, because it didn't cost us tens of thousands of dollars to get the first copy out the door. While it's nice that to make revenue on a book like Modern Perl, it's still a lot better to think of writing or publishing a book as "a hobby that'll buy you nice dinners here and there" instead of "I will never have to work again, unless you count sleeping on a pile of gold coins to be work".

Book publishing is driven by the semi-random lightning strikes of big hits, and you have to be in the right place at the right time. Like a poker player, you need to know when to go all in—and, more important, you need to know when to get out. This is one of the reasons the tech book publishing market is so awful. Ruby and Rails were the hottest things since JSP in 2005 and 2006, but the market had reached its peak of sales in 2007 even as publishers were still pumping out the kind of ancillary books around Rails that they'd pumped out around Perl in 1998, 1999, 2000, and 2001 even as talking puppets, razorfish, and drones delivering fresh vegetables even before you one-click order them had become a bubble about to pop, taking with it countless jobs and lots of paper money in an inflated stock market. (Maybe that's what you should expect from hiring a member of an adolescent power fantasy fiction cult to manage the world's largest economy, but keep in mind music degree not economics degree and everything's more obvious in retrospect.)

That's one reason why there hasn't been a Modern Perl Testing book; I can't convince myself to give up increasingly valuable weekends and evenings when there are other things I could do which are either much more relaxing (spending time with friends and family), much healthier (cooking, exercising), or profitable (investing). At some point, there might be a Programming Rust book, if only because I find Rust fascinating for what it both does and doesn't do. I wish I'd had it ten years ago for Parrot.

With that said, writing and publishing Modern Perl was a good thing. Giving it away for free was a good thing. Maybe I would have made as much revenue if I'd spent the same amount of time slinging milkshakes at the local SBUX, but I'm more satisfied about the social good done by giving away PDFs and hosting a web site than I am about most anything else done in my career. (Perhaps Test::Builder has been more influential from a code standpoint, but once I start going down that line of thought, I feel a complex insert-lengthy-German-compound-philosophical-term-here emotion about those large projects I worked on which seemed influential and important at the time and haven't actually gone anywhere.)

At worst, Modern Perl gave Perl as practiced by its best programmers in the 2010s a concrete name. At best, it helped existing Perl programmers and hopefully some new ones avoid some of the worst parts and adopt some of the best parts.

Golden Handcuffs For a Very Modern Perl Consultant

With a brief foray into running my own non-publishing businesses (hey, if you want to give this value investing for individual investors or the family recipe how to make a smoothie guide a link somewhere, that'd be great) revealing that starting a business from nothing is very, very difficult.

This, I think, is one reason why so many small consultancies are consultancies and never manage to turn their brilliant ideas into shipping products because it's too lucrative to chase consulting dollars and too difficult to invest in products which may not pay off. This isn't the first time I ran into this, either. If my first job after the dot com crash had succeeded in producing its product, it would have owned a lucrative vertical market. Unfortunately, consulting dollars ran out before it could finish a proof of concept to jumpstart presales. (Of course, the investors also wanted to rewrite the Perl backend and the Python/Wx client application in Java, in 2002, to run on point of sale systems for small businesses, in 2002.)

When consulting is good, it can be lucrative and that lucre can be tempting. Consultants have to charge a rate far in excess of a normal hourly salary because there's overhead: you may have a lot of spare time between contracts, you don't get PTO or benefits you don't pay for yourself, and you have to negotiate contracts and pursue work which may or may not materialize.

If you find a good client, you treat him or her like a combination of the Pope, the Queen, and the Dalai Lama, because a trustworthy client who pays you on time and doesn't argue over little details of the contract is better than gold.

Yet suppose you budget for eight or nine months of steady work and luck into a contract that gives you twice that. Some people deal well with that boom and bust. Others don't. (For me, the mechanics of budgeting are easy but the psychology of riding the business cycle is difficult.) It's difficult to look at other opportunities with an objective eye because when the contract is great, that money looks very nice and it represents a lot of freedom, but when the contracts are awful or far between, the stability of a salaried job represents a lot of security.

Finding a Perl Job

Am I a Perl programmer?

If you look at my résumé, I've been fortunate (in one sense) to have spent the past 16 years working with Perl at a deep level on many, many projects in many jobs in many domains. I've also made money working with Python, SQL, shell, Ruby, C, C++, JavaScript, assembly, Java, SQL, PL/pgSQL, Visual Basic, PHP, LaTeX, and I'm sure I left out at least one or two.

As a professional programmer with an interest in my professional development and someone who is physiologically unable not to program occasionally outside of 9-5, I've dabbled in other technologies—not enough to call myself an expert the way I call myself a Perl expert—but enough that I don't completely embarrass myself in conversation about them. (Chuck Moore once tried to convince me of the value of writing my own operating system. I have an MP3 of this.)

Yet on my CV, one technology keeps coming up again and again. Perl

It's no secret that that's where I've spent most of my time. I like the language. I'm used to its flaws (even though a couple of hundred entries on this site will reveal that I don't like all of them) and I know how to play to its strengths. I like its community (people like Tim Bunce and Karen Etheridge and Rik Signes and Andreas König and Jess Robinson and Tatsuhiko Miyagawa are, frankly, irreplaceable elsewhere). I'm exceedingly productive with Perl in a way that would take me a while to get productive in other languages. That's what you get from a decade and a half of experience. That's what you get from at least 20,000 hours of practice.

But Did You Learn to Program?

You tell me. I can give you references who will speak of me in positive terms. I can refer to to people who will tell you I helped solve problems or helped teach them something or helped them think about their problems in better ways.

(You'll also find people who will say I can be sarcastic and moody and bitter, and for better or worse you're reading this, so why pretend?)

This is the important question that a CV can't answer. This is an important question that rifling through the small subset of my professional work that's on Github or the CPAN won't tell you. They won't tell you about all of the mistakes I made that you don't see, if they exist. They won't tell you about all of the thrashing around that I did or didn't do which Git let me squash out of existence. They won't tell you if I churned out 1400 lines of code one afternoon in a marathon coding session and it compiled on my second try and hasn't been touched since, because it just works and no one needs to touch it.

Then again, asking me to write a sorting algorithm on a whiteboard won't tell you much either beyond "Does this person actually seem like he knows how to program at all?" and that's still a thing in interviews.

You can't look at a CV and tell whether I'm the kind of person who will go above and beyond the task of transcribing what a software architect dictates that the project manager has teased out of what the director of sales has forced the enterprise sales representative to admit that he promised to the customer for the next release. (I can tell you that I would find that environment frustrating.) You can't predict only from a page and a half whether I have an enthusiasm for writing the best darn medical insurance billing code ever or whether that enthusiasm will translate into an attempt to delight users even if they never think about all of the bugs and frustrations that we squashed and eliminated before they escaped to production.

You can't tell that by reading Perl or Ruby or PostgreSQL on my résumé. You can't even tell that by reading my code. That's the risk you take in hiring someone, but isn't that at least as important as answering the question "Can you make a decent attempt to write something resembling code on a whiteboard, even though your muscle memory has been tuned over several years for Vim, because you're ostensibly a professional who cares about tools and automation and making it easy to do the easy work of typing?" or the question "How well do you fit with our team?"

But Where are the Perl Jobs?

I'm not a unique and special snowflake. Programmers aren't fungible, but there aren't so many difficult programming tasks that you need the world's singular expert in a subject. (The fact that my degree says Music and not "Computer Science" from Stanford, Berkeley, MIT, or CMU probably keeps me out of computer vision, AI research, and compiler design, however, just as much as it's garlic to the Google recruiters who want me to believe it's an honor to spend weeks answering questions about ping pong balls and sorting phone books for the opportunity to babysit a data center full of machines dedicated to slapping ads on the world's information.)

It should be no secret that, if I stay in programming, my preference is to work with the language of my expertise and preference. I believe that the fashion-driven chasing of programming fads is bad for business and programmer alike, but that world isn't changing any time soon.

As I see it, you can find a Perl job in one of a few ways:

• Create it yourself by starting your own company
• Create it yourself by recruiting a client without a strong technology preference
• Create it yourself by introducing Perl slowly into an existing job and making it irreplaceable
• Find one of the existing companies using Perl (read "Move to Amsterdam" or "spend time maintaining a codebase from 1997" or "get very lucky")
• Create something so amazing and compelling and useful that people can't not use it, like mod_perl or Movable Type or, let's throw cPanel and Catalyst in there
• ...

Fifteen or fourteen or even ten years ago, I might have railed against the inherent unfairness of a cruel and uncaring universe of technology driven by fashion, where technical superiority is less important than buzz, but I've tried these approaches over my career. As I ponder the midpoint of my career (asking myself "Should I get a CS degree?" or "Wouldn't it be more effective to get a degree in finance and manage a boutique legal firm?"—I am not making this up, but then again I thought about buying a bakery a couple of years ago and it took Robert Buels only a few minutes to talk me out of it), the broader question of patterns and trends comes up.

Is Any Technology Immune to This?

Apart from COBOL? Mumps? No.

Do I want to program in Cobol or Mumps for the rest of my life, or even the rest of the afternoon? No.

Look, I'm a programmer. I solve problems. I've managed teams and I've been managed. I've written documentation and I've written tests. Some days I've deleted more code than I've written and I've been glad of it. Some days I've solved customer problems by not writing code. Some days I've been elbow deep in profiles or Valgrind reports and other days the best thing I've done was add an index to a column in a database and it was totally worth all of the research to reach the point where that was the solution.

I have my preferences in technology, sure. I also have my experiences. When you hire a programmer, you're getting his or her preferences and experiences and biases.

Maybe I'm being too picky. Maybe it's natural to reach the mid-career crisis and say "Arguing over which syntax is best on the Internet is a poor use of time I could spend walking a dog or playing pinball with my eleven year old nephew or celebrating a housewarming with friends". Maybe it's acceptable to by cynical and say "I'm a professional, which means that even if I have no personal interest in managing clinical research records, you are paying me enough to bring my knowledge and experience and hard-won judgment to bear to solve this problem in an effective and trustworthy way", but I'm probably not going to spend my nights and weekends thinking about it.

Alternately, perhaps I should have bought that bakery. Sure, you have to hire people you can trust and pay them a lot of your revenue and you have to get up at 4 am every day and health inspectors and land zoning and the prices of dairy products...

... but you're not dealing with VCs who underpay you and dangle ever more diluted RSUs in front of you for that 1% chance of an acquihire payout that'll slightly pay more than if they'd paid you a market rate for the start, or clients who want the world in a week for $20 an hour because some teenager in Albania only charges $8 an hour but at least you speak English, or an HR department that sees "Perl" sprinkled liberally through your résumé and thinks either you're the system administrator you started as in 1998 or someone who fell through a time warp in 1999, and by the way, they have the Internet on computers now, har har, JavaScript is a real language, would you kindly join us in the new spring lineup for 2014?

The Real Question

For someone whose life obligations preclude selling everything and living on a beach in Thailand or Belize for a year, how do you evaluate a career reaching its midpoint? How do you keep programming fresh? How do you market yourself as someone who solves problems instead of someone who transcribes ideas in the language du jour? Or do you leave programming altogether?

I don't have the answers, but if there's anything a decade and a half of experience has given me, it's the ability even to ask those questions.