This weekend a colleague described his frustrations with a software project at his job. I suggested that he leave a few copies of
The Art of Agile Development (Powell's affiliate link) lying around his office. In particular, the largest problem I heard in his discussion is that his team has no idea how to release software.
"It's weird," he told me. "You'd think we'd stop making the same mistakes over and over again."
One such problem is rampant and divergent customization. The project makes an embedded device for sale to OEMs. They've based their product on a standard hardware set and use the Linux kernel to drive the hardware and provide a common API for manufacturers to develop their own products. The hardware and software are both flexible. A manufacturer can specify a custom set of behaviors, and the hardware and software configuration will enable or disable features from the baseline set of components.
One of the danger signs James Shore wrote about in the Version Control chapter of the Art of Agile Development is rampant branching for customization. One common and dangerous temptation is to create a branch for each customer, making changes for that particular customer in code. If you must develop a new feature for that customer, branch the current stable codebase at that point and develop that feature on the branch. If another customer needs that feature as well, branch from the branch and maintain that branch for the new customer. Repeat until you have little hope of unifying all of those features and additions again. (In my experience, that happens the first time you branch from a branch.)
When I first read the draft of Jim's chapter, I said "I don't believe in branching." Fortunately, he didn't take my advice (and I've since repented of that). I don't believe in long-lived branches. A branch should be a simple, single-minded, and temporary divergence from trunk. The goal should always be to merge the branch back into trunk as soon as possible.
Besides the complexity of managing (and even remembering) the state of all of these customer branches, my colleague casually mentioned that his team is trying to upgrade from Linux kernel 2.6.23 to the newest version. Imagine propagating that through the gnarly tree of branches!
If instead, as Jim suggests, customer-specific configuration used a data-driven approach, and if current development always took place on an unambiguous and unified point, the work of upgrading dependencies would be far easier. It's the Don't Repeat Yourself principle expressed in terms of your version control system.
(Please note that I have no problem with distributed version control systems, as long as there is a single unification point and a strong community push toward unification. Tracking and merging changes, especially at the individual level, is very useful. Rampant and divergent branching is not.)
