Chad Fowler's Big Rewrite

Chad Fowler by pragdave

The hirsute Chad Fowler, author of My Job Went to India and Rails Recipes, has an excellent series of blog posts up about The Big Rewrite, when a successful software product is targeted for a major rewrite, often in the latest and greatest new language / framework / database / platform / etc.

“You’ve got an existing, successful software product. You’ve hit the ceiling on extensibility and maintainability. Your project platform is inflexible, and your application is a software house of cards that can’t support another new feature.

You’ve seen the videos, the weblog posts and the hype, and you’ve decided you’re going to re-implement your product in Rails (or Java, or .NET, or Erlang, etc.).

Beware. This is a longer, harder, more failure-prone path than you expect.”

This is something we're struggling with at Elcom Technology. We have a successful content management system product that is highly flexible, configurable and chock-full of features - but it is also carrying a legacy of non-web standards compliant design decisions. A major rework of the current system is possible, but it is tempting to see the amount of work as so great that it justifies spending that time on a complete rewrite in a newer language/framework/database/platform/etc.

Amongst the options we have are to use Ruby on Rails, especially if we can do it within Visual Studio 2008 (via IronRuby), increment to .NET 3.0/3.5, replace SQL Server 2005 with an Object DB, or even just look at adding our unique intellectual property as webparts in Microsoft's SharePoint 2007 (now including their CMS solution).

Having just helped do the initial design on an upgrade of our Image Library component Chad's point about Software as Spec really hits hard. In his words:

"Make it do what it already does." That’s a tempting and simple way to view software requirements on a rewrite project. After all, the system already exists. The question of "what should it do when…" can presumably always be answered with: "what it already does".

“programmers can easily oversimplify the interface, and assume they know the capabilities of the system.”

There are two major problems with this assumption. The first, and most disruptive, is that the programmers don’t know what questions to ask. This is especially true if the programmers weren’t the original developers of the system (most often the case on a major technology shift), but even a programmer who did the original implementation of a product won’t remember every nook, cranny, and edge case. What’s worse, with the fragile safety net of an existing implementation, programmers can easily oversimplify the interface, and assume they know the capabilities of the system. If a combination of drop-down selections results in a whole new corner of the system, how are they to know without stumbling onto it (or performing an exhaustive and expensive test cycle)?

...

“Existing code is good for discovering algorithms—not complex, multistep processes.”

Based on my own experiences and conversations with thousands of software developers around the planet, I unscientifically conclude that almost all production software is in such bad shape that it would be nearly useless as a guide to re-implementing itself. Now take this already bad picture, and extract only those products that are big, complex, and fragile enough to need a major rewrite, and the odds of success with this approach are significantly worse.

Existing code is good for discovering algorithms—not complex, multistep processes.

An initial reading of the Image Library code made it clear that determining the existing functionality from the code only would be a very difficult task. As a core part of CommunityManager.NET it was called from all sorts of parts of the system, with subtly different behaviours depending upon where it was called from. To make matters worse the configuration options to allow it to handle the range of functionality different clients required added another layer of obfuscation and confusion. Talking about the functionality further with other developers that had worked on previous iterations of the Image Library made it clear that there were a large number of edge cases lurking to trap an unwary developer that attempted a simple rewrite.

The solution was to start with the existing module and work on adding functionality incrementally, initially replacing the image upload control, and then eventually moving to new screen designs, and perhaps enhanced search functionality. In other words we avoided a rewrite and went for changing what was already there (but in a major way).

It does provide food for thought, if this is representative of our modules (and I know it is of at least some of them) then a complete rewrite needs to be approached very carefully, and preferably with a well-understood specification, or permission to vary functionality significantly from our existing application. Peter McBreen's thinking about software craftsmanship suggests that successful applications should not be rewritten, but rather nurtured and taken care of so that there is little justification for a major rewrite and plenty of reasons to keep improving the existing codebase.

Perhaps you've recently worked on this sort of Big Rewrite, what were your experiences?

P.S. This rant in favour of simple software is a fun read (if, um, fruity) and educational dissection of arguments against simple software. Perhaps what we need is a simpler CMS?

P.P.S> James Shore has a great blog post about software rewrites too, and he includes some advice on what to do.