How often have you heard heard it said, “How much will it cost me to have new feature (x)?”. If you’re in business and you commission bespoke software or if you’re a software developer or project manager you’ll have heard this before. The truth is there are two answers to this question: “as cheap as possible despite making things harder in the future”, or “take the time now to re-factor (re-work) existing code to make a clean enhancement”.
Invariably the common approach is to take the cheapest route from A to B. But cheapest route is almost certainly not the best value in the long term. Ward Cunningham coined the phrase “Technical Debt” to help us understand the problem of how taking the quick and dirty approach sets up a system with a technical debt, similar to a financial debt. Each time a quick (and dirty) approach is taken, additional debt is built up and increases like a financial interest payment. All further work to a system is encumbered by the unpaid debt and we can choose to either continue paying the interest or to spend time to “make payments” by refactoring the quick and dirty design into a better design.
The below charts show in orange, the area representing the development effort required to implement a small feature of a consistent size as a system grows (x axis = time). The blue line represents the complexity of the code within the system. The left hand chart represents a system that is being continually worked on, but never re-factored. The right hand chart shows a system with a level of technical debt that is regularly paid down by reworking the system design regularly.
Charts show the complexity of a project which grows organically (left), and a project which is regularly maintained and re-factored (on the right).
This is how end users get into software debt.
1. A customer requests a modification to the software.
2. The developer will estimate the amount of time, plus any expenses (components, hardware) that is required to accomplish the task. A little lee-way might is usually added. The cost is determined.
3. The cost is passed to the customer, the customer agrees and the work is commissioned. Often cost-constraints imposed by the customer will push the developer towards the quickest, dirtiest option and away from suggesting refactoring (and paying off some of the technical debt principal).
4. The solution/changes are delivered and the customer is happy. However this piece of work has incurred a maintenance overhead of future work.
Sure, if a developer is working on a green-field project starting with a blank sheet of paper it is possible to design a robust system from the ground up, however additional feature creep has shown that the system will always tend towards chaos. What is needed is a regular code clean to untangle any mess introduced intentionally or otherwise.
Spaghetti code. Tasty to eat, complex to unravel.
A common example
The system shown on the left started as a “Customer Accounts” module with a connection to a single database. As the customer’s needs grew, an “Orders” module was added and then later, a “Products” module. Each new module added was quoted for at the cheapest possible price, that is, the bare essential amount of work in order to achieve the desired result. This means new code for each module, often business and data access layers merged with the user interface and a lot of similar code unnecessarily reproduced in many places around the system. The customer is happy for now as the system works, but each time development work is required on any part of this system, changes will need to be made to multiple different areas and every module will have to be tested instead of just the one under development. The end result is an exponential increase in the amount of work required to maintain the ever-increasing complexity of code within the system and when coupled with a lack of “financial” understanding of the technical debt a project has, often leads to confusion and misunderstanding of the cost of adding new modules. Often a customer might not wish for any rework or to “pay down the principal” on a debt since they don’t actually receive anything “new” by undertaking the expense of doing so.
Alternatively from the customer’s point of view the same end result (the same three modules) can be obtained, but at a higher initial cost which means cheaper work later on. with an abstracted data layer where each of the modules shares a common set of business objects and logic. It takes longer to develop the model on the right but the cheaper alternative has a hidden technical debt overhead that the customer may not see immediately, but it will become noticeable when new features for the system take longer to build and result in a less-clean and less-reliable design than can be achieved by a system that has it’s technical debt serviced regularly.
But when should you always go with the lowest amount of time, cheap as possible, dirty approach?
When the system being built today won’t change tomorrow, and the code accurately captures all the requirements and there are no planned future enhancements and when the software you deliver is the first and final release. (Yeah right!). But if your customer insists on driving down the costs of the project, it can help to explain the principal of Technical Debt.
Martin Fowler suggests that it can be tactically advantageous to incur some short-term technical debt, and that where technical debt can be identified it should be treated as the equivalent financial debt would be, categorising each type of technical debt into categories from credit card to car loan.
Martin Fowler’s Technical Debt Taxonomy
Non Debt
Feature backlog, deferred features, cut features, etc. Not all incomplete work is debt. These aren’t debt, because they don’t require interest payments.
Debt
I. Debt incurred unintentionally due to low quality work
II. Debt incurred intentionally
II.A. Short-term debt, usually incurred reactively, for tactical reasons
II.A.1. Individually identifiable shortcuts (like a car loan)
II.A.2. Numerous tiny shortcuts (like credit card debt)
II.B. Long-term debt, usually incurred proactively, for strategic reasonshttp://blogs.construx.com/blogs/stevemcc/archive/2007/11/01/technical-debt-2.aspx
What now?
The concept of “technical debt” as a metaphor for understanding the increased technical overhead on a growing project is a new concept and one that I feel makes understanding the necessity for refactoring and redesign as part of overall maintenance easier to understand for both technical and non-technical people. Personally, I’m looking at introducing the concept of technical debt into our proposals and design specifications from this point as it’s a usual mechanism for understanding the type and severity of the technical debt, and putting in place a debt repayment plan for both our own products and those of our customers.
Further reading:
- Martin Fowler: Technical Debt – http://martinfowler.com/bliki/TechnicalDebt.html
- Steve McConnel: Technical Debt – http://blogs.construx.com/blogs/stevemcc/archive/2007/11/01/technical-debt-2.aspx
- Complexity As Debt (c2.com) – http://www.c2.com/cgi/wiki?ComplexityAsDebt
Excellent article, Mike.
This is why MixAction hasn’t been released. Your “common example” is a case in point. Complex code, especially IMHO resource intensive code, becomes impossible to work with over iterations unless one breaks it down into the smallest logical pieces. Of course, this doesn’t always fit in with some modern design paradigms we won’t mention here, nor to some extent that common view of what “Rapid Release” means (which has somehow become associated with weeks instead of months which is quite incorrect).