Popular just now

ad1

Technical Debt: Drupal, Development, Deadlines and Deaths

Technical debt (also known as design debt or code debt) is a new word coming out of a metaphor that refers to the use of bad code, short cuts and poor architecture in software. The so-called debt manifests itself as incomplete work and sometimes as the real world cost of hiring and firing personnel. The reason the concept of technical debt is taking the software world by storm because it is such a simple way of explaining the constant struggle of businesses to get projects done on time and on budget. One of the things that is not talked about is the usage of open source web based software and the technical debt that doing so incurs.



There are dozens of popular open source CMS projects but I am going to talk about Drupal because it is something that I understand and it is today one of the largest contributors to open source technical debt. There is always a rallying cry in web development firms and Start-ups, "We can do it faster and better with Drupal". What they don't realize is that by using Drupal or similar software they acquire an unbelievable amount of code debt. They are using the CMS as a silver bullet for solving time management and development issues. This effects their projects design, working development, deadlines and possible death, the four d's. By not recognizing this they are shooting themselves in the foot.

Open source Technical Debt


Many Open source content management systems have design debt. But when compared to the others, Drupal usage carries with it the greatest amount of code debt and is the hardest to pay back. In previous years this debt was paid by removing backwards compatibility and short release cycles between versions. But now the debt is so high that this method of payment is not effective. Many open source projects are "technical debt ghettos" that are torn down with each new release but still remain ghettos. Because you can build new housing for the poor but their debts comes back with a vengeance. The cost of living in the ghetto keeps increasing while its tenants ability to repay the debt dwindles.

Drupal tech debt is ignored when the decision is made to use a CMS rather than doing things from scratch. Common causes for using a CMS and incurring technical debt include:
  • Business pressures, where the business considers getting something released sooner before all of the necessary changes are complete, builds up technical debt comprising those uncompleted changes.

    These pressures come from miscalculations, errors in judgement and inexperience in real world software creation.

  • Lack of building processes or understanding, where businesses are blind to the concept of technical debt and make decisions without considering the implications.

    Does anyone ever look at the issue queue before making a decision to use Drupal or a contributed module? No! Read the documentation before building or prototyping "the next great idea" on the Drupal platform? No! Do a code review on the modules needed? No!

  • Lack of building loosely coupled components, where functions are hard-coded, when business needs change and the software becomes inflexible.

    Hard-coding takes on many different forms but the most common type is seen when the core products code base cannot be changed for fear of maintainability and breaking the system. There is a constant background chant in all Drupal forums, " don't change core". When in fact adding any contributed modules and their API dependencies does this automatically.

  • Lack of test suite, which encourages quick and less risky band-aids to fix bugs.

    There is no test suite for a CMS. While there is an attempt in the Drupal core to use test driven design it is only there for show. No one uses it, talks about it or cares. The community at large just wants things to work so TDD outside of a couple of core developers doesn't exist.

  • Lack of documentation, where code is created without the necessary supporting documentation. The work to create the supporting documentation represents a debt that must be paid.

    Drupal changes often and the documentation cannot keep up. Attempts to document modules are meet with a wall of complexity disguised as flexibility.

  • Lack of collaboration, where knowledge isn't shared around the organization and business efficiency suffers.

    The greater Drupal community has not known how Drupal works since version 5.5. Moving forward to versions 6 and 7 there are so few that know that they can be counted using the fingers of one hand. It's safe to say that Drupal 8 is worse and its only partial use of the Symfony 2 PHP Framework is making wanting to know less attractive.

  • Parallel Development at the same time on two or more branches can cause the build up of technical debt because of the work that will eventually be required to merge the changes into a single source base. The more changes that are done in isolation, the more debt that is piled up.

    This is a very hidden bit of information because end-users are tricked into thinking that they only maintain a single branch. The truth is since the combination of add on modules, configuration options (rules, views, features, content types .. etc) and dependencies are unique to each website or project they are parallel branches of the core system and have to be maintained as such. Not to mention that all Drupal only shops are using more than one version of Drupal.

  • Delayed Refactoring. As the requirements for a project evolve, it may become clear that parts of the code have gotten unwieldy and must be refactored in order to support future requirements. The longer that refactoring is delayed, and the more code is written to use the current form, the more debt that piles up that must be paid at the time the refactoring is finally done.

    Drupal based businesses rarely update modules on their own sites and almost never on a clients. This is because a security update or version updating of modules could lead to a catastrophic failure of the system. Again because the usage in each case is so unique and there are no tests, knowing whether or not an update will decrease or increase technical debt is a coin toss at best.


The most commonly talked about and unpaid types of technical debt that are unique to Drupal usage are:

  • black box technology 
  • alpha beta releases
  • deployment
  •  upgrading
  • performance bottlenecks 
  • abstraction of complexity
  • dependencies
  • diminishing returns when adding in more modules

Not all technical debt is bad and some of it is obvious and acceptable in some cases. If Drupal is used as tool for creating websites a large amount of technical debt will not have to be paid. If it is used as a Rapid Application Development (RAD) package then there will be more bad debt than good. Content Management systems have more built-in technical debt than true frameworks. Good frameworks can be used as loosely coupled libraries and are designed to carry as little debt as possible.  In a later article I will explain this more and how to take control of and reduce Drupal technical debt. But first I will provide some other critical information that you should know if are using Drupal or plan to use it in the future. Technical debt cannot be planned away or ignored away. When development starts it that is the time to control and pay the technical debt.


"We need a large community..." 

The above is a typical phrase used by those that are making the decision to use a CMS or other open source software. They tend to think that by using open source software with a large and vibrant community that the communities contributions will pay their technical debt for them. They are disappointed later when they find that this does not happen quickly enough or doesn't happen at all.  Not surprising since there is a large percentage of Drupal users that do not contribute anything to the community that would help reduce technical debt, either for themselves or anyone else. Many web development shops and Start-ups try to purchase their way out of debt by hiring "experts" and "senior" developers. You cannot buy your way out of technical debt and it cannot be fully eliminated so don't try. Doing so only increases the debt.

Development


Developing web applications with software like Drupal and the many other CMS systems may seem like a good way to jumpstart getting an idea to work. When choosing a CMS over starting from scratch or using a dedicated framework compare technical debt. The more the code base does for you, the greater the debt and its translation into actual monetary costs. At some point accountability for the incurred technical debt has to be taken. This is when the blame game starts and most of it is placed on personnel and not the amount of unpaid technical debt. Businesses rarely can diagnose the problem but the symptoms of having too much debt keep recurring daily if not hourly.

Here are a few symptoms of out of control technical debt in the form that most IT shops and developers recognize:

“What do you mean it’s going to take ____  to make that change?… 

"It seems like such a simple thing!…”

“The IT department is always putting out fires. No new development ever seems to happen…”

“We have so many systems and none of them seem to talk to each other…”

“We can’t seem to keep good developers here very long…”

- Ken Schwaber  

My all time  favorite:

"Just ...."

The most popular being that point where the debt has gotten so high that they are forced to buy or bororrow time:

"Wanted .... with .... experience... We have the basics done ... we just need ... that can meet deadlines"

Deadlines


There a number of ways to use Drupal to meet a deadline and each way carries with it varying amounts of debt.  Some of the ways are the result of good decision making and some bad. It is rare that all the fallout from bad decisions is looked at after a deadline has been met. Meeting deadlines and time estimates based on the Rapid Application Development (RAD) use of Drupal is cancelled out by later having to pay the technical debt incurred by doing so. Bad code and bad decisions become written in stone and the illusion of success covers it all.

Design Death

Design death is built-in Drupal. Three identifiable attributes of code that has entered design death:
  • The code is considered part of a core or legacy system. The functionality is interconnected to so many other parts of the system that it’s impossible to isolate (and hence reduce) any one component.

    If you have used Drupal to do anything other than create a website then you'll come across this and start thinking about not using Drupal for the project.

  • There is either no testing or minimal testing surrounding the code. This is really tautology by implication. Without comprehensive unit tests it is impossible to refactor the code to a more manageable state.

    Testing the code in a module is difficult. Testing your particular mix of modules and options is impossible because it is a moving target.

  • There is highly compartmentalized knowledge regarding the core/legacy system, and it may be supported by only one or two people in the company.

    In the Drupal eco-system there are not enough people that know how the code really works. The likelihood of one of them working for you is the same as filling and inside flush.


Drupal 8


The Drupal community is becoming aware of the technical debt that is in Drupal 8.  There is a struggle to reduce design debt in Drupal and the techniques like testing which reduce the debt have the same problems as previous versions, quirky behavior. Though there is no discussion of the debt incurred with D6 or D7 it is there and has always been there.

Awareness, Recognition and control are the only cures for excessive technical debt. Prevention though requires more. Having a good solution architect, one that is familiar with Drupal, will significantly reduce design debt.

Related:

Drupal's Excessive Complexity



1 comment: