Garima Singh - Blog - Complexity in Software Engineering

The word “complex”, when used as an adjective, implies “consisting of many different and connected parts” and when used as a noun implies “a group or system of different things that are linked in a close or complicated way; a network”. Ant colonies, nervous system, public transportation systems of a country and software systems are examples of complex systems. The common denominator in all these examples is the interconnected nature of their parts. The reason for the interconnected nature of these systems is to deliver value. Complex systems allow distribution of tasks and responsibilities among its parts. Ant colonies leverage the complex nature for survival, achieved through the division of labour among the ants - such as finding food, maintaining the colony, and protecting the queen. The nervous system is a complex network of nerves and cells that allow your body to conduct various functions by carrying signals and messages to and from the brain to organs. Public transportation systems are complex networks of buses, trains, and trams that allow people and goods to move from point A to point B. Software systems are complex networks of code, data and infrastructure that allows a business or an organisation to deliver value to the user of their products.

By definition complexity is "the state or quality of being intricate or complicated". Complexity is the inherent nature of something that is composed of many interconnected parts. In simple words, if anything has many parts, it continues to get tougher to understand or comprehend with each added part.

"No Silver Bullet-Essence and Accident in Software Engineering" was a paper authored by Fred Brooks in 1986. He first introduced the concept of essence and accident to categorise complexity. Later, in 2006 Ben Moseley and Peter Marks in their paper “Out of the Tar Pit” formalised these definitions as follows:

Essential Complexity is inherent in and the essence of, the problem (as seen by the users).
Accidental Complexity is all the rest - complexity with which the development team would not have to deal in the ideal world (e.g. complexity arising from performance issues and from suboptimal language and infrastructure).

Let's take an example

Digital Transformation for a Hire Shop

Imagine you have been tasked to solve a problem for a local hire shop. The team members are currently using paper-based forms where manual customer details and contracts are maintained to allow renting of different tools such as rug cleaner, sander, etc. They are keen to move to a digital platform. The goal is to transition from the manual system to a digital platform that empowers team members to:

Book Equipment: Allow team members to record new hires and create digital contracts.
Manage Hires: Provide tools to update or modify ongoing hires and customer details.
Process Payments: Support charging relevant security deposits and hire fees.
Release Hires: Enable the seamless conclusion of hires, including return confirmations.

Here these stated objectives are “essential complexity” i.e. inherent in the problems as seen by the user.

To build a digital application for the same which would be usable, other underlying requirements would be acceptable performance, ease of use, and ability for multiple team members to concurrently access the system without impact on data integrity. These are all "accidental complexity" i.e. not inherent to the user.

What causes essential complexity?

If there is a problem to be solved via means of a software application/system, there would be a set of requirements that would be needed to be delivered to solve the problem. All these requirements are implemented in form of code and data. This code contains state and logic to solve the problem. This is what represents the essential complexity and that is why it is inherent to the problem.

What causes ~~Accidental~~ Incidental Complexity?

As I wrote my first draft of this article and reached out to friends and colleagues for review, the word “accidental” sparked some interesting reactions. It appears to have a negative connotation. I much prefer the word “incidental” (I was patting myself on the back for coming up with this terminology on my own but a quick Google search ruined the aha moment for me. There are a couple of blogs, discussions and articles which have gone down this road already).

In software engineering, we often accumulate code cruft over time. The word "accidental" seems to imply (even though it was not intended to) all of this everything-else bucket is cruft. Why calling it "incidental" changes it to a neutral connotation i.e. "happening as a result of (an activity)".

So what causes incidental complexity? ~~Accidental~~ Incidental complexity comes from 3 main sources:

Cross-functional requirements (Sarah Taraporewalla's blogpost on CFR): These are set of requirements/specifications that describe the operations of the system in contrast to functions of the product. To implement these requirements, the team must write code, or create data, or set up infrastructure. This leads to increased complexity in the system which is not inherent to the user.
Tech Debt: This is an engineering trade-off, a conscious decision team makes to use suboptimal design to deliver under constrains (time, budget, tooling, etc.) with an intention to come back and fix it. Tech Debt is an intentional act to borrow against the future, with an agreement to pay it back later. This suboptimal design also contributes to the incidental complexity of the system.
Code Mess: This is unconscious or ignorant act of accumulating code cruft over a period of time. We can attribute this to not following best practices, sensible defaults, knowledge silos, improper training, etc. This cause of incidental complexity is the most dangerous of all, as it is often left undiscovered until later.

Let's revisit the hire shop example.

Cross-functional requirements: Security, Performance, Scalability, Observability, Fault Tolerance, Ease of Expression, etc.

Tech Debt: Using batch processing to send emails over event-based approach due to technology constraints, using unstructured logs due to time constraints, etc.

Mess: Not writing unit tests, not following consistent naming conventions, constantly ignoring runtime warnings, etc.

Conclusion

Complexity is a nature of software systems, an unavoidable one. By understanding the types of complexities your systems are composed of principles can be applied to manage complexity. Any evolving software system sees a growth in complexity - essential and incidental. There are principles that allow managing these complexities (probably a topic for a follow-up blog post). Once we understand these complexities, the aim is to assist with essential complexity and reduce or eliminate as much incidental complexity as possible. The ultimate goal is to deliver software systems which are effective and sustainable.