The trap of over-engineering and over-design

One major pitfall I’ve seen projects fall into (that includes my personal ones) again and again throught my career is over-engineering the solution by setting some grand, and most importantly unneeded, long-term goals on both the technical side but also in a more general sense on the product side (here I would guess we would drop the word “engineering” and use the more general “design”).

The problem

On the technical side this most commonly manifests on two levels as:

• Code — usually creating layers of abstraction over methods and procedures instead of using concrete implementations directly even though there is nil possibilty of the implementation changing, layers upon layers of interfaces and convoluted hierarchies of data structures made for “extensibility”
• Architecture — currently I would say the most common is trying to make the solution “ultra-scalable” even though the total userbase for the near future consists of the QA team and some product people but also as above creating layers of abstraction that hinder fast development and maintanance

To the above I would like to add the “refactor paralysis” which happens when a change is added that wasn’t in the initial design and architecture so parts of the solution get constantly rewritten. The aim here is to have a “perfect” architecture where everything ideally fits but for non-toy problems and solutions that is basically impossible in any reasonable timescale.

When we look at the product it very much mimics the issues we found above in the more general sense:

• Functionality — planning and adding new functionality even though the already existing features aren’t fully working as intended and there is no customer/market signal for needing them to be added
• Userbase — the same issue as mentioned in the Architecture point which is planning usage for a giant amount of users that don’t exist and very likely (as is the case with 90%+ of products) never will

There are many other manifestations of this in both technology and product development but I think those above show the gist of the problem: aiming for the stars and falling on your face when you trip.

The negative impact

So you may wonder “Ok, but why any of this matters for me and my project/company/etc.?”. It is a solid question and it touches on the subject itself — to develop an usable product and working software implementation of it we should mainly concentrate on problems that we currently know apply to us directly instead of planning for a gazillion possible scenarios that may happen in some distant future.

In the most general form the problem is the increase in scope — and as we know from Project Management 101 an increase in scope is either:

• Increased costs — more people/resources needed to implement the extended scope
• Delays — not having enough manpower and resources forces the timeline to be pushed back

The above is especially dangerous for products in the early phases of design and development. Extremally deadly for startups that are just creating their MVP to put it out into the wild. If you search around the Web a bit you’ll find many stories where a startup was killed because the constant “new ideas” kept popping in into the MVP and either it was presented to the market too late (and competition took most of it over) because of that, resulting in a major hit in adoption or simply killed as the budget dried up.

Specifically for tech this almost always can be seen as an increase in effort needed to implement a piece of new functionality, in a ironic way akin to what happens when we have a too much tech-debt pilled up. The reason for this is simple — as we add more pieces to the puzzle moving a single piece requires more effort as we want the whole picture remain consistent. So for example to modify some method we need to additionally worry about not breaking and properly handling all of the layers of abstraction that are to be used “in the future”. Very, very often those abstractions are never utlizied and just add a bunch of work that doesn’t produce any added value.

Not becoming another Icarus

So we’ve talked about the problem in general, how it manifests and the issues it might bring. But how do we avoid it? Can we even?

Yes, at least for the most part. Unfortuantely it is one of those “fuzzy” problems that doesn’t have a perfect solution and no step-by-step algorithm to follow. But there are a few rules that you should follow in general as they apply to both the technical and product side:

• Identify the core functionality needed — what do you need for the solution to work as it is required
• Always focus and prioritize the core functionality — a bug or improvement in the core functionality should always take precedece over adding new stuff
• Plan the scope for given milestones and don’t change it — the scope of planned changes should always aim to achieve some milestone and shouldn’t be modified if there isn’t a direct need regarding the core functionality
• Understand that perfect is the enemy of good — perfection doesn’t exist for any real-world solution and chasing it will only stop you from delivering the good one

The core functionality mentioned above of course can evolve but it should always be a slow, well-though out process instead of some ad-hoc “Eureka!” moment that flips the table in the room. In short — set your priorities straight and stick to them!

What is said above might sound simple and like a truism but in reality it is pretty hard to follow, especially for the more ambitious people— there is a certain allure presented by new, shiny features and complex software designs. The challenge is mostly about self discipline and not getting sucked into it.

Properly working authorization and authentication is much more important than a new look of the product list in a fancy marketplace app. But you wouldn’t believe how many people would choose the latter or software wise re-engineering the whole thing instead of just fixing the bug letting the customers properly use the service, or even use it at all. Again — priorities!

Seeing this happen so many times is the reason why I think one should require at least some product-oriented thinking from technical leadership on all levels — so that they know how to identify the real engineering needs and reconcile them with the needs of the product and the company, stopping people and projects from going down the rabbit hole before it’s too late.

A little afterword

I’ve wrote this article after some thought about all of the projects and companies I’ve been a part of. This piece was the first one that I wanted to share but the plan is to write some more when a good subject comes up in my mind.

Hope that you’ve liked it and would be very happy to hear from you about the stuff that can be improved in my writing.

Thank you for letting me share my experience with you all and take care!