Overengineering in software development

Origins

In this article, I would like to discuss some situations that could lead developers to overengineer and what the consequences of that could be. It might come as a surprise, but this could be a problem experienced by very ambitious and talented engineers who lack business skills.

Overengineering could be problematic since these programmers are usually of great value to the company, but at some point need to be supervised by analytics and more business-savvy developers.

Read this article to find out what overengineering is, what its common origins are, and what you can do about it.

What is overengineering?

Let’s start with some definitions. In this case, they might be problematic and not that straightforward.

According to Wikipedia, overengineering is "the act of designing a product to be more robust or have more features than often necessary for its intended use, or for a process to be unnecessarily complex or inefficient"

To be honest, this is not very helpful - especially for software developers. We should come up with a definition that is more understandable to the developers. A very straightforward simple definition could be: "Code or design that solves problems you don’t have."

It’s slightly better, but it still doesn’t answer all of the possible questions about what overengineering is. However, let's stick to it for now and refill this definition throughout the article.

Common reasons why developers overengineer their code

As I mentioned at the beginning, this problem very often concerns young and talented programmers. That’s because they usually learn about some useful principles like SOLID, DRY, or design patterns from book or course examples that can be very naive. They have a tendency to show that it’s easy to extend every kind of project and all the use cases will fit into SOLID principles easily.

Writing code based on speculation

Unfortunately, that's not always true since business requirements know nothing about these principles and follow their own patterns. That leads to speculation and wrong usage of design principles which I’ll try to explain in detail later on.

Let's take a look at some real-life examples

Using generics to support the "future" use cases

Generics in Java programming is quite handy and can be a real help when we want to build an easy to extend abstraction. However, less-experienced developers often make the mistake of using generics prematurely.

In one of our projects, we use Amazon Athena to search within CSV files. It’s SQL based so we get ResultSet that needs to be transformed to some more useful DTO object. Let's take a look at the following example:

public interface RowTransformer {
    T transform(ResultSet rs);
} 

public class QueryResultTransformer implements RowTransformer<QueryResult> {
    @Override
    public QueryResult transform(ResultSet rs) {
        // Transformation here
        return new QueryResult();
    }

We made generics just for one case and guess what - it was enough! It hasn’t been changed throughout the entire project. There were no gains thanks to that approach and it confused new developers.

Blindly applying Quality concepts

In OO programming, we have quite a lot of concepts that can improve quality. I’ve already mentioned SOLID, but there are also other concepts like encapsulation and abstractions - and, of course, design patterns. There are also generics and exceptions. All these tools can improve the quality and extensibility of your code. But you need to apply them wisely because these rules aren’t going to make your code better automatically.

You need a closer look at the macro picture of your application to make sure which parts need abstraction and which requirements will most likely change. Take a look at the enterprise version of FizzBuzz. At the micro-level, it looks perfect as it follows SOLID principles, uses design patterns, generics, etc. In the end, it just solves a simple problem that could be easily written in less than 10 lines. Of course, the requirements could change in the future, but right now we’re using all of these principles based on speculations!

To sum up, always take a look at the macro picture - the current requirements and the easiest way to solve the problem.

• Don’t generalize parameters when you know their current exact type.
• Consider skipping the interface when using just one implementation that most likely won't change.
• Don't convert every "if" statement into a separate strategy - use design patterns to complex problems.

Developers try to be smarter than business

Adding additional properties for "future" requirements

Some tasks during development can be a little harder than just writing code. Usually, they’re linked to database changes since a database requires migration, cooperating with other teams, running them on all environments, etc. It’s usually a risky operation and developers aren’t keen on taking that risk. This is where the approach of adding as much data as possible in one migration, "just in case" derives from.

So, let's imagine that there’s a requirement to introduce a new entity in the system. Right now, we need only the id and name. We don’t know anything else so far, but developers often expect to need other fields in the future, so it's better to introduce them and avoid making another database change. And that’s how we end up with additional fields like address, tax number, etc. Most of those fields won’t be used for a long time or will never be used in the system. We may end up with a wrong data model that confuses every new developer and requires some ugly hacks, either in code or in the database model.

Another real-life example is Bugzilla - you can read more about it here. To make a long story short, developers decided to introduce a new column to the database which indicates if an option is available for a certain user because they think this feature is so obvious and will be eventually available in Bugzilla. It turns out that business guys have a different view on that because this feature hasn’t been present till now.

Too much speculation leads to premature generalization

A common principle in OO programming is DRY (“Do not repeat yourself”). In most cases, it’s quite handy at the micro-level since we shouldn’t repeat our code in classes or methods. However, in the macro picture, following this principle could lead to premature generalization since we use abstract concepts almost everywhere. What is worse, we try to speculate over further requirements, hoping that they match our abstraction model. We would like to have something like this:

While we should rather start by following >KISS (Keep it simple, stupid) and write our system to be more like this:

Our shared logic will be more obvious when the system becomes mature, then we will see which parts can be shared easily.

There are two lessons to be taken from this paragraph:

• Business requirements never converge. They always diverge.
• At the beginning, it’s best to stick to the KISS principle over DRY.

Summary

In this article, I tried to explain the origins and mistakes of overengineering. It may sound like it’s not a big deal and a better approach than underengenerring, but the consequences of overengineering are also quite painful.

Overengineering can lead to building hard-to-extend and hard-to-maintain systems. At some point, developers may conclude that such a system needs to be rewritten because it’s too complicated. But if they repeat the same mistakes, it will end up exactly the same.