Read this article to find out what overengineering is and how to tackle this problem.

Being a co-founder of Solidstudio, Paweł has got to see all the development phases. He’s a skilled professional with years of experience, both in the software and in the eMobility worlds.

Paweł Głogowski

In this article, I would like to discuss some situations that could lead developers to overengineer and what the consequences of that could be. It might come as a surprise, but this could be a problem experienced by very ambitious and talented engineers who lack business skills.

Overengineering could be problematic since these programmers are usually of great value to the company, but at some point need to be supervised by analytics and more business-savvy developers.

Read this article to find out what overengineering is, what its common origins are, and what you can do about it.

Let’s start with some definitions. In this case, they might be problematic and not that straightforward.

"the act of designing a product to be more robust or have more features than often necessary for its intended use, or for a process to be unnecessarily complex or inefficient"

To be honest, this is not very helpful - especially for software developers. We should come up with a definition that is more understandable to the developers. A very straightforward simple definition could be: 

"Code or design that solves problems you don’t have."

It’s slightly better, but it still doesn’t answer all of the possible questions about what overengineering is. However, let's stick to it for now and refill this definition throughout the article.

Common reasons why developers overengineer their code

As I mentioned at the beginning, this problem very often concerns young and talented programmers. That’s because they usually learn about some useful principles like 

, or design patterns from book or course examples that can be very naive. They have a tendency to show that it’s easy to extend every kind of project and all the use cases will fit into 

Unfortunately, that's not always true since business requirements know nothing about these principles and follow their own patterns. That leads to speculation and wrong usage of design principles which I’ll try to explain in detail later on.

Let's take a look at some real-life examples

Using generics to support the "future" use cases

 programming is quite handy and can be a real help when we want to build an easy to extend abstraction. However, less-experienced developers often make the mistake of using generics prematurely.

 to search within CSV files. It’s SQL based so we get ResultSet that needs to be transformed to some more useful DTO object. Let's take a look at the following example:

We made generics just for one case and guess what - it was enough! It hasn’t been changed throughout the entire project. There were no gains thanks to that approach and it confused new developers.

In OO programming, we have quite a lot of concepts that can improve quality. I’ve already mentioned 

, but there are also other concepts like encapsulation and abstractions - and, of course, design patterns. There are also generics and exceptions. All these tools can improve the quality and extensibility of your code. But you need to apply them wisely because these rules aren’t going to make your code better automatically.

You need a closer look at the macro picture of your application to make sure which parts need abstraction and which requirements will most likely change. Take a look at the enterprise version of 

. At the micro-level, it looks perfect as it follows 

 principles, uses design patterns, generics, etc. In the end, it just solves a simple problem that could be easily written in less than 10 lines. Of course, the requirements could change in the future, but right now we’re using all of these principles based on speculations!

To sum up, always take a look at the macro picture - the current requirements and the easiest way to solve the problem.

Don’t generalize parameters when you know their current exact type.

Consider skipping the interface when using just one implementation that most likely won't change.

Don't convert every "if" statement into a separate strategy - use design patterns to complex problems.

Developers try to be smarter than business

Adding additional properties for "future" requirements

Some tasks during development can be a little harder than just writing code. Usually, they’re linked to database changes since a database requires migration, cooperating with other teams, running them on all environments, etc. It’s usually a risky operation and developers aren’t keen on taking that risk. This is where the approach of adding as much data as possible in one migration, "just in case" derives from.

So, let's imagine that there’s a requirement to introduce a new entity in the system. Right now, we need only the id and name. We don’t know anything else so far, but developers often expect to need other fields in the future, so it's better to introduce them and avoid making another database change. And that’s how we end up with additional fields like address, tax number, etc. Most of those fields won’t be used for a long time or will never be used in the system. We may end up with a wrong data model that confuses every new developer and requires some ugly hacks, either in code or in the database model.

Another real-life example is Bugzilla - you can read more about it 

. To make a long story short, developers decided to introduce a new column to the database which indicates if an option is available for a certain user because they think this feature is so obvious and will be eventually available in Bugzilla. It turns out that business guys have a different view on that because this feature hasn’t been present till now.

Too much speculation leads to premature generalization

A common principle in OO programming is DRY (“Do not repeat yourself”). In most cases, it’s quite handy at the micro-level since we shouldn’t repeat our code in classes or methods. However, in the macro picture, following this principle could lead to premature generalization since we use abstract concepts almost everywhere. What is worse, we try to speculate over further requirements, hoping that they match our abstraction model. We would like to have something like this:

While we should rather start by following >KISS (Keep it simple, stupid) and write our system to be more like this:

Our shared logic will be more obvious when the system becomes mature, then we will see which parts can be shared easily.

There are two lessons to be taken from this paragraph:

Business requirements never converge. They always diverge.

At the beginning, it’s best to stick to the KISS principle over DRY.

In this article, I tried to explain the origins and mistakes of overengineering. It may sound like it’s not a big deal and a better approach than underengenerring, but the consequences of overengineering are also quite painful.

Overengineering can lead to building hard-to-extend and hard-to-maintain systems. At some point, developers may conclude that such a system needs to be rewritten because it’s too complicated. But if they repeat the same mistakes, it will end up exactly the same.

In this article, I would like to discuss some situations that could lead developers to overengineer and what the consequences of that could be. It might come as a surprise, but this could be a problem experienced by very ambitious and talented engineers who lack business skills.  Overengineering could be problematic since these programmers are usually of a great value to the company, but at some point need to be supervised by analytics and more business-savvy developers.

Achieve business goals and gain the edge by migrating your architecture.

Paweł Majkutewicz

A long time ago in a galaxy far, far away there was application. A pretty standard one, some UI with a 

 database. After a while a few problems arose:

once a while app crashes, so someone must restart it

the same situation with backups, they are handled by a custom script

So decisions were made - the app needs to be updated:

all crashes must be handled without user interaction

Therefore in the next chapters, we will migrate existing data to new MongoDB cluster and we will handle AWS application "upgrade".

Migrating MongoDB instance to the cluster

Having one instance of MongoDB available for the app is fine (😉). It is cheap and fast to create and deploy. Manage it also doesn't require too much effort. But it comes with few drawbacks - when something goes wrong and your instance goes down you have to bring it back online. This forces you to think about backups/snapshots, configurations, connections and so one. And this is a thing that no one wants to do it.

Instead of that, we can have a cluster, that will have redundancy, replication, automatic snapshots, so a lot of tasks will be done automatically.

Dump data from the current instance of MongoDB

The first step is to fetch all existing data from the running instance. This will be done using 'mongodump' utility. Unfortunately, I didn't have MongoDB installed on my machine, so it was obvious for me to use docker instance - I'll be able to switch mongo version in no time.

After a quick research, I've come up with bash script for making a dump for me. Most of it is boilerplate code for error handling, but the main part goes like this:

$MONG_VER - mongo version, so we can easily switch between 3.6 and 4.x

$SRC_HOST:$SRC_PORT - points to running MongoDB instance

-v $(pwd):/workdir - this mounts current dir as /workdir in docker instance

--archive=/workdir/dump/dump.gz - tells 'mongodump' to create archive file in /workdir/dump/ so effectively file will be created in ./dump/ directory

 - guessing that those guys should have the best knowledge and experience with mongo databases. Having own cluster is not complicated, go to their site, register and create a new project. And in project dashboard hit Build a New Cluster button. Next select Cloud Provider & Region and Cluster tier.

The most important thing here is to choose the appropriate cluster tier for our needs. In this case, M2 would be fine, but it has one drawback - it does not support peering (described later on):

And that is why we went with M10 cluster. In terms of backup, Cloud Provider Snapshots are enough, in that case, so we choose it. Last options are Cluster name, and naming thing is the hardest thing so after a couple of hours (😉) I was able to hit Create Cluster and after a while cluster is up and ready.

To be able to restore/import data our data we have to allow access to DB. First, we have to create a user that we will use and we can do it using Security -> Database Access menu. We have there ADD NEW USER button. After clicking it we have to fill a new user form:

Also, we have to whitelist our IP (or IP from which restore will be done). We can do it using ADD IP ADDRESS button in Security -> Network Access. If restore is performed from our machine, then we can use ADD CURRENT IP ADDRESS button.

It is time to import dumped data to the newly created cluster. Same situation here got bash script with a lot of error handling. The most important line is:

$MONG_VER - mongo version, so we can easily to switch between 3.6 and 4.x,

-v $(pwd):/workdir - this mounts current dir as /workdir in docker instance,

--archive=/workdir/dump/dump.gz - our dump,

$DEST_USER, $DEST_PASS - user credentials created in mongo cluster,

$DEST_HOST - points to running mongo instance,

$SRC_DB_NAME and $DEST_DB - allow us to rename our database, eg. original name was my-app-prod and we want to have dev instance named my-app-dev.

$DEST_HOST domain name of the created cluster. You can get this after clicking CONNECT on cluster dashboard:

And copy host part from connection string:

And that is all folks. We have running mongo cluster with our database. Now it is time to migrate the application.

Migrating AWS app to high availability architecture

In this part, we will take care of our running app. And now instead of using UI to create/deploy we will write CloudFormation script. This will give us the possibility to quickly redeploy the app.

Let's start from the smallest parts and go up in the hierarchy.

Our application is containerized in docker, so our basic block is TaskDefinition. We define here which docker image to use and some container configuration. All parameters required by the app are exposed as Parameters.

Next, we wrap the app instance with service. Service that runs and maintains the requested number of tasks and associated load balancers. All traffic that goes to app goes through service listeners, so let's check them now, cause as you can see we have a dependency on two of it.

Basically listeners handle all incoming traffic, we have two listeners.

The first listener redirects all HTTP traffic to HTTPS port.

HTTPS traffic are redirected to application (target group).

Specifies a target group for an Application Load Balancer or Network Load Balancer. Also it does health checks of our app. This solves our problem, when app die then will be automagically restarted.

AWS::ElasticLoadBalancingV2::LoadBalancer

And now lets define out load balancer. Basically we are allowing traffic from ouw group and from chosen one in ParAllowedSecurityGroup parameter - this is UI security group.

AWS::AutoScaling::AutoScalingGroup and AWS::AutoScaling::LaunchConfiguration

This part allows us to control how much instances of our app are running. For now, we just need one instance, but when appropriate time and our app will have over 2000 million users like facebook have than we can just increase our instances count and handle traffic flawlessly (😉).

Of course we have few more minor things. For example, after deploying our application we are updating api domain therefore UI is instantaneously redirected to newly created cluster.

AWS::EC2::SecurityGroup and AWS::EC2::SecurityGroupIngress

We have also security groups configured. Group is only allowing traffic from own security group - to manage load balancing.

Migrating AWS app with MongoDB to HA architecture. Visit our blog to learn more!

Migrating application to high availability architecture

 on overengineering. I have explained its origins and provided some general tips on how to avoid it. However, the principles such as KISS and YAGN that I mentioned are quite complex, and it pays to take a deep dive into them. So, in this article, I will focus on those two principles and give some advice on how to handle them.

YAGNI - Why developers want to foresee the future?

Speaking from my experience, I have seen many times how developers try to outsmart the business and foresee the future by implementing a ton of unnecessary features that will "for sure" be used someday.

Usually, that's because, for developers, the original task often seems to be simple, and they think like that: "I already make some changes in this area of application so I will sneak some additional features into it to save some time in the future." This is a very bad practice because we usually end up with unnecessary code in the codebase, and the foreseen future might never come up. Usually, after 1-2 years, you might decide to delete that code after-hours lost on maintaining it.

At first glance, it does not seem to be dangerous enough to become a source of worry. However, if we end up with more and more unused pieces of code in the application, then it easily becomes very confusing to new developers. Also, someone needs to maintain all of this code, even the bits that aren't being used. This might end up costing a significant amount of time (and money) to your team.

"Always implement things when you actually need them, never when you just foresee that you need them."

It might sound very simple, but it's that kind of advice that is easy to remember, but hard to follow. I will try to provide some real-life examples that break the YAGNI rules.

Creating abstractions having just one well-defined case

Let's say that we have to implement a functionality to send some data to an external destination. For now, it is just one destination. Usually, good developers will think that additional destinations will eventually be the case for the application, so it's worth introducing some kind of abstraction. Let's take a look at following pieces of code:

This can be dangerous from two perspectives. First, you write unnecessary code and also make assumptions about further destination systems. So you may require a huge refactor even if other destinations finally come to life.

"Never introduce an interface or abstract class for only one implementation. Create an abstraction only when you actually need it."

In general, the strategy pattern is a great option to deal with algorithms based on various criteria. However, some developers abuse it by trying to use it everywhere to replace if or switch statements. If the logic is pretty simple and there is just one if - then there is no need to rebuild it and use strategy patterns. Let's take a look at the following example:

It's 5 classes just to implement something that can be done within ten lines of code. And in that case, it seems like we don't actually need it (YAGNI).

While YAGNI seems pretty straightforward, the KISS principle is much more problematic and misleading. Most of the time, that's because for many people, "simple" may sound like something incomplete or primitive. Also, each developer will claim that their solution is "as simple as possible" and not too complex. However, I will try to explain what the main concepts around simplicity are. Let's start by defining what complexity actually is.

In computer science, you can find two types of complexity - essential and accidental. Essential complexity is something that we cannot really cope with. Basically, it refers to the domain and the actual problem we are trying to solve. It's pretty obvious that writing a new Facebook or Twitter app will be much more complex than a simple web page with just one form to send a message.

On the other hand, there is accidental complexity - a complexity that is not related to a problem and originates in bad technical decisions, wrongly implemented design patterns, and so on. Let's try to go deep into some common bad architecture designs that make your system complex.

Layered architecture was very popular in the early days of programming and served as a great concept to separate presentation view from business logic. Then the layers came to the rescue. The ability to develop parts of the application independently was a huge change. However, times have changed, and new technologies have come up. It turns out that now the number of layers is growing, and it's harder to follow the layering rules and write an application that isn't too complex. It leads to a situation in which to fix some simple issues, you need to make appropriate changes in most layers, including unit tests, which might be painful and time-consuming.

Also, developers tend to have strict rules about layers - for example, that a layer:

Is independent of the layers on top of it, having no knowledge of the layers using it.

Layers are a good way of implementing encapsulation, data separation, and so on. However, try not to abuse them, and in case you need to use some layer just to proxy it to the layer beneath it - go ahead and use that layer directly.

“Use just the layers we need, the tiers we need, and nothing more!”

Why do developers want to outsmart everyone?

I experienced a few situations in my career when I started to contribute to an existing project and noticed that there are a lot of 'cool' patterns all over the place. I saw complicated CQRM systems, event-driven systems, factories, decorators, etc., created just to solve some really simple CRUD application.

This is very often the result of isolating developers from business people. They usually don't know where the business is all about and what is valuable to the client. They can't see the application from this perspective, and, as a result, they feel that delivering business value can be as satisfying as exploring new technologies and toys to solve even the simple problems. I have seen a ton of situations, where a group of really talented developers could not deliver anything valuable to the users because they were too focused on technical improvements.

If you're a developer, my advice to you is: try to understand what is valuable to your client and get out of your comfort zone. Understanding what the priorities of the client are will be beneficial for you. Then you could focus on implementing those features and solving real problems rather than building complex systems full of unnecessary patterns and designs that serve your ego rather than users.

 try not to be just a proxy between the client and the developers. Encourage your developers to understand the core business needs and let them join some non-technical meetings with managers and users. Try to change their priorities to regularly deliver something valuable to users rather than introducing new technical improvements.

Managing complexity in IT projects is a really difficult process. In this article, I highlighted just some hints, but it's a really complex topic that cannot be captured in a single article. The most important part is to change the attitude of developers - once they are aware of what is important, they will find solutions that serve clients and not just themselves.

How to deal with overengeenering? Read the article from our expert.

Deep dive into KISS and YAGNI principles

Overview of CI and CD pipelines - why is it so important?

Tomasz Sławiński

With outgrowing technology and amount of services around us manual integration and deployment process is no longer the right approach to build and ship your projects safely. Small and large businesses builds and ships their products more often than it was years ago. It is said that each day there is over a billion builds triggered. Can you imagine each of them started manually? I certainly can’t.

That’s when CI/CD pipelines come handy. CI/CD is a concept of delivering any application to a production environment with safety, speed and reliability and as least as possible impact on the customer side. It can be a manual process but in most cases, it’s fully automized and sometimes it will require pressing just one button in some CI/CD tool’s dashboard.

For any software that is built these days, there should be a CI/CD pipeline that will consist of individual steps responsible for different elements of the integration and deployment process.

When working in teams or even individually it is crucial to know that the code that is getting produced is reliable and stable. That’s where CI practice comes in handy. That’s a part of the CI/CD process which gets triggered each time the developer pushes a code change to the repository. That is when verification of newly created code should happen. It’s responsible for proving that the new code doesn’t break existing functionalities and integrates with current business requirements or alarming developers that there are some issues with it. Thanks to it the developers can take action before the code gets to production. Usually, it consists of two or more steps which are: build the project -> run unit tests -> run integration tests. More sophisticated pipelines would integrate with additional tools (such as SonarQube) in further steps to provide a detailed report on the quality of the code. A best practice is that the CI pipeline is getting triggered whenever any push to the repository happens.

Continuous Delivery is a follow-up of the Continuous Integration step and is responsible for getting the code changes deployed into the live environment. Thanks to this step being after CI we have certainty that the code is in the deployable state even if there were thousands of code commits done by multiple developers. In most cases, CD pipeline will get triggered only when push happens to certain branches (master/develop/tag) but it can vary based on companies' standards. Also in some cases, CD pipeline will be executed automatically or manually.

Nowadays there are numbers of tools that can be used for automated CI/CD process for any software. This can be done in Jenkins, Gitlab CICD, Bitbucket pipelines or if the code is hosted on cloud infrastructure Azure Pipelines, AWS Code Build and Code deploy or Code Pipeline. Most of those tools have numerous plugins and integrates with currently used project management tools. When all of that is properly configured life of any person involved in the development becomes much simpler as those plugins are capable of relating code commits with tasks in a way that the whole development history can be accessed from one place.

Let’s get some hands-on CI/CD process. Let’s say that you’ve been asked to set up CI/CD pipeline for a corporation that has several microservices that integrates with other various products and you are responsible for preparing CI/CD pipeline for one backend microservice written in 

How should then such pipeline look like? Let’s have a look at some most popular approaches here:

For this use case, we will be using GitLab CI and will focus only on the account microservice and has pom configured with appropriate execution goals. It’s a quite popular tool these days especially because it comes along with the GitLab repository and the CI/CD pipeline configuration sits within the project code in .gitlab-ci.yml file.

For this case we want that CI pipeline consist of three steps:

So how can we achieve it with GitLab CI? Let’s have a look:

The configuration above defines three stepped CI pipeline. The most interesting one is the integration test step, so let's have a bit closer look. For integration testing, we need some sort of data source and microservices that our account microservice talks to. The assumption here is that each microservice is dockerised and the docker images are held within GitLab docker registry. Within the .yml file we define that as a part of this step we want three other services running alongside – for that GitLab runner will pull latest images and start them up. After all that we can pass those to the integration-test goal as an environment variable. Those stages will be run for any code commit as we don’t specify a branch on which given step should be run. Below is a graphical representation of how will it look like on GitLab console:

As a next step, this CI pipeline could generate code quality report and push it to third-party provider tools (such as SonarQube).

Let’s assume that the business has DEV, QA and PROD environment. For each deployment we want it to be manually triggered by the developer/DevOps. In such case our pipeline would consist of four steps:

build docker image and push it to the repository with appropriate tag on it

For given CD pipeline only the build-image-and-push is an automatic step and will be trigger only when integration-test step passes. The deploy steps are manual and branch-specific. We won’t go into the details how does the deploy-* scripts look like as they will be different per infrastructure type.

When working with microservices it is crucial to have CI pipelines that could tell us if the whole system is consistent across all the microservices that builds it.

For that, we should have a separate project that would consist of a set of end to end tests and its own CI pipeline that would get triggered when a push to any of the microservices projects that build the solutions happens. Such CI pipeline can be set up on Gitlab Ci or Jenkins or any other standalone tool such as 

Overengineering in software development

08 JANUARY 2020 • 10 MIN READ