Overview of CI and CD pipelines - why is it so important?

Tomasz Sławiński

With outgrowing technology and amount of services around us manual integration and deployment process is no longer the right approach to build and ship your projects safely. Small and large businesses builds and ships their products more often than it was years ago. It is said that each day there is over a billion builds triggered. Can you imagine each of them started manually? I certainly can’t.

That’s when CI/CD pipelines come handy. CI/CD is a concept of delivering any application to a production environment with safety, speed and reliability and as least as possible impact on the customer side. It can be a manual process but in most cases, it’s fully automized and sometimes it will require pressing just one button in some CI/CD tool’s dashboard.

For any software that is built these days, there should be a CI/CD pipeline that will consist of individual steps responsible for different elements of the integration and deployment process.

When working in teams or even individually it is crucial to know that the code that is getting produced is reliable and stable. That’s where CI practice comes in handy. That’s a part of the CI/CD process which gets triggered each time the developer pushes a code change to the repository. That is when verification of newly created code should happen. It’s responsible for proving that the new code doesn’t break existing functionalities and integrates with current business requirements or alarming developers that there are some issues with it. Thanks to it the developers can take action before the code gets to production. Usually, it consists of two or more steps which are: build the project -> run unit tests -> run integration tests. More sophisticated pipelines would integrate with additional tools (such as SonarQube) in further steps to provide a detailed report on the quality of the code. A best practice is that the CI pipeline is getting triggered whenever any push to the repository happens.

Continuous Delivery is a follow-up of the Continuous Integration step and is responsible for getting the code changes deployed into the live environment. Thanks to this step being after CI we have certainty that the code is in the deployable state even if there were thousands of code commits done by multiple developers. In most cases, CD pipeline will get triggered only when push happens to certain branches (master/develop/tag) but it can vary based on companies' standards. Also in some cases, CD pipeline will be executed automatically or manually.

Nowadays there are numbers of tools that can be used for automated CI/CD process for any software. This can be done in Jenkins, Gitlab CICD, Bitbucket pipelines or if the code is hosted on cloud infrastructure Azure Pipelines, AWS Code Build and Code deploy or Code Pipeline. Most of those tools have numerous plugins and integrates with currently used project management tools. When all of that is properly configured life of any person involved in the development becomes much simpler as those plugins are capable of relating code commits with tasks in a way that the whole development history can be accessed from one place.

Let’s get some hands-on CI/CD process. Let’s say that you’ve been asked to set up CI/CD pipeline for a corporation that has several microservices that integrates with other various products and you are responsible for preparing CI/CD pipeline for one backend microservice written in 

How should then such pipeline look like? Let’s have a look at some most popular approaches here:

For this use case, we will be using GitLab CI and will focus only on the account microservice and has pom configured with appropriate execution goals. It’s a quite popular tool these days especially because it comes along with the GitLab repository and the CI/CD pipeline configuration sits within the project code in .gitlab-ci.yml file.

For this case we want that CI pipeline consist of three steps:

So how can we achieve it with GitLab CI? Let’s have a look:

The configuration above defines three stepped CI pipeline. The most interesting one is the integration test step, so let's have a bit closer look. For integration testing, we need some sort of data source and microservices that our account microservice talks to. The assumption here is that each microservice is dockerised and the docker images are held within GitLab docker registry. Within the .yml file we define that as a part of this step we want three other services running alongside – for that GitLab runner will pull latest images and start them up. After all that we can pass those to the integration-test goal as an environment variable. Those stages will be run for any code commit as we don’t specify a branch on which given step should be run. Below is a graphical representation of how will it look like on GitLab console:

As a next step, this CI pipeline could generate code quality report and push it to third-party provider tools (such as SonarQube).

Let’s assume that the business has DEV, QA and PROD environment. For each deployment we want it to be manually triggered by the developer/DevOps. In such case our pipeline would consist of four steps:

build docker image and push it to the repository with appropriate tag on it

For given CD pipeline only the build-image-and-push is an automatic step and will be trigger only when integration-test step passes. The deploy steps are manual and branch-specific. We won’t go into the details how does the deploy-* scripts look like as they will be different per infrastructure type.

When working with microservices it is crucial to have CI pipelines that could tell us if the whole system is consistent across all the microservices that builds it.

For that, we should have a separate project that would consist of a set of end to end tests and its own CI pipeline that would get triggered when a push to any of the microservices projects that build the solutions happens. Such CI pipeline can be set up on Gitlab Ci or Jenkins or any other standalone tool such as 

Scaling means everything and automation is the key to do this effectively.

Being an eMobility leader in various projects across multiple startups, Piotr was responsible for OCPI and OCPP integration as well as EMP and CPO platforms development. With extensive expertise in the open standards field and eMobility tech solutions, he is leading teams of experienced developers and contributing with invaluable knowledge and experience.

Piotr Majcher

It seems like all anyone can talk about in the business world these days is scaling their technology and automating their processes, but the actual process behind this - including the 

Too many organizations are unaware of how much time and effort it actually takes to effectively scale technology and automate processes, leading to frustration when companies discover that the process is more involved and time-consuming than they initially realized. The purpose of this article is to give you a general idea of what scaling and automating really mean, how to determine if a process should be automated, and show you how 

 can help you determine your business objectives and create custom solutions to meet your needs.

Every organization relies on multiple processes to get work done, either as part of their core business or the operations surrounding it. To scale effectively, companies need to understand the details of their processes and find a way to automate them, whether using technology or other methods.

For example, many organizations rely on automated sales funnels, which rely on automated reporting programs to gather relevant data on potential customers instead of relying on employees to complete this labor-intensive task manually.

understanding project delays and how to avoid them

, the planning stage is one of the most critical phases of any project. When you invest time and energy into the planning phase, you are setting yourself and your team up for success later on. That is why the team at SolidStudio always takes the time to learn the details of each customer’s business model in order to create the best solution to meet their needs.

First, we analyze the process itself: If the current process is flawed or can't be improved, then automating it won't improve the situation because it doesn't actually address the underlying issues. By doing our due diligence and analyzing your business, the SolidStudio team can verify that the current process in place meets your organization's needs and can't be improved before determining if the process could benefit from automation or additional technology.

Once the process has been assessed and found to be effective, we know that we are working on a solid foundation, and we can turn our attention toward the question of automation.

As a product development company, our goal isn't to just throw software at a problem and hope it solves the issue at hand. Instead, our goal is to create solutions for our clients so they can achieve their business objectives. While most of our solutions involve using software, even the most elegant piece of software isn't effective if it hasn't been created with the client's actual needs in mind. By taking the time to understand the details of a client's business, we are able to gain a deep understanding of which processes are vital and are worth automating and which processes won't benefit from automation or need to be improved first.

Determining whether or not a process should be automated can be part of a total 

, which involves integrating the right digital technology tools into different aspects of a business to improve productivity and increase the value they deliver to their customers.

The Benefits of Automation: T-Cup & Cracow Local Tours Case Studies

In this section, we will discuss 2 SolidStudio clients who benefited greatly from automating processes that were previously performed manually: T-Cup and Cracow Local Tours.

 is a health and wellness app that helps customers track their health and overall well-being and sends them regular reports based on the data the app has collected. While these reports were initially created manually by T-Cup’s staff, all of the data was already stored in the app itself, making the report generation process ripe for automation.

 for the T-Cup team that allowed them to take the data in the app and sort it into the correct schema so it could be automatically formatted and sent out to clients. By automating this process, T-Cup was able to bring on more clients while also scaling back on the number of workers they needed in the back office, improving revenue while reducing costs.

Cracow Local Tours offers tours to hundreds of tourists each day, and the sheer amount of bookings made it difficult for them to manage their pick-up and one-day trips effectively.

To help them increase efficiency and better allocate their resources, we worked with them to create an ERP system that automatically matched tour drivers with tourists in the same location, thereby reducing the non-revenue generating driving time between tours. This new system also made it much easier for customers to automatically book tickets for larger groups, which are more profitable than smaller groups taking the same tours, allowing the company to maximize earnings while improving the customer experience.

Using technology to scale and automate processes can be a great way to increase efficiency and make it easier for your business to grow if applied to processes that are already effective and can benefit from automation and technology. Before you commit to scaling or automating a particular process, make sure you have a thorough understanding of the processes you are considering automating and have verified that they can't be improved further. The SolidStudio team is here to help you get this valuable information and create automated and scalable solutions that meet your needs.

What Does it Really Mean to “Scale” With Technology & Automation?

Krzysztof is our Head of Products and he leads the technical development behind the suite of our ready-made solutions. Throughout the years in Solidstudio, he gained an invaluable insight into the eMobilty industry’s requirements and is now more than qualified to share his knowledge with others.

Krzysztof Ciombor

. Read it first, if you haven’t already. Today, we’re going to take a closer look at snapshot testing to show you how we improved it and took it to the next level.

The original idea of Snapshot Testing was first included in Jest - have a look 

 to learn more. Its main purpose was testing frontend component changes over time to catch any unexpected UI changes. This concept, however, is more universal and can be applied to almost all codebases. The overall process is always the same:

Execute your code (either by rendering the UI, calling a method, or issuing an HTTP request).

If you were running the test for the first time, save the result to a “snapshot” file.

Otherwise, find an existing “snapshot” file and compare it with the result. If the contents are the same, pass the test. If the contents don’t match, report the test failure.

By using snapshot testing, we get a few benefits:

The tests are easy to write and maintain.

Since they are essentially similar to integration/acceptance tests, we’ll be able to validate the flow through the entire application stack (i.e. HTTP requests, Service layer, DB integration, etc.).)

They are a “simpler” alternative to Contract Based Testing (see 

) that still give us confidence that we’re not introducing any breaking changes to our APIs.

Our initial implementation of Snapshot Tests was based on the 

 library. It worked great at the beginning, but a couple of issues started to arise once we added more and more tests. One of the most frequent ones were non-deterministic JSON outputs, for example, when generating random UUIDs. We were frequently facing failures like this:

Even though it makes sense from the String equality perspective, it doesn't really matter from business requirements side. As long as the output JSON contains the id field, we don’t really care about its content. This can be mitigated by creating deterministic helpers for UUID creation.

The next issue started happening when we compared arrays. Our output was very often serialized with arrays contents in random order.

The second biggest issue we faced was the order of arrays. Even though the serialization itself should preserve the elements order, we can’t guarantee that the collections were always sorted the same (for example, a simple .addAll() on a Collection doesn’t have any ordering guarantees). Again, this became a problem when comparing snapshots as strings, even though from a logical standpoint, as long as the array elements are the same, the ordering doesn’t really matter.

So the question we started asking ourselves: “Can we do better than this?" And the answer is… “Yes we can!”

For our purpose here, we’re using snapshots for validating API responses. Since they always come in the form of JSON, we can take advantage of that. As it turns out, there’s a library out there that is perfect for our 

. However, this also means that we had to ditch the KotlinSnapshot library in favor of a custom implementation. Fortunately, that’s not so complicated to achieve and in our PoC we were able to (almost) replicate all the functionalities of KotlinSnapshot, re-using a couple of its code snippets. One of our goals was to keep the compatibility between both to facilitate the migration of the existing tests. In the end, the only required for existing test code is as simple as:

So what exactly do we get by switching from String comparison to JSON comparison? Here are two major benefits:

Taking advantage of JSONassert’s JSONCompareMode.NON_EXTENSIBLE comparison mode, we can compare an array’s contents disregarding their order, but still guarantee that no element is missing and no element was added. This means that we no longer have to focus our efforts for making the array order deterministic and can instead concentrate on the array’s contents.

Maybe a more interesting part comes with custom validation config which can be passed to validateSnapshot method. Using JSONassert’s full potential, we can ignore properties that we don’t want to validate during the snapshot comparison. Going back to our previous example, we can now easily exclude id fields from being checked for equality by doing:

Moreover, we can use a wildcard operator for more sophisticated exclusions, namely:

 - Will match all id fields, no matter the nesting level

 - Will match ALL fields (useful for doing selected fields comparison only)

Keep in mind that ignoring the field for validation only skips the comparison part. The field still needs to be present in the response, so we don’t need to worry that we’ll introduce API-breaking changes by removing some response fields.

One should always keep in mind that there’s no such thing as “silver bullet” in software development. Each solution comes with its tradeoffs. That’s the case with snapshot testing as well. There are still a couple of pain points with using it as the main testing method, namely:

These tests are much “heavier” than unit tests, meaning that the “feedback loop” is considerably longer.

Sometimes it will be difficult to reason about failures and pinpoint the exact erroneous place in the code.

Snapshot files require maintenance, they must be updated every time a change in the API is correct and required.

Assertions are not obvious and it may not be immediately visible what is being tested just by looking at the test code.

Pull Request diffs “blow up”. A single change to add one more parameters to the response DTO may cause dozens of snapshot to be updated. Since snapshot files should always be committed to the repo, those changes will show up in the code review.

As usual, it’s best to always evaluate potential solutions and new approaches with care and adjust them to our needs in the project. What worked well for us may not work so well for others. We are quite happy with our current test stack, and we’re able to achieve ~80% code coverage easily with just snapshot tests and a couple of unit tests mixed here and there for the most crucial business logic. This allows us to be confident when making changes, adding new features, and refactoring the old code.

How to take snapshot testing to a next level? Read our article.

Snapshot testing

Being a co-founder of Solidstudio, Paweł has got to see all the development phases. He’s a skilled professional with years of experience, both in the software and in the eMobility worlds.

Paweł Głogowski

 on overengineering. I have explained its origins and provided some general tips on how to avoid it. However, the principles such as KISS and YAGN that I mentioned are quite complex, and it pays to take a deep dive into them. So, in this article, I will focus on those two principles and give some advice on how to handle them.

YAGNI - Why developers want to foresee the future?

Speaking from my experience, I have seen many times how developers try to outsmart the business and foresee the future by implementing a ton of unnecessary features that will "for sure" be used someday.

Usually, that's because, for developers, the original task often seems to be simple, and they think like that: "I already make some changes in this area of application so I will sneak some additional features into it to save some time in the future." This is a very bad practice because we usually end up with unnecessary code in the codebase, and the foreseen future might never come up. Usually, after 1-2 years, you might decide to delete that code after-hours lost on maintaining it.

At first glance, it does not seem to be dangerous enough to become a source of worry. However, if we end up with more and more unused pieces of code in the application, then it easily becomes very confusing to new developers. Also, someone needs to maintain all of this code, even the bits that aren't being used. This might end up costing a significant amount of time (and money) to your team.

"Always implement things when you actually need them, never when you just foresee that you need them."

It might sound very simple, but it's that kind of advice that is easy to remember, but hard to follow. I will try to provide some real-life examples that break the YAGNI rules.

Creating abstractions having just one well-defined case

Let's say that we have to implement a functionality to send some data to an external destination. For now, it is just one destination. Usually, good developers will think that additional destinations will eventually be the case for the application, so it's worth introducing some kind of abstraction. Let's take a look at following pieces of code:

This can be dangerous from two perspectives. First, you write unnecessary code and also make assumptions about further destination systems. So you may require a huge refactor even if other destinations finally come to life.

"Never introduce an interface or abstract class for only one implementation. Create an abstraction only when you actually need it."

In general, the strategy pattern is a great option to deal with algorithms based on various criteria. However, some developers abuse it by trying to use it everywhere to replace if or switch statements. If the logic is pretty simple and there is just one if - then there is no need to rebuild it and use strategy patterns. Let's take a look at the following example:

It's 5 classes just to implement something that can be done within ten lines of code. And in that case, it seems like we don't actually need it (YAGNI).

While YAGNI seems pretty straightforward, the KISS principle is much more problematic and misleading. Most of the time, that's because for many people, "simple" may sound like something incomplete or primitive. Also, each developer will claim that their solution is "as simple as possible" and not too complex. However, I will try to explain what the main concepts around simplicity are. Let's start by defining what complexity actually is.

In computer science, you can find two types of complexity - essential and accidental. Essential complexity is something that we cannot really cope with. Basically, it refers to the domain and the actual problem we are trying to solve. It's pretty obvious that writing a new Facebook or Twitter app will be much more complex than a simple web page with just one form to send a message.

On the other hand, there is accidental complexity - a complexity that is not related to a problem and originates in bad technical decisions, wrongly implemented design patterns, and so on. Let's try to go deep into some common bad architecture designs that make your system complex.

Layered architecture was very popular in the early days of programming and served as a great concept to separate presentation view from business logic. Then the layers came to the rescue. The ability to develop parts of the application independently was a huge change. However, times have changed, and new technologies have come up. It turns out that now the number of layers is growing, and it's harder to follow the layering rules and write an application that isn't too complex. It leads to a situation in which to fix some simple issues, you need to make appropriate changes in most layers, including unit tests, which might be painful and time-consuming.

Also, developers tend to have strict rules about layers - for example, that a layer:

Is independent of the layers on top of it, having no knowledge of the layers using it.

Layers are a good way of implementing encapsulation, data separation, and so on. However, try not to abuse them, and in case you need to use some layer just to proxy it to the layer beneath it - go ahead and use that layer directly.

“Use just the layers we need, the tiers we need, and nothing more!”

Why do developers want to outsmart everyone?

I experienced a few situations in my career when I started to contribute to an existing project and noticed that there are a lot of 'cool' patterns all over the place. I saw complicated CQRM systems, event-driven systems, factories, decorators, etc., created just to solve some really simple CRUD application.

This is very often the result of isolating developers from business people. They usually don't know where the business is all about and what is valuable to the client. They can't see the application from this perspective, and, as a result, they feel that delivering business value can be as satisfying as exploring new technologies and toys to solve even the simple problems. I have seen a ton of situations, where a group of really talented developers could not deliver anything valuable to the users because they were too focused on technical improvements.

If you're a developer, my advice to you is: try to understand what is valuable to your client and get out of your comfort zone. Understanding what the priorities of the client are will be beneficial for you. Then you could focus on implementing those features and solving real problems rather than building complex systems full of unnecessary patterns and designs that serve your ego rather than users.

 try not to be just a proxy between the client and the developers. Encourage your developers to understand the core business needs and let them join some non-technical meetings with managers and users. Try to change their priorities to regularly deliver something valuable to users rather than introducing new technical improvements.

Managing complexity in IT projects is a really difficult process. In this article, I highlighted just some hints, but it's a really complex topic that cannot be captured in a single article. The most important part is to change the attitude of developers - once they are aware of what is important, they will find solutions that serve clients and not just themselves.

How to deal with overengeenering? Read the article from our expert.