Krzysztof is our Head of Products and he leads the technical development behind the suite of our ready-made solutions. Throughout the years in Solidstudio, he gained an invaluable insight into the eMobilty industry’s requirements and is now more than qualified to share his knowledge with others.

Krzysztof Ciombor

. Read it first, if you haven’t already. Today, we’re going to take a closer look at snapshot testing to show you how we improved it and took it to the next level.

The original idea of Snapshot Testing was first included in Jest - have a look 

 to learn more. Its main purpose was testing frontend component changes over time to catch any unexpected UI changes. This concept, however, is more universal and can be applied to almost all codebases. The overall process is always the same:

Execute your code (either by rendering the UI, calling a method, or issuing an HTTP request).

If you were running the test for the first time, save the result to a “snapshot” file.

Otherwise, find an existing “snapshot” file and compare it with the result. If the contents are the same, pass the test. If the contents don’t match, report the test failure.

By using snapshot testing, we get a few benefits:

The tests are easy to write and maintain.

Since they are essentially similar to integration/acceptance tests, we’ll be able to validate the flow through the entire application stack (i.e. HTTP requests, Service layer, DB integration, etc.).)

They are a “simpler” alternative to Contract Based Testing (see 

) that still give us confidence that we’re not introducing any breaking changes to our APIs.

Our initial implementation of Snapshot Tests was based on the 

 library. It worked great at the beginning, but a couple of issues started to arise once we added more and more tests. One of the most frequent ones were non-deterministic JSON outputs, for example, when generating random UUIDs. We were frequently facing failures like this:

Even though it makes sense from the String equality perspective, it doesn't really matter from business requirements side. As long as the output JSON contains the id field, we don’t really care about its content. This can be mitigated by creating deterministic helpers for UUID creation.

The next issue started happening when we compared arrays. Our output was very often serialized with arrays contents in random order.

The second biggest issue we faced was the order of arrays. Even though the serialization itself should preserve the elements order, we can’t guarantee that the collections were always sorted the same (for example, a simple .addAll() on a Collection doesn’t have any ordering guarantees). Again, this became a problem when comparing snapshots as strings, even though from a logical standpoint, as long as the array elements are the same, the ordering doesn’t really matter.

So the question we started asking ourselves: “Can we do better than this?" And the answer is… “Yes we can!”

For our purpose here, we’re using snapshots for validating API responses. Since they always come in the form of JSON, we can take advantage of that. As it turns out, there’s a library out there that is perfect for our 

. However, this also means that we had to ditch the KotlinSnapshot library in favor of a custom implementation. Fortunately, that’s not so complicated to achieve and in our PoC we were able to (almost) replicate all the functionalities of KotlinSnapshot, re-using a couple of its code snippets. One of our goals was to keep the compatibility between both to facilitate the migration of the existing tests. In the end, the only required for existing test code is as simple as:

So what exactly do we get by switching from String comparison to JSON comparison? Here are two major benefits:

Taking advantage of JSONassert’s JSONCompareMode.NON_EXTENSIBLE comparison mode, we can compare an array’s contents disregarding their order, but still guarantee that no element is missing and no element was added. This means that we no longer have to focus our efforts for making the array order deterministic and can instead concentrate on the array’s contents.

Maybe a more interesting part comes with custom validation config which can be passed to validateSnapshot method. Using JSONassert’s full potential, we can ignore properties that we don’t want to validate during the snapshot comparison. Going back to our previous example, we can now easily exclude id fields from being checked for equality by doing:

Moreover, we can use a wildcard operator for more sophisticated exclusions, namely:

 - Will match all id fields, no matter the nesting level

 - Will match ALL fields (useful for doing selected fields comparison only)

Keep in mind that ignoring the field for validation only skips the comparison part. The field still needs to be present in the response, so we don’t need to worry that we’ll introduce API-breaking changes by removing some response fields.

One should always keep in mind that there’s no such thing as “silver bullet” in software development. Each solution comes with its tradeoffs. That’s the case with snapshot testing as well. There are still a couple of pain points with using it as the main testing method, namely:

These tests are much “heavier” than unit tests, meaning that the “feedback loop” is considerably longer.

Sometimes it will be difficult to reason about failures and pinpoint the exact erroneous place in the code.

Snapshot files require maintenance, they must be updated every time a change in the API is correct and required.

Assertions are not obvious and it may not be immediately visible what is being tested just by looking at the test code.

Pull Request diffs “blow up”. A single change to add one more parameters to the response DTO may cause dozens of snapshot to be updated. Since snapshot files should always be committed to the repo, those changes will show up in the code review.

As usual, it’s best to always evaluate potential solutions and new approaches with care and adjust them to our needs in the project. What worked well for us may not work so well for others. We are quite happy with our current test stack, and we’re able to achieve ~80% code coverage easily with just snapshot tests and a couple of unit tests mixed here and there for the most crucial business logic. This allows us to be confident when making changes, adding new features, and refactoring the old code.

How to take snapshot testing to a next level? Read our article.

API Acceptance testing with Kotlin, DSL, testcontainers, and snapshots.

Let's be honest for a moment: Testing is hard. The promise of TDD is delusive. In the beginning, everything works great - we add new test cases, write the code, and then are happy to see green, passing tests everywhere. However, the longer a project is developed, the harder it becomes to maintain those tests. At some point, we're spending twice as much time on adding new features and refactoring the old ones because we need to update our test code as well constantly.

I think the main source of this problem is that we tend to focus on the implementation details too much.

What we could do instead is focus on the system as a whole and treat its internals as a black box. This is commonly known as integration and acceptance testing.

For the purpose of this article, we assume that we're working on a simple application for managing Electric Vehicles Charge Points. Imagine that we have a simple service capable of performing basing CRUD operations for Charge Points. We use the Spring Framework, Kotlin, and PostgreSQL as the DB.

The ideas and technologies we're showing here aren't anything new and have been around for a while now. This article describes how to use them together. Although the examples are based on Kotlin and Spring, the general principles they demonstrate are universal and can be applied to any language or tech stack.

Our goal with this architecture is providing a pleasant developer experience (DX) so that everyone is happy to write tests, the tests themselves are easy to implement and maintain, and the code is easy to follow and understand.

Before we dive into the test code, we need to explore testcontainers. For our integration tests, we have to connect to a database. Here are the available options for getting that done:

Use in-memory DB, like H2 - this method could cause trouble when using a DB-specific code in the implementation (for example, geospatial features).

Connect to a local DB instance - this option could be troublesome to set up. It requires each developer (+CI environment) to have a DB instance set up locally.

Use testcontainers - this will automatically take care of initializing the DB docker image, setting up a running DB instance, and cleaning up after the tests once they're completed.

As you can see, using testcontainers greatly simplifies the DB setup process. Please refer to the testcontainers 

 for full setup instructions. Here’s a simplified version:

1. Add testcontainers to build.gradle dependencies.

2. Override Spring properties for test, providing the correct testcontainers JDBC connection url (in application-test.properties):

The actual DB name is irrelevant. Adding TC_DAEMON=true allows us to keep the container running and be reused across multiple tests.

3. (Optional) To simplify things further, we'll create a base class for all the integration tests to inherit from. This will take care of selecting the correct test properties file (defined in the previous step) and provide a couple of useful helper methods. Such class looks as follows:

We'll see how those helpers come in handy once we start writing tests.

Here's where our custom Kotlin DSL comes into play. We want to be able to set up our DB with test data quickly. However, creating our object hierarchies is sometimes very tedious and requires a lot of boilerplate code. Creating a custom DSL helps with reducing that unnecessary noise from the test code by moving object creation responsibilities into the DSL code. Compare these two examples:

Not only the Kotlin code is less verbose, the DSL can do much more for us. For example, initializing default properties by generating random values using 

Kotlin code is also more explicit. How would you know if the geocoordinates constructor parameters order is lat/long or long/lat without looking into the code or using IDE to help you?

Once we have the objects ready we can easily persist them to the DB:

For acting step we will reproduce the usual app’s use-case, that is, handling an HTTP request. Inside our helper base class we already initialized mockMvc. Let’s take a look at a simple action of retrieving Charge Points by sending a GET request:

Notice, that by using mockMvc we’re emulating a real-world HTTP request which also goes through (and validates) elements like the Spring validation, authorization, JSON body parsing etc.

Finally, we'd like to confirm that the HTTP response is correct. We could check if the returned result includes all the correct fields manually, but that might include a lot of manual assertions for complex responses. What we could validate instead is whether the response is the same as before when we make changes inside our code. After all, the most important thing for us is not introducing some API breaking changes by accident.

This is where snapshot tests come into play. Let's see an example (based on the code above)

Create a new snapshot file when it's run for the first time.

Check if an existing snapshot is already present in the code and compare the result to see whether it is the same or not.

If the result and the saved snapshot differ, report a test failure.

An example snapshot could look like this:

As you can see, that's the exact JSON representation of the HTTP response, as expected. Now we can safely modify the internals of our app, and the test will validate that no breaking changes were made to the API.

However, sometimes we need to apply these changes - for example, when adding new fields. What then? We can easily update the snapshot executing:

Note that one should still manually validate that the update makes sense and the API should indeed change. We can do that easily during the code review process. Remember that snapshots need to be committed to the git repository!

There's one more scenario where we need to perform a different type of validation. Let's look at creating a new Charge Point. We still have to send an HTTP request via mockMvc, but the response could simply be HTTP 201, without any data returned. A snapshot would be of no use to us here.

How do we assert that the request did what it was supposed to do? We could "invert" the process a little. We can create an expected object using our DSL, retrieve the saved entity from the DB, and compare these two.

Note: Since we're clearing the DB for each test, it's fine to do findAll().first() since no other Charge Point (than the one created during this test) should exist in the DB. Using the recently introduced AssertJ functionality to compare objects recursively, we can easily validate that the saved entity matched the expected one. Skipping id fields is useful since they're only important for the persistence layer and will be regenerated by Hibernate automatically, so there's no point in comparing them.

Of course, there's no silver bullet when it comes to testing, and this approach has some trade-offs that you need to conside.

Integration tests are usually way "heavier" than unit tests, meaning that the test suite will take longer to complete (having to set up the DB, Spring context, API requests, etc.), leading to longer "feedback cycles."

It's difficult to reason about failures. Since the entire stack is tested at once, it may be tricky to pinpoint the exact place where an error occurred.

If you'd like to check out the full sample project, you can clone it from here:

https://gitlab.com/solidstudio-team/acceptance-testing-demo

We've achieved over ~80% of code coverage with a relatively small amount of test code. Adding new tests is easy and painless. The tests themselves are clear and concise. Note that I haven't said that Unit Tests are bad or unnecessary anywhere in this article. On the contrary, I believe that they're still useful and have their place. Unit Tests are always super useful for testing business logic. However, I believe that they should be applied only to "units" that don't have any dependencies. Once we start introducing mocks, are the tests really validating the correct behavior?

Developing a well-defined set of integration tests that emulates the real-world scenarios performed against our app's API is a great way to ensure that the app is behaving correctly. However, remember that what worked for us here may not work equally well for everyone.

API testing is not an easy task. Learn about making it a bit easier.

API Acceptance testing

Onboard the API standards into your system.

Being an eMobility leader in various projects across multiple startups, Piotr was responsible for OCPI and OCPP integration as well as EMP and CPO platforms development. With extensive expertise in the open standards field and eMobility tech solutions, he is leading teams of experienced developers and contributing with invaluable knowledge and experience.

Piotr Majcher

It’s quite common to create a custom API when starting a business. It’s hard to identify the patterns of how our features could be used by third parties. The same thing happens to the entire market. Standards are created (or at least they should be) based on the experience, once enough parties are interested in using the same language.

If you work in the energy market (both traditional and mobile), finance, or any other well-established area, it’s likely that there are already some standards describing what your business does.

But when you are launching a startup, who cares about standards? When you’re following the LEAN approach, fulfilling standard protocols isn’t the most important thing at the beginning.

But if you succeed, there is a chance you will want to open your API. And if yes, you will want to sell it to many customers. If you answered yes twice, it’s the right moment to onboard the API standards into your system.

Compare your model with the model described by the standard

Whether you want to introduce a new e-commerce platform, an 

, or a payment platform, there are some concepts that are common for all systems, no matter the vendor. It’s common to have checkout in an online store, charging session in an electric vehicle charging network, and some way of money transfer in a payment platform.

So there is a big chance that your model will fit more or less to the model from the protocol you’d like to implement. But every component has its essential parts and some additions. It’s always good to check upfront if your model matches the standard, at least in these essential parts. A good practice is filling a lookup table with all of the parameters from the standard and related parameters from our model. It’s smart to make such a table upfront. It’s work that needs to be done anyway, but it’s much cheaper to do it before writing the first line of code.

If you create something special, protocol designers may not foresee it. It doesn’t mean that the standard is not for you. If you use an extensible data format, like JSON, you can make some changes and still comply with the standard. API clients may use the standard form, but you can still give them access to your cutting-edge feature. What’s more, potential clients who already have the standard implemented will require much less effort to use your additional feature than if everything would have to be prepared from scratch.

Check if you need to implement all or nothing

It’s good if the standard has some parts that can be introduced independently. There may be some common parts, like everything related to the security and other non-functional parts. But core components don’t have to depend on each other. Let’s take a look at some examples. The 

open roaming protocol for electric vehicles

) standard has few non-functional modules:

You need to have Version and Credentials to start working with OCPI. But you can expose Locations without providing Sessions, Tokens, or CDRs. This is a great advantage of this protocol. With a relatively small effort, you can start talking to your partners using OCPI, even if you don’t have some other concepts covered by the standard.

Like many other things that have strict definitions, standards are versioned. Every developer knows that it’s good to carry out security updates, use the latest versions of libraries, etc. It’s a bit different from standards. If your goal is to open your API to as many clients as possible, you probably can’t just go with the latest version. It’s good to invest some time and do the research upfront.

Standards are maintained by an organization or company/companies. But it’s common for communities to drive changes in the protocol. You can try to join the community before you decide which versions you’re going to support. The community is usually well-informed and knows that, for instance, almost no one is going to implement the current version because everyone is waiting for a breakthrough change in the next version, and the previous one is used by most big players.

If you have a microservice architecture implemented, there are some patterns you will probably need when introducing the standard into your ecosystem so that it’s not mixed with the existing core components. These are:

As your application is probably divided into components based on your domain, you will need to query multiple services to prepare the response. This is where the API composition pattern comes into play.

The second thing worth considering is where to put the code, which serves as the entry point of your implementation of the standard. Separate component for such a thing sounds like a good solution, but it really depends on your system, and maybe implementing it as a part of existing API gateway is good enough.

Introducing a standard API to expose features that were initially developed only for internal needs may be a challenging task. The most important thing is to gather as much information as possible upfront to address bigger changes earlier.

In a pessimistic scenario, you will need to put the protocol you choose away and look for something different. But it’s still much better than doing the same thing after spending weeks on coding. As standards are not something we can change easily, we need to pay attention and choose the right protocol with the right version.

Best practices for openning API following protocols

How to open API following protocols?

Serveless testing - tools and techniques.

Being a co-founder of Solidstudio, Paweł has got to see all the development phases. He’s a skilled professional with years of experience, both in the software and in the eMobility worlds.

Paweł Głogowski

Serverless architecture was one of the hottest topics in 2018. Just take a look at Medium articles and the number of threads and discussions on Reddit, Twitter, and other social media. It’s becoming more and more popular - and some well-known companies have started to use serverless architecture in their projects (Netflix or Coca-Cola, for example). However, popularity also serves to reveal some drawbacks and challenges of introducing serverless architecture. One of the biggest challenges is testing which is much different than in “traditional” applications. In this article, I would like to share some hints and tips about testing a 

The obvious difference between a traditional and serverless application is that it requires a lot of collaboration with external services - in the AWS world, it can be API Gateway, Dynamodb or S3, for instance. It’s often hard or even impossible to integrate with real services. Fortunately, there are some useful tools that can help with emulating some external services to help with writing tests. Let’s talk a little about possible tests that we can perform in the serverless world.

Firstly, to speed up every local development developers should be able to invoke code locally. At Solidstudio Software House we choose NodeJS to write our lambda functions because it’s lightweight and it’s easy to execute your code locally. We also find the serverless framework very handy as it that allows us to execute NodeJS code with some input parameters that can be really helpful.

Let’s imagine that our lambda subscribes to AWS S3 event for further processing. It thus has to consume the S3 event. It could be very helpful if we could use that event and invoke lambda as if it was consuming a real S3 event. We can easily capture the real AWS event and save it in a local JSON file. We can later use it for input of our lambda without actually using our infrastructure.

Of course, invoking lambda locally for several scenarios every time we want to change a single line of code isn’t very efficient. That’s why we need to write some automated tests. The most basic and easiest type of tests is, of course, unit testing. It's a must-have in all kinds of projects and the serverless environment is no different. The good thing is that serverless functions are "well-modularize” and so can be unit tested easily.

Like in any other system, we need to have decent code coverage in unit testing. We need to simply mock every interaction with external services. If we want to simulate some AWS events, we can use the captured AWS events discussed earlier. Every modern programming language has a framework to write and mock unit tests with. For NodeJS, we can use Mocha or Jasmine which offers a very easy way to simulate external events. Building and running tests must be a part of building a deployable package.

Once you have your code well-tested locally and are confident that the code is working correctly, you should deploy it to the development environment. This process need to be automated to avoid the potential risk of manual deployment. In serverless development, it’s crucial to test your function on the real environment because of some configurations like IAM permissions or AWS lambda limitations that don’t occur in the local environment like memory, disk space, timeout, payload size. Sometimes we forget about these limitations and something that works well on our local machine may fail in the real AWS world.

When our code has successfully reached the dev environment, you can check its behaviour by emitting event that particular functions are subscribed to. You can, for instance, upload a file to the target bucket and check if particular lambda behaves as we expect. Obviously, it's just a technique to debug and improve the development process - and testing cannot rely on it. We need some automated tests to make sure that all our functions work well after every change. We should run the entire test suite after applying changes to a single function.

Integration testing in serverless application

In a serverless project, we also need to have a decent integration of the test suite to make sure our application works as expected. Each lambda usually handles one event and most of the time we need several lambdas to cover our business story. Creating a user by the REST API can trigger events that will be consumed by several lambdas that will handle adding a user to DynamoDB, uploading photos to S3, or sending greeting emails. So, one integration test must make sure that all these three lambdas are invoked and done their job properly.

With a serverless framework along with Event Gateway, we can very simple hook up our functions to particular events and trigger our functions. Then we need to have code that will verify whether the lambdas have done their job well. It's good to write tests in one language rather than mix several techniques. The programming language should work well with AWS and update their SDK quite often so it works with the newest version of services.

Each test suite has to start with an emitting event and wait till lambdas react on it. That could be challenging as there’s no easy way to indicate whether a lambda is complete. One approach could be invoking some lambda and waiting for some fixed calculated time before testing - for instance, we could take lambda timeout as our wait time. However, it's not a good idea since delays may happen or some lambdas will finish before that time - its either risky or inefficient.

The best approach is checking periodically if some condition is fulfilled - if we expect an object to appear on S3 then we can check whether it’s there, as well as check for data in our storage (DynamoDB, RDS etc). We can then finish the test as soon as the lambda has finished its job, or fail if the outcome is not what we expected or the lambda hasn’t finished its work in a reasonable amount of time.

At Solidstudio Software House, we’ve started our latest project in a fully serverless architecture. We noticed that it requires a significant mindset change, but developers who have in microservice architecture should be able to move to serverless quite smoothly. As serverless architecture becomes more and more popular, they can take advantage of reliable and easy-to-use tools. We find serverless framework and event gateway as the best technology choice.

 If you’d like to stay in touch, please sign up for our newsletter: no spamming, just information about the latest content.

Read this article to get familiar with serverless testing.

Snapshot testing

17 DECEMBER 2019 • 8 MIN READ