API Acceptance testing with Kotlin, DSL, testcontainers, and snapshots.

Krzysztof is our Head of Products and he leads the technical development behind the suite of our ready-made solutions. Throughout the years in Solidstudio, he gained an invaluable insight into the eMobilty industry’s requirements and is now more than qualified to share his knowledge with others.

Krzysztof Ciombor

Let's be honest for a moment: Testing is hard. The promise of TDD is delusive. In the beginning, everything works great - we add new test cases, write the code, and then are happy to see green, passing tests everywhere. However, the longer a project is developed, the harder it becomes to maintain those tests. At some point, we're spending twice as much time on adding new features and refactoring the old ones because we need to update our test code as well constantly.

I think the main source of this problem is that we tend to focus on the implementation details too much.

What we could do instead is focus on the system as a whole and treat its internals as a black box. This is commonly known as integration and acceptance testing.

For the purpose of this article, we assume that we're working on a simple application for managing Electric Vehicles Charge Points. Imagine that we have a simple service capable of performing basing CRUD operations for Charge Points. We use the Spring Framework, Kotlin, and PostgreSQL as the DB.

The ideas and technologies we're showing here aren't anything new and have been around for a while now. This article describes how to use them together. Although the examples are based on Kotlin and Spring, the general principles they demonstrate are universal and can be applied to any language or tech stack.

Our goal with this architecture is providing a pleasant developer experience (DX) so that everyone is happy to write tests, the tests themselves are easy to implement and maintain, and the code is easy to follow and understand.

Before we dive into the test code, we need to explore testcontainers. For our integration tests, we have to connect to a database. Here are the available options for getting that done:

Use in-memory DB, like H2 - this method could cause trouble when using a DB-specific code in the implementation (for example, geospatial features).

Connect to a local DB instance - this option could be troublesome to set up. It requires each developer (+CI environment) to have a DB instance set up locally.

Use testcontainers - this will automatically take care of initializing the DB docker image, setting up a running DB instance, and cleaning up after the tests once they're completed.

As you can see, using testcontainers greatly simplifies the DB setup process. Please refer to the testcontainers 

 for full setup instructions. Here’s a simplified version:

1. Add testcontainers to build.gradle dependencies.

2. Override Spring properties for test, providing the correct testcontainers JDBC connection url (in application-test.properties):

The actual DB name is irrelevant. Adding TC_DAEMON=true allows us to keep the container running and be reused across multiple tests.

3. (Optional) To simplify things further, we'll create a base class for all the integration tests to inherit from. This will take care of selecting the correct test properties file (defined in the previous step) and provide a couple of useful helper methods. Such class looks as follows:

We'll see how those helpers come in handy once we start writing tests.

Here's where our custom Kotlin DSL comes into play. We want to be able to set up our DB with test data quickly. However, creating our object hierarchies is sometimes very tedious and requires a lot of boilerplate code. Creating a custom DSL helps with reducing that unnecessary noise from the test code by moving object creation responsibilities into the DSL code. Compare these two examples:

Not only the Kotlin code is less verbose, the DSL can do much more for us. For example, initializing default properties by generating random values using 

Kotlin code is also more explicit. How would you know if the geocoordinates constructor parameters order is lat/long or long/lat without looking into the code or using IDE to help you?

Once we have the objects ready we can easily persist them to the DB:

For acting step we will reproduce the usual app’s use-case, that is, handling an HTTP request. Inside our helper base class we already initialized mockMvc. Let’s take a look at a simple action of retrieving Charge Points by sending a GET request:

Notice, that by using mockMvc we’re emulating a real-world HTTP request which also goes through (and validates) elements like the Spring validation, authorization, JSON body parsing etc.

Finally, we'd like to confirm that the HTTP response is correct. We could check if the returned result includes all the correct fields manually, but that might include a lot of manual assertions for complex responses. What we could validate instead is whether the response is the same as before when we make changes inside our code. After all, the most important thing for us is not introducing some API breaking changes by accident.

This is where snapshot tests come into play. Let's see an example (based on the code above)

Create a new snapshot file when it's run for the first time.

Check if an existing snapshot is already present in the code and compare the result to see whether it is the same or not.

If the result and the saved snapshot differ, report a test failure.

An example snapshot could look like this:

As you can see, that's the exact JSON representation of the HTTP response, as expected. Now we can safely modify the internals of our app, and the test will validate that no breaking changes were made to the API.

However, sometimes we need to apply these changes - for example, when adding new fields. What then? We can easily update the snapshot executing:

Note that one should still manually validate that the update makes sense and the API should indeed change. We can do that easily during the code review process. Remember that snapshots need to be committed to the git repository!

There's one more scenario where we need to perform a different type of validation. Let's look at creating a new Charge Point. We still have to send an HTTP request via mockMvc, but the response could simply be HTTP 201, without any data returned. A snapshot would be of no use to us here.

How do we assert that the request did what it was supposed to do? We could "invert" the process a little. We can create an expected object using our DSL, retrieve the saved entity from the DB, and compare these two.

Note: Since we're clearing the DB for each test, it's fine to do findAll().first() since no other Charge Point (than the one created during this test) should exist in the DB. Using the recently introduced AssertJ functionality to compare objects recursively, we can easily validate that the saved entity matched the expected one. Skipping id fields is useful since they're only important for the persistence layer and will be regenerated by Hibernate automatically, so there's no point in comparing them.

Of course, there's no silver bullet when it comes to testing, and this approach has some trade-offs that you need to conside.

Integration tests are usually way "heavier" than unit tests, meaning that the test suite will take longer to complete (having to set up the DB, Spring context, API requests, etc.), leading to longer "feedback cycles."

It's difficult to reason about failures. Since the entire stack is tested at once, it may be tricky to pinpoint the exact place where an error occurred.

If you'd like to check out the full sample project, you can clone it from here:

https://gitlab.com/solidstudio-team/acceptance-testing-demo

We've achieved over ~80% of code coverage with a relatively small amount of test code. Adding new tests is easy and painless. The tests themselves are clear and concise. Note that I haven't said that Unit Tests are bad or unnecessary anywhere in this article. On the contrary, I believe that they're still useful and have their place. Unit Tests are always super useful for testing business logic. However, I believe that they should be applied only to "units" that don't have any dependencies. Once we start introducing mocks, are the tests really validating the correct behavior?

Developing a well-defined set of integration tests that emulates the real-world scenarios performed against our app's API is a great way to ensure that the app is behaving correctly. However, remember that what worked for us here may not work equally well for everyone.

API testing is not an easy task. Learn about making it a bit easier.

. Read it first, if you haven’t already. Today, we’re going to take a closer look at snapshot testing to show you how we improved it and took it to the next level.

The original idea of Snapshot Testing was first included in Jest - have a look 

 to learn more. Its main purpose was testing frontend component changes over time to catch any unexpected UI changes. This concept, however, is more universal and can be applied to almost all codebases. The overall process is always the same:

Execute your code (either by rendering the UI, calling a method, or issuing an HTTP request).

If you were running the test for the first time, save the result to a “snapshot” file.

Otherwise, find an existing “snapshot” file and compare it with the result. If the contents are the same, pass the test. If the contents don’t match, report the test failure.

By using snapshot testing, we get a few benefits:

The tests are easy to write and maintain.

Since they are essentially similar to integration/acceptance tests, we’ll be able to validate the flow through the entire application stack (i.e. HTTP requests, Service layer, DB integration, etc.).)

They are a “simpler” alternative to Contract Based Testing (see 

) that still give us confidence that we’re not introducing any breaking changes to our APIs.

Our initial implementation of Snapshot Tests was based on the 

 library. It worked great at the beginning, but a couple of issues started to arise once we added more and more tests. One of the most frequent ones were non-deterministic JSON outputs, for example, when generating random UUIDs. We were frequently facing failures like this:

Even though it makes sense from the String equality perspective, it doesn't really matter from business requirements side. As long as the output JSON contains the id field, we don’t really care about its content. This can be mitigated by creating deterministic helpers for UUID creation.

The next issue started happening when we compared arrays. Our output was very often serialized with arrays contents in random order.

The second biggest issue we faced was the order of arrays. Even though the serialization itself should preserve the elements order, we can’t guarantee that the collections were always sorted the same (for example, a simple .addAll() on a Collection doesn’t have any ordering guarantees). Again, this became a problem when comparing snapshots as strings, even though from a logical standpoint, as long as the array elements are the same, the ordering doesn’t really matter.

So the question we started asking ourselves: “Can we do better than this?" And the answer is… “Yes we can!”

For our purpose here, we’re using snapshots for validating API responses. Since they always come in the form of JSON, we can take advantage of that. As it turns out, there’s a library out there that is perfect for our 

. However, this also means that we had to ditch the KotlinSnapshot library in favor of a custom implementation. Fortunately, that’s not so complicated to achieve and in our PoC we were able to (almost) replicate all the functionalities of KotlinSnapshot, re-using a couple of its code snippets. One of our goals was to keep the compatibility between both to facilitate the migration of the existing tests. In the end, the only required for existing test code is as simple as:

So what exactly do we get by switching from String comparison to JSON comparison? Here are two major benefits:

Taking advantage of JSONassert’s JSONCompareMode.NON_EXTENSIBLE comparison mode, we can compare an array’s contents disregarding their order, but still guarantee that no element is missing and no element was added. This means that we no longer have to focus our efforts for making the array order deterministic and can instead concentrate on the array’s contents.

Maybe a more interesting part comes with custom validation config which can be passed to validateSnapshot method. Using JSONassert’s full potential, we can ignore properties that we don’t want to validate during the snapshot comparison. Going back to our previous example, we can now easily exclude id fields from being checked for equality by doing:

Moreover, we can use a wildcard operator for more sophisticated exclusions, namely:

 - Will match all id fields, no matter the nesting level

 - Will match ALL fields (useful for doing selected fields comparison only)

Keep in mind that ignoring the field for validation only skips the comparison part. The field still needs to be present in the response, so we don’t need to worry that we’ll introduce API-breaking changes by removing some response fields.

One should always keep in mind that there’s no such thing as “silver bullet” in software development. Each solution comes with its tradeoffs. That’s the case with snapshot testing as well. There are still a couple of pain points with using it as the main testing method, namely:

These tests are much “heavier” than unit tests, meaning that the “feedback loop” is considerably longer.

Sometimes it will be difficult to reason about failures and pinpoint the exact erroneous place in the code.

Snapshot files require maintenance, they must be updated every time a change in the API is correct and required.

Assertions are not obvious and it may not be immediately visible what is being tested just by looking at the test code.

Pull Request diffs “blow up”. A single change to add one more parameters to the response DTO may cause dozens of snapshot to be updated. Since snapshot files should always be committed to the repo, those changes will show up in the code review.

As usual, it’s best to always evaluate potential solutions and new approaches with care and adjust them to our needs in the project. What worked well for us may not work so well for others. We are quite happy with our current test stack, and we’re able to achieve ~80% code coverage easily with just snapshot tests and a couple of unit tests mixed here and there for the most crucial business logic. This allows us to be confident when making changes, adding new features, and refactoring the old code.

How to take snapshot testing to a next level? Read our article.

Snapshot testing

Serveless testing - tools and techniques.

Being a co-founder of Solidstudio, Paweł has got to see all the development phases. He’s a skilled professional with years of experience, both in the software and in the eMobility worlds.

Paweł Głogowski

Serverless architecture was one of the hottest topics in 2018. Just take a look at Medium articles and the number of threads and discussions on Reddit, Twitter, and other social media. It’s becoming more and more popular - and some well-known companies have started to use serverless architecture in their projects (Netflix or Coca-Cola, for example). However, popularity also serves to reveal some drawbacks and challenges of introducing serverless architecture. One of the biggest challenges is testing which is much different than in “traditional” applications. In this article, I would like to share some hints and tips about testing a 

The obvious difference between a traditional and serverless application is that it requires a lot of collaboration with external services - in the AWS world, it can be API Gateway, Dynamodb or S3, for instance. It’s often hard or even impossible to integrate with real services. Fortunately, there are some useful tools that can help with emulating some external services to help with writing tests. Let’s talk a little about possible tests that we can perform in the serverless world.

Firstly, to speed up every local development developers should be able to invoke code locally. At Solidstudio Software House we choose NodeJS to write our lambda functions because it’s lightweight and it’s easy to execute your code locally. We also find the serverless framework very handy as it that allows us to execute NodeJS code with some input parameters that can be really helpful.

Let’s imagine that our lambda subscribes to AWS S3 event for further processing. It thus has to consume the S3 event. It could be very helpful if we could use that event and invoke lambda as if it was consuming a real S3 event. We can easily capture the real AWS event and save it in a local JSON file. We can later use it for input of our lambda without actually using our infrastructure.

Of course, invoking lambda locally for several scenarios every time we want to change a single line of code isn’t very efficient. That’s why we need to write some automated tests. The most basic and easiest type of tests is, of course, unit testing. It's a must-have in all kinds of projects and the serverless environment is no different. The good thing is that serverless functions are "well-modularize” and so can be unit tested easily.

Like in any other system, we need to have decent code coverage in unit testing. We need to simply mock every interaction with external services. If we want to simulate some AWS events, we can use the captured AWS events discussed earlier. Every modern programming language has a framework to write and mock unit tests with. For NodeJS, we can use Mocha or Jasmine which offers a very easy way to simulate external events. Building and running tests must be a part of building a deployable package.

Once you have your code well-tested locally and are confident that the code is working correctly, you should deploy it to the development environment. This process need to be automated to avoid the potential risk of manual deployment. In serverless development, it’s crucial to test your function on the real environment because of some configurations like IAM permissions or AWS lambda limitations that don’t occur in the local environment like memory, disk space, timeout, payload size. Sometimes we forget about these limitations and something that works well on our local machine may fail in the real AWS world.

When our code has successfully reached the dev environment, you can check its behaviour by emitting event that particular functions are subscribed to. You can, for instance, upload a file to the target bucket and check if particular lambda behaves as we expect. Obviously, it's just a technique to debug and improve the development process - and testing cannot rely on it. We need some automated tests to make sure that all our functions work well after every change. We should run the entire test suite after applying changes to a single function.

Integration testing in serverless application

In a serverless project, we also need to have a decent integration of the test suite to make sure our application works as expected. Each lambda usually handles one event and most of the time we need several lambdas to cover our business story. Creating a user by the REST API can trigger events that will be consumed by several lambdas that will handle adding a user to DynamoDB, uploading photos to S3, or sending greeting emails. So, one integration test must make sure that all these three lambdas are invoked and done their job properly.

With a serverless framework along with Event Gateway, we can very simple hook up our functions to particular events and trigger our functions. Then we need to have code that will verify whether the lambdas have done their job well. It's good to write tests in one language rather than mix several techniques. The programming language should work well with AWS and update their SDK quite often so it works with the newest version of services.

Each test suite has to start with an emitting event and wait till lambdas react on it. That could be challenging as there’s no easy way to indicate whether a lambda is complete. One approach could be invoking some lambda and waiting for some fixed calculated time before testing - for instance, we could take lambda timeout as our wait time. However, it's not a good idea since delays may happen or some lambdas will finish before that time - its either risky or inefficient.

The best approach is checking periodically if some condition is fulfilled - if we expect an object to appear on S3 then we can check whether it’s there, as well as check for data in our storage (DynamoDB, RDS etc). We can then finish the test as soon as the lambda has finished its job, or fail if the outcome is not what we expected or the lambda hasn’t finished its work in a reasonable amount of time.

At Solidstudio Software House, we’ve started our latest project in a fully serverless architecture. We noticed that it requires a significant mindset change, but developers who have in microservice architecture should be able to move to serverless quite smoothly. As serverless architecture becomes more and more popular, they can take advantage of reliable and easy-to-use tools. We find serverless framework and event gateway as the best technology choice.

 If you’d like to stay in touch, please sign up for our newsletter: no spamming, just information about the latest content.

Read this article to get familiar with serverless testing.

Introduction to serverless testing

In this article, I want to share my thoughts on accessibility testing included in the Continuous Deployment (CD), with Jenkins as the environment.

In general, when creating and improving our CI/CD Pipelines it’s good to pay a lot of attention to functional and integration tests. From the product quality perspective, they are crucial, of course. But I will try to convince you that you get significant benefits from introducing accessibility tests into your pipeline. In this article, we’re going to focus on web applications (with the exclusion of streaming or playing videos).

The first question that comes to my mind is: Why do I need this kind of tests and what's in it for me? Well, first of all, according to studies up to 15% of people has some kind of disability. Creating applications that may not be used by them is like ignoring a substantial part of the market.

Also, some legislation all over the world may require IT products to be accessible to people with disabilities. Your company might even get sued if your products are not designed to be accessible. The bottom line is - with a little effort, you can make your application user-friendly to everyone - including people with disabilities.

Let's list all some of the most widespread kinds of disabilities that we need to take into account if we want to build an inclusive web application:

Vision disabilities - the most important issues here are colour blindness and poor vision, also problems with strobing or flashing effects.

Physical disabilities - that may impact the use of the mouse or keyboard.

Literacy and learning disabilities - includes potential problems with reading and writing.

So we need to think of some amenities in our application that will allow it to be used by all users. The most important of them are as follows:

The application should have keyboard equivalents for all mouse operations.

Tabs must be ordered logically to ensure smooth navigation.

Shortcut keys need to be provided for menus.

All labels in the application should be accurate and understandable.

Colours in your applications need to be distinguishable for all users.

Images and icons should be easily understood by all end users.

The user should be able to adjust or disable flashing, rotating, or moving elements.

Accessibility rules for vision disabilities

Let’s take a closer look at the real-life examples of what we need to consider when designing our product for people with vision disabilities like colour blindness and poor vision.

Colour blindness means that people cannot see particular colours, in most cases it’s red and blue. The first rule is to use a high contrast between special elements like buttons or headers and the background colour. The second rule is using universal colours like black or white.

The other vision disability I’d like to discuss here is poor vision. To address this user need, we need to remember about several things and most importantly - it’s best to avoid small text and enable users to zoom and enlarge text without ruining the page layout. Besides text, all other elements on the page need to be visible and easily accessible.

Manual testing of accessibility can be tricky for testers, especially if they are not disabled themselves. It could be hard for them to step into the shoes of someone with a disability. For them, a web page can look perfectly fine while for someone with vision problems, it could be quite difficult to make sense of. Of course, each tester can use some guidelines or specifications, but still, that kind of testing is very challenging for most people.

So if manual testing is difficult and complicated, we can try automated accessibility tests which can be more reliable. We can use various tools available on the web for that. Most of them are open-source so you don’t need to spend any extra money on adding them to your development project.

You can run tools like that from the command line or by a build tool like Maven or Gradle. You just need to have your application deployed under an accessible URL address. Most of the tools provide various accessibility metrics and generate reports we can easily publish in our CD environment. If our rate is low, we should fail our pipeline.

At Solidstudio, we use Lighthouse for that. We integrate it with a built tool to our CI/CD provider. Most recently, we introduced such a step in our Jenkins pipeline which lets us to enable automatic deployment to production. If you cover security, performance and accessibility testing (apart from the standard units and integrations) you can be sure that your application meets the highest standards all the time.

Introducing accessibility tests in your pipeline is quite easy and cheap, while gives you a lot of advantages. You can ensure that your product can be used by people with disabilities and avoid risking any legal problems. It also helps to build a higher quality product in general - after all, accessibility rules work great for all end users.

Get familiar with accessibility testing and learn why it's so important.

API Acceptance testing

13 SEPTEMBER 2019 • 15 MIN READ