Our introduction and the first article of the series about Consumer-Driven Contact.

Being an eMobility leader in various projects across multiple startups, Piotr was responsible for OCPI and OCPP integration as well as EMP and CPO platforms development. With extensive expertise in the open standards field and eMobility tech solutions, he is leading teams of experienced developers and contributing with invaluable knowledge and experience.

Piotr Majcher

Some time ago, we had a discussion in one of our projects whether it was the right time to introduce the Consumer-Driven Contract (CDC) approach. After a short debate, we came to the conclusion that while CDC may be a very useful tool, it’s not a silver bullet. It’s extra to what we already have and makes some processes much faster and less error-prone.

Normally, as a part of the API developers prepare some documentation like a swagger or wiki page. They specify all the possible input values with information on whether a particular parameter is mandatory or not. They do the same with the returned value. All values returned by the API are described in this document. We share documentation with third parties (or with other teams inside a company), and that’s all.

There is no space for feedback from the API’s user. We don’t know how our API is used on the data level. We probably have some monitoring in place, and we know how often users are calling our service. But do they use the entire response? Or maybe just one field? As in many other places, a short feedback loop may bring value here.

CDC is nothing else than the feedback from users about how they’re going to use our API. CDC is all about reversing the most common approach I described above. With CDC, the contract is not documentation written by the provider, but information from the consumer describing how they use the API.

Contracts over communication and documentation?

Does it mean that someday instead of receiving an email from someone who is going to use our service or ask us for something custom we’ll receive just a formal contract? Or does it mean that the consumer will define the API and we will just have to implement it? Of course, in most cases, that’s not going to happen.

The CDC doesn’t remove the need for good documentation. And for sure, it doesn’t replace face-to-face communication with a formal contract. Especially the second element - communication - is crucial for implementing the CDC successfully. In most cases, the CDC is only about feedback on how people use our API. We’re still responsible for providing documentation and discussing the details if the API we’re providing is something new. Sometimes not only the contract but the entire API may be driven by the consumer, using the same tools (some kind of formal contract). It all depends on our process and relations between the consumer team and the producer team.

How risky is refactoring code without test coverage? We would have to go to each place where the refactored function is used and check whether everyone is working in some way. Sometimes, it’s too expensive and risky so developers might give up.

Here we have a similar situation but in the HTTP world. Since we may not work with all the API consumers in the same office, such verification is usually even more expensive. Even in the same location, asking all developers to check the code because we are going to fix a typo in one of many fields returned by our API (which may be never used by someone) doesn’t sound good. By keeping contracts from all consumers in one place, we can quickly check whether anyone uses a specific parameter. We can safely modify our code without thinking about backward compatibility. The CI won’t pass any element that breaks any of the existing contracts.

It may sound as if we’re looking for a quick fix instead of proper API versioning. But API versioning is very expensive and complicated. We should avoid it for as long as possible (

). The CDC may allow us postpone that moment when versioning is the only solution.

Another aspect, in some cases an even more important one, is testing distributed systems. When we’re writing integration tests for a single service, we have two options - set up all dependencies or mock them. Since a contract consists of input and expected output, it can be used for automatic stubs generation. Having stubs for all the dependent services, we can easily prepare an isolated test that doesn’t take hours to start or need as many resources as integration test setting up many services.

What if we don’t know our consumers? Of course, we can’t ask them to prepare contracts. It means the CDC isn’t going to work in our case. In such a situation, we have to treat our documentation as a contract and be very careful when modifying anything resulting from changes to API. We have to assume that all the data we return are consumed by someone. We need to keep backward compatibility on the basis of our documentation, not specific use cases.

The second case when the CDC may not be a good fit is when we’re working with a public API. Even if we know all of our consumers, it may be really difficult, or in some cases impossible, to force them to spend time on preparing a contract and keeping it updated. It’s sometimes the matter of a paper-based contract signed by a business that states the expectations of both parties. In such cases, the CDC may not be the best solution. The biggest benefit of using the CDC will be visible if both sides of the contract work at the same company and share the same development practices.

One of the most important steps in CDC is sharing the contract written by the consumer with the producer. It can be done, for instance, through a merge request containing a contract in a format supported by the producer. Contracts may be also shared using a special tool, for instance, Pact (Pact Broker). Pack is a great tool that offers much more than just sharing contracts. It supports the entire CDC flow, including testing consumers using stubs created based on contract, sharing the contract, and verifying the contract on the producer side. Another tool that may be helpful here is the Spring Cloud Contract. You may also use Pact Broker as a contract repository.

Contract-Driven Design and contract testing are two useful tools that bring stability to distributed systems. They allow getting rid of late validation of problems regarding the integration between services. It doesn’t require deploying a service and checking integration manually. The feedback loop is quite short and can stop the building process before an artifact is created. It’s also good documentation showing who uses the API we provide and how they do it.

Our introduction and the first article of the series about Consumer-Driven Contact. Click to learn more!

Second article of our series. An example of CDC implementation with the usage of Pact. 

, we described the principles of the Consumer-Driven Contract approach. We went through the potential use cases of CDC and scenarios where the CDC doesn’t fit. Since the Consumer-Driven Contract is a concept, it may be implemented in different ways.

This article focuses on the implementation based on Pact and using the following tools/frameworks:

Spring Boot-based applications as API provider and consumer,

Pact libraries used by the API provider and API consumer, and responsible for publishing (consumer) and verifying (provider) Contracts,

For the purpose of this article, we prepared a sample API. It’s a simple HTTP API exposing sensors data. Three methods are exposed here:

GET: /api/sensors/{sensorId} - get single sensor

Each sensor has two properties: id and name.

Everything starts from a technical perspective here. Before the agreement between provider and consumer is made, the documentation defined by the provider needs to be shared with the consumer. But the CDC flow begins with the consumer defining the contract. Before we start writing code, we have to add the following dependency to our project:

Defining the contract takes the form of an integration test. Let’s stop thinking about the CDC for a moment and consider preparing an integration test for the component that makes HTTP calls. You need two things: the stub for the API and the test. Let’s start with the stub.

Almost everything is self-explanatory. It’s an API stub definition prepared using the Pact fluent API. We haven’t touched on the topic of the CDC yet. It’s very similar to what we can do with HTTP-based API simulators like Wiremock. We can define the input which is HTTP GET method against the /api/sensors path and the output - in this case, following the JSON body (sensors.json):

Sensor in the kitchen

Sensor in the bedroom

SensorsFacade is a proxy class that translates methods invocation into an HTTP call using the RestTemplate. After removing boilerplate code, it’s as simple as that:

As above, if we skip the @PactTestFor annotation, it’s just a simple test. It has the standard “given, when, then” test structure (the given is the stub we have prepared earlier). We can stop here, and it will still bring us some value. Pact can be used as a mock server, similarly to Wiremock, which we mentioned earlier.

Let’s focus on two annotations now. The first one (@Pact) defines the consumer name and marks the method as a Pack method. The second annotation (@PactTestFor) connects the Pact method with a test case. That’s how the test knows how to stub API.

The last thing we need to add before we run the tests is the annotations used to let the test class know that we want to bring up the Spring context and enable Pact.

The test passed, and that’s the last point that CDC testing and testing with a Mock Server have in common. That’s because apart from the verification of our test case, the JSON file containing a contract has been generated in the target directory (target/pacts).

sensor_management

sensor_provider

As you can see, the contract is quite concise. Apart from json boilerplate, it contains only useful information about the expected API behavior.

Note that this contract has been simplified. Usually, the interactions node has many more children, each one describing a single Pact test case. I choose one GET and one POST request. Every interaction entry has:

provider state - it allows the provider to set up its state when validating the contract

response - expected, returned by the provider

Everything until this moment happened on the consumer's side. Now we need to send the contract to the provider and offer them a chance to validate it.

There are many ways we can share our contracts. It’s possible to use almost any way that includes S3/FTP/SCP. But Pact offers a component for storing and sharing contracts called Pact Broker. It provides many features out-of-the-box, which we would need to implement manually in other solutions (for example, S3).

We can use several different options for implementing the Pact Broker into the CDC flow. It can be deployed into our existing infrastructure because it’s an open-source project. It can also be used like a Software-as-as-Service available on https://pactflow.io/. In this article, we’re going to use the SaaS version of Pact Broker. After the registration, we get our own Pact Broker URL and API tokens (read and read/write).

Pact Broker exposes HTTP API we can use to manage contracts. But there are also plugins that make the process of sharing contracts easy to integrate with the existing build process.

And we’re ready to generate a contract:

That concludes everything we need to do on the consumer side. Now it’s time for the provider part.

The consumer generates a contract that describes what they want rather than referring to the provider’s real capacity to fulfill it. Each contract has to be verified by the provider to be treated as applicable. In our case, the provider is a simple 

 app that exposes two HTTP endpoints. It has the logic needed to validate the contract by design because is has some domain implemented. But we need to tweak the provider a bit to be able to fetch the contract and use it as an input to a test suite.

We need to do two things. First, we have to add a dependency to the Pact provider library (in our case, it’s a version for Junit5, but there are many more options):

Then we need to prepare code capable of verifying the contract.

This is the Junit5 test that can be divided into two parts: lines 1-5 are responsible for setting up Pact. We need to define how to connect with Pact Broker. We have also to tell Pact where it can expect the provider API (lines 16-19). Finally, we have also enabled the publishing verification process result to Pact Broker.

The second part manages the test flow. We inform the JUnit of how to perform the test in lines 26-30. It’s one line delivered by the Pact library we have added.

The next three methods are responsible for setting up the system to the state expected by the contract. Each case written in a contract defines the state in which the system has to be to act in some way. For instance, the contract we have prepared started with the following lines:

We now have to define this state so that the two sensors are stored and then returned by the API.

Some states don’t require any specific set up steps because the system is able to handle some behaviors on its own - for example, validating requests and managing access to non-existing resources.

Let’s verify whether the consumer expectations and producer capabilities have met.

The Pact junit library has quite good logs. Among them, the most important one for us is the following:

Let’s briefly check the pactflow.io dashboard to check if everything we did is reflected here:

As you can see, the contract has been shared and verified by the provider.

You may have noticed that even though we used a few tools, we still several manual steps we needed to take to produce, share, and verify the contract. More importantly, we didn’t do anything with the result yet. The ultimate goal of the CDC is to integrate it with the CI/CD pipelines and make everything work automatically.

In the end, the most important outcome of this process is the decision to go/no go to production. We’re going to explore this topic further in the next article of this short series about Consumer-Driven Contracts. Stay tuned!

Consumer-Driven Contract (2)

Third part of our Consumer-Driven Contact series of articles.

Consumer-Driven Contract (1) - how to increase stability in distributed systems.

Consumer-Driven Contract (2) - Pact-based implementation

 we described the base principles and implementation of the Consumer-Driven Contract without any automation.

In this article, we’re going to orchestrate CDC and make them part of the existing CI/CD pipelines. We will refer to the code from this post, so if you haven’t read it, some parts of this post may be unclear.

We’ve already chosen Pact as a tool that will help us introduce CDC. Naturally, we also decided to use the Pactflow platform that plays the role of a Pact Broker. This time, we will focus much more on the platform than on our code. We will prepare a simple pipeline using Gitlab CI/CD cooperating with a Pact Broker. Let’s dive into the details now!

This Picture describes the Customer CI/CD flow responsible for validating contracts.

Everything starts with a change in the Consumer codebase. It triggers the entire pipeline and, as a part of it, we’re going to check whether the Consumer assumptions of what can be delivered by the producer are valid. We described the process of preparing the contract and uploading it to the broker 

, but just to recall it: we need two commands here. One for generating the contract:

Before we move those two commands into the pipeline, let’s focus on the important part of publishing contract which is contract versioning.

The contract version is the internal Pact value that is not exposed to the users. But each contract may be identified by the consumer version and provider version. So instead of thinking in terms of contract versioning, we should switch to versioning of consumer and provider apps.

According to Pact best practices - which are similar to the best practices of versioning in general - we should be able to refer from the app version to some point in SCM easily.

The easiest way to do it is by appending the commit hash to the version. In real life, we would need to prepare some release scripts for our projects, but for the sake of simplicity, we’re going to use the feature of the pact Gradle plugin that allows specifying the consumer version explicitly.

As you can see, the parameter name is “providerVersion”. It’s a bit tricky because it's the consumer, not the provider, who publishes the contract. But maybe designers decided to name the parameter that way because the API consumer actually acts as a contract provider.

CONSUMER_VERSION will be populated by the GitLab pipeline with the current value during the pipeline run. We’re ready to prepare the first box from the diagram.

Here is the part of the GitLab pipeline responsible for generating and pushing the contract.

There are two steps we need to take here:

running a tests suite that builds the contract

and invoking the pact gradle plugin for publishing it.

Triggering the producer to validate the contract

Pact Flow offers two main types of webhooks:

A webhook has a form of an HTTP POST call. When a new contract is published, what we want to do is trigger the API Provider pipeline able to validate the contract. We’re going to prepare the Provider pipeline in the next section.

Let’s focus now on triggering the Provider pipeline. The Gitlab platform has an HTTP API allowing us to integrate the service with other tools. We can easily generate an API token to trigger project pipelines:

We have the url now, so we can configure the Pack flow webhook:

The code responsible for validating the contract has been presented in a previous 

. Let’s just go back to the command used to execute the check.

As we can see, it looks like something we can move to the CI/CD pipeline easily.

There some additional commands required by test containers used for testing, but functionally, it’s just invoking a single Gradle task. As we build the contract validation logic using the Pact library, the verification result is shared with the Pact Broker automatically. Now it’s time to push the customer’s pipeline waiting for information about the finished contract validation.

Here’s where Pact Flow webhooks come into play again. But it’s going to be a bit more complicated this time. Gitlab’s API allows us to continue a stopped pipeline, but we need to know the id of the job we want to push. Any time a job runs, a new instance with a new id of the job is created. We can’t prepare a static URL to continue the consumer’s pipeline. Since you can run two pipelines at the same time, referring to a specific pipeline for running it further is impossible. How to determine which pipeline should be unlocked?

Fortunately, there are few features and tools that make it possible.

First, there are the dynamic variables that can be used to construct a Pact Flow webhook url or body. So our endpoint may contain, for instance, consumer’s name and version, producer’s name and version, and more different variables. We decided to use the commit’s short id as a part of the customer version. So the webhook url may contain a commit identifier. But that’s not enough to trigger a blocked pipeline job. We have to find the job id.

As we already mentioned, Gitlab exposes the HTTP API. A job is a first-class entity, so we can perform a useful operation on it. For instance, we can look for all project’s jobs with a specific scope. So we can have something like this:

Note: projectId can be found in the Gitlab frontend.

We’re interested in jobs with a manual scope because that’s how we implemented the job waiting to be triggered. A manual job is a job that can be triggered by the click or API call. As a response, we get a detailed description of jobs. I removed 90% of the response to focus on the parts that are most important for us:

can_i_deploy

Good news: we have the commit short id and the job’s id! So in theory, we found a way to map the commit short id to the job id (or multiple ids). Now we have to trigger the the job. Here’s the Gitlab API method we can use:

We have the theory ready. But where to put all this code? Pact Flow allows entering a string as a webhook url. It sounds like a good candidate for AWS Lambda. And that’s how we implemented it. Lambda exposes the endpoint accepting the commit short id (acting as consumer version) and creates a really simple algorithm:

Get all jobs from a specified project with scope "manual,"

Find jobs with the commit short id matching requested one,

Trigger jobs based on job ids from step 2.

Lambda is exposed to the world using the AWS API Gateway and can be called by the Pact Flow webhook. Here is the most important part of the Lambda querying jobs and filtering the commit short id:

We’re not going to discuss the topic of security at the moment but, of course, such an endpoint needs to be secured somehow.

The code above is simple, but someone may say that it’s not the best option to maintain such a custom code. And actually, who should maintain it? But if you think about how many different CI/CD platforms companies use, there are some concepts in one platform that don’t exist in others, or have different definitions.

A successful introduction of CDC requires work on both sides, consumer and producer. In my opinion, this specific code should be maintained by the consumer. The consumer should know how to trigger its pipeline.

There’s another very important reason to keep this code on the consumer side which is security. Giving external company access to the API of our Gitlab may not be an option. Exposing one very specific method sounds much safer.

The webhook is triggered when contract is verified, no matter the result of the verification. The HTTP POST call to the registered url will be performed even if the contract doesn’t pass verification. It’s the customer’s responsibility to decide whether the new version should be deployed or not. Pact prepares a cli tool that ensures we can deploy some version of our code to the server safely. The tool has a self-descriptive name “can-I-deploy”. We have to specify a few parameters and, as a result, we get a go/no go decision. The invocation in our case looks as follows:

We can install cli manually, but Pact also delivers a Docker image containing all required dependencies. That makes wrapping this invocation in a pipeline job really straightforward:

This crowns the introduction of Customer-Driven Contracts into the Consumer’s CI/CD pipeline. Here’s the full Consumer’s pipeline diagram:

Customer-Driven Contracts/Testing is a pattern that is relatively expensive to set up. Once set up, it also requires cooperation between different teams, systems, and tools - which can be time-consuming. But in the end, we can have much more confidence when applying any changes to our API.

How to build consumer CI/CD pipelines with Gitlab and Pactflow. Click to read the whole article.

Consumer-Driven Contract (3)

Krzysztof is our Head of Products and he leads the technical development behind the suite of our ready-made solutions. Throughout the years in Solidstudio, he gained an invaluable insight into the eMobilty industry’s requirements and is now more than qualified to share his knowledge with others.

Krzysztof Ciombor

. Read it first, if you haven’t already. Today, we’re going to take a closer look at snapshot testing to show you how we improved it and took it to the next level.

The original idea of Snapshot Testing was first included in Jest - have a look 

 to learn more. Its main purpose was testing frontend component changes over time to catch any unexpected UI changes. This concept, however, is more universal and can be applied to almost all codebases. The overall process is always the same:

Execute your code (either by rendering the UI, calling a method, or issuing an HTTP request).

If you were running the test for the first time, save the result to a “snapshot” file.

Otherwise, find an existing “snapshot” file and compare it with the result. If the contents are the same, pass the test. If the contents don’t match, report the test failure.

By using snapshot testing, we get a few benefits:

The tests are easy to write and maintain.

Since they are essentially similar to integration/acceptance tests, we’ll be able to validate the flow through the entire application stack (i.e. HTTP requests, Service layer, DB integration, etc.).)

They are a “simpler” alternative to Contract Based Testing (see 

) that still give us confidence that we’re not introducing any breaking changes to our APIs.

Our initial implementation of Snapshot Tests was based on the 

 library. It worked great at the beginning, but a couple of issues started to arise once we added more and more tests. One of the most frequent ones were non-deterministic JSON outputs, for example, when generating random UUIDs. We were frequently facing failures like this:

Even though it makes sense from the String equality perspective, it doesn't really matter from business requirements side. As long as the output JSON contains the id field, we don’t really care about its content. This can be mitigated by creating deterministic helpers for UUID creation.

The next issue started happening when we compared arrays. Our output was very often serialized with arrays contents in random order.

The second biggest issue we faced was the order of arrays. Even though the serialization itself should preserve the elements order, we can’t guarantee that the collections were always sorted the same (for example, a simple .addAll() on a Collection doesn’t have any ordering guarantees). Again, this became a problem when comparing snapshots as strings, even though from a logical standpoint, as long as the array elements are the same, the ordering doesn’t really matter.

So the question we started asking ourselves: “Can we do better than this?" And the answer is… “Yes we can!”

For our purpose here, we’re using snapshots for validating API responses. Since they always come in the form of JSON, we can take advantage of that. As it turns out, there’s a library out there that is perfect for our 

. However, this also means that we had to ditch the KotlinSnapshot library in favor of a custom implementation. Fortunately, that’s not so complicated to achieve and in our PoC we were able to (almost) replicate all the functionalities of KotlinSnapshot, re-using a couple of its code snippets. One of our goals was to keep the compatibility between both to facilitate the migration of the existing tests. In the end, the only required for existing test code is as simple as:

So what exactly do we get by switching from String comparison to JSON comparison? Here are two major benefits:

Taking advantage of JSONassert’s JSONCompareMode.NON_EXTENSIBLE comparison mode, we can compare an array’s contents disregarding their order, but still guarantee that no element is missing and no element was added. This means that we no longer have to focus our efforts for making the array order deterministic and can instead concentrate on the array’s contents.

Maybe a more interesting part comes with custom validation config which can be passed to validateSnapshot method. Using JSONassert’s full potential, we can ignore properties that we don’t want to validate during the snapshot comparison. Going back to our previous example, we can now easily exclude id fields from being checked for equality by doing:

Moreover, we can use a wildcard operator for more sophisticated exclusions, namely:

 - Will match all id fields, no matter the nesting level

 - Will match ALL fields (useful for doing selected fields comparison only)

Keep in mind that ignoring the field for validation only skips the comparison part. The field still needs to be present in the response, so we don’t need to worry that we’ll introduce API-breaking changes by removing some response fields.

One should always keep in mind that there’s no such thing as “silver bullet” in software development. Each solution comes with its tradeoffs. That’s the case with snapshot testing as well. There are still a couple of pain points with using it as the main testing method, namely:

These tests are much “heavier” than unit tests, meaning that the “feedback loop” is considerably longer.

Sometimes it will be difficult to reason about failures and pinpoint the exact erroneous place in the code.

Snapshot files require maintenance, they must be updated every time a change in the API is correct and required.

Assertions are not obvious and it may not be immediately visible what is being tested just by looking at the test code.

Pull Request diffs “blow up”. A single change to add one more parameters to the response DTO may cause dozens of snapshot to be updated. Since snapshot files should always be committed to the repo, those changes will show up in the code review.

As usual, it’s best to always evaluate potential solutions and new approaches with care and adjust them to our needs in the project. What worked well for us may not work so well for others. We are quite happy with our current test stack, and we’re able to achieve ~80% code coverage easily with just snapshot tests and a couple of unit tests mixed here and there for the most crucial business logic. This allows us to be confident when making changes, adding new features, and refactoring the old code.

How to take snapshot testing to a next level? Read our article.