Extra Pact Matrix documentation

[TimothyJones]

There's information in this blog post that should probably be distilled and added to the documentation:

http://rea.tech/enter-the-pact-matrix-or-how-to-decouple-the-release-cycles-of-your-microservices/

[mboudreau]

@bethesque could you help with this?

[YOU54F]

Just preserving this content and will check where we don't have information in the docs and transpose accordingly

---

Enter the Pact Matrix. Or, how to decouple the release cycles of your microservices

So, you’re writing microservices! You’re feeling pretty smug, because microservices are all the rage. All the cool kids are doing it. You’re breaking up your sprawling monoliths into small services that Do One Thing Well. You’re even using consumer driven contract testing to ensure that all your services are compatible.

Then… you discover that your consumer’s requirements have changed, and you need to make a change to your provider. You coordinate the consumer and provider codebases so that the contract tests still pass, and then, because you’re Deploying Early and Often, you release the provider. Immediately, your monitoring system goes red – the production consumer still expects the provider’s old interface. You realise that when you make a change, you need to deploy your consumer and your provider together. You think to yourself, screw this microservices thing, if I have to deploy them together anyway, they may as well go in the one codebase.

This is one of the challenges you will face when writing microservices. The aim is to decouple your codebases from one another, but occasionally and unavoidably, things will need to change in a breaking way. What can you do then? You could try versioning your API, but anyone who has tried maintaining more than one version of an API at a time won’t be particularly enthusiastic about this option.

Another option is to try to make all your changes backwards compatible until the dependent systems have updated their code, but how can you be sure that the head version of your service is backwards compatible with the other services in its context without actually deploying to production? One option that we tried at realestate.com.au prior to using contract testing was to give each service its own integration test environment, called a “certification environment”, with a copy of all the production services in its context, and the latest version of the service under test. The integration tests were then run in each certification environment …. turns out that if you have n services in your ecosystem, you need n² instances to run these tests, and the amount of effort to maintain all those environments and all those slightly differing test suites, as well as a copy of their “old world” monolith dependencies… well, let’s just say it Did Not Scale.

But remember that we (and the hypothetical you in this story) were using consumer driven contracts? Those contracts are the key to entering the Matrix, where you will discover a whole new way of looking at the world…

When you use contracts, you generally test the head versions of the consumer and provider codebases against each other, so you know whether or not they are compatible. The production versions of the consumer and provider are probably also compatible, because you tested them against each other back when they were both at head. But if you want to be able to deploy your consumer and provider independently of each other there are two other things you need to know. Is the head version of your consumer compatible with the production version of your provider? And is the head version of your provider compatible with the production version of your consumer?

If you put those 4 things together, you get a matrix that looks like this.

	Consumer Head	Consumer Prod
Provider Head	Contract tests	Unknown!!!
Provider Prod	Unknown!!!	Already tested

By testing the Pact Matrix, you can be confident to deploy any service at any time, because your standalone CI tests have told you whether or not you are backwards compatible – no “certification environment” needed. And when there are multiple services in a context, this approach scales linearly, not exponentially like the certification environment approach.

At realestate.com.au, we use a tool to help us share pacts between our consumer and provider projects that also gives us the ability to tag the production version of a pact for use in the Pact Matrix. But that is a topic for another post! Keep an eye out on the tech blog for my upcoming post on the Pact Broker.

So, you’re writing microservices! You’re feeling pretty smug, because microservices are all the rage. All the cool kids are doing it. You’re breaking up your sprawling monoliths into small services that Do One Thing Well. You’re even using consumer driven contract testing to ensure that all your services are compatible.

Then… you discover that your consumer’s requirements have changed, and you need to make a change to your provider. You coordinate the consumer and provider codebases so that the contract tests still pass, and then, because you’re Deploying Early and Often, you release the provider. Immediately, your monitoring system goes red – the production consumer still expects the provider’s old interface. You realise that when you make a change, you need to deploy your consumer and your provider together. You think to yourself, screw this microservices thing, if I have to deploy them together anyway, they may as well go in the one codebase.

This is one of the challenges you will face when writing microservices. The aim is to decouple your codebases from one another, but occasionally and unavoidably, things will need to change in a breaking way. What can you do then? You could try versioning your API, but anyone who has tried maintaining more than one version of an API at a time won’t be particularly enthusiastic about this option.

Another option is to try to make all your changes backwards compatible until the dependent systems have updated their code, but how can you be sure that the head version of your service is backwards compatible with the other services in its context without actually deploying to production? One option that we tried at realestate.com.au prior to using contract testing was to give each service its own integration test environment, called a “certification environment”, with a copy of all the production services in its context, and the latest version of the service under test. The integration tests were then run in each certification environment …. turns out that if you have n services in your ecosystem, you need n2 instances to run these tests, and the amount of effort to maintain all those environments and all those slightly differing test suites, as well as a copy of their “old world” monolith dependencies… well, let’s just say it Did Not Scale.

But remember that we (and the hypothetical you in this story) were using consumer driven contracts? Those contracts are the key to entering the Matrix, where you will discover a whole new way of looking at the world…

When you use contracts, you generally test the head versions of the consumer and provider codebases against each other, so you know whether or not they are compatible. The production versions of the consumer and provider are probably also compatible, because you tested them against each other back when they were both at head. But if you want to be able to deploy your consumer and provider independently of each other there are two other things you need to know. Is the head version of your consumer compatible with the production version of your provider? And is the head version of your provider compatible with the production version of your consumer?

If you put those 4 things together, you get a matrix that looks like this.

Consumer Head Consumer Prod
Provider Head Contract tests Unknown!!!
Provider Prod Unknown!!! Already tested
Let’s get specific about the contract testing. Let’s assume you’re using Pact, a consumer driven contract testing library developed at realestate.com.au. When you’re doing contact testing with Pact, the process goes like this:

The CI build in the consumer project runs tests against a mock provider, provided by the Pact library.
The mock provider records the requests and the expected responses into a JSON pact file.
The consumer CI build publishes pact file to a known URL or copies it into the provider codebase.
The provider CI build retrieves the pact from that known location, and runs the “pact verification” task against the provider codebase.
The verification task replays each request in the pact against the provider, and compares the actual responses with the expected responses.
If they match, we know the two projects are compatible.
Once a pact file has been generated by a consumer, all we need to determine if version X of the consumer is compatible with version Y of the provider, is the pact from version X of the consumer, and the codebase from version Y of the provider. If, instead of just verifying the latest version of a pact against the provider, we also verify the production version of the pact, we will know that our provider is backwards compatible, and can be deployed at any time. Conversely, if we check out the production version of the provider codebase, and verify the latest pact against it, we will know that our consumer is backwards compatible, and can be deployed at any time.

Let’s enter the Pact Matrix.

Consumer Head (pact) Consumer Prod (pact)
Provider Head (codebase) Contract tests Contract tests
(ensure provider is backwards compatible)
Provider Prod (codebase) Contract tests
(ensure consumer is backwards compatible) Already tested
By testing the Pact Matrix, you can be confident to deploy any service at any time, because your standalone CI tests have told you whether or not you are backwards compatible – no “certification environment” needed. And when there are multiple services in a context, this approach scales linearly, not exponentially like the certification environment approach.

At realestate.com.au, we use a tool to help us share pacts between our consumer and provider projects that also gives us the ability to tag the production version of a pact for use in the Pact Matrix. But that is a topic for another post! Keep an eye out on the tech blog for my upcoming post on the Pact Broker.

[YOU54F]

Although not relevant to the original issue, this is possibly good content somewhere, even if just referenced from the community page, so just preserving here for myself as a note more than anything else

---

A microservices implementation retrospective

Over the last year at realestate.com.au (REA), I worked on two integration projects that involved synchronising data between large, third party applications. We implemented the synchronisation functionality using microservices. Our team, along with many others at REA, chose to use a microservice architecture to avoid the problems associated with the “tightly coupled monolith” anti-pattern, and make services that are easy to maintain, reuse and even rewrite.

Our design used microservices in 3 different roles:

Stable interfaces – in front of each application we put a service that exposed a RESTful API for the underlying domain objects. This minimised the amount of coupling between the internals of the application and the other services.
Event feeds – each “change” to the domain objects that we cared about within the third party applications was exposed by an event feed service.
Synchronisers – the “sync” services ran at regular intervals, reading from the event feeds, then using the stable interfaces to make the appropriate data updates.
Integration Microservices Design

Things that worked well
Using a template project to get started
At REA we maintain a template project called “Stencil” which has the bare bones of a microservice, with optional features such as a database connection or a triggered task. It is immediately deployable, so a simple new service can be created and deployed within a few hours.

Making our services resilient
We started “lean” with synchronous tasks that were triggered by hitting an endpoint on the service. One of the down sides of splitting out code into separate services is that there is an increased likelihood of errors due to network gremlins, timeouts and third party systems going down. Failure is always an option. In our synchronous, single-try world, the number of unnecessary alerts which required manual intervention just to kick of the process again was a drain on our time. So, we changed all our services to use background jobs with retries, and revelled in the relative calm.

Making calls idempotent
Given that we had built in retries, each part of our retry-able code needed to be idempotent so that a retry would not corrupt our data. Using PUT and PATCH is great for this, but sometimes we did have to do a GET and make a check before making the next request.

Using consumer driven contract testing
Testing data flows involving 4 microservices, two third party applications and triggered jobs using traditional integration tests would have been a nightmare. We used Pact, an open source “consumer driven contracts” gem developed by one of REA’s own teams, to test the interactions between our services. This gave us confidence to deploy to production knowing our services would talk to each other correctly, without the overhead of integration test maintenance.

Where possible, exposing meaningful business events, not raw data changes
It took a while to really grasp the meaning of this, however once we “got” it, it made sense. It is probably easiest to explain by example. One domain object had a “probability” percentage field that could be changed directly by a user. Instead of exposing “probability field changed” as an event, we exposed “escalations”. This meant we were hiding the actual implementation of how the “rise in probability” was executed in the system, and not asking every other system that inspected the event feed to have to re-implement the logic of “new value of probability is greater than old value, therefore its likelihood has increased”.

Automating all the things
We are lucky enough to be able to use AWS for all our development, test and production environments. We used continuous deployment to our development environment, and we had a script to deploy the entire suite of microservices to the test environment at one click. This made the workflow painless, and helped counteract the overhead of having so many codebases.

Using HAL and the HAL Browser
HAL is a lightweight JSON (and XML if you are so inclined) standard for exposing and navigating links between HTTP resources. The “Stencil” app comes with Mike Kelly’s HAL browser already included (this is just an HTML page that lets you navigate through the HAL responses like a web browser). As well as the resources for the business functionality, we created simple endpoints that exposed debugging and diagnostic data such as “status of connection to dependencies” or “last processed event” and included links in the index resource so that finding information about the state of the service was trivially easy, even for someone who didn’t have much prior knowledge of the application.

Things that didn’t work well
Coming up with a way to easily share code between projects
Our first microservices implementation used the strict rule of “one service has one endpoint”. This made services that could be easily deployed separately without affecting or holding up other development work. However, it also increased the maintenance overhead, as each new service was made from a copy of the previous one and then modified to suit. When a problem was found in the design of one of them, or we wanted to add a new feature, then we had to go and change the same code (with just enough variations to be annoying) in each of the other projects. The common code was more structural than business logic (eg. Rakefile, config.ru, configuration, logging), and it was not written in a way that made it easy to put in a gem for sharing.

Things we have questions about
What is the “right size” for a microservice?
Soon after having completed the first microservices integration project, we had an opportunity to do a second. This time, instead of making many different “event feed” services that each exposed a single type of event, we made one event service that had an endpoint for each different type of event. Some might argue that we were stretching the definition of “microservice”, however, there was still at tight cohesion between the endpoints, as they were all exposing events for objects in the same underlying aggregate root. For us, the payoff of having fewer codebases and less code to maintain made the trade-off worth it, as the turnaround for a exposing a new type of event was a matter of hours, instead of a matter of days.

Integration Microservices Design, Take 2

I suspect that the “right size” is going to vary between projects, languages, companies and developers. I’m actually glad we made our services what I now consider to be “too small” in the first project, just as an experiment to work out where the line was for us. I now think of the “micro” as pertaining more to the “purpose” of a service than the size. Perhaps “single purpose service” would be a better term – but it just ain’t as catchy!

[bethesque]

I suspect that the “right size” is going to vary between projects, languages, companies and developers. I’m actually glad we made our services what I now consider to be “too small” in the first project, just as an experiment to work out where the line was for us. I now think of the “micro” as pertaining more to the “purpose” of a service than the size. Perhaps “single purpose service” would be a better term – but it just ain’t as catchy!

If I ever went on another conference presentation campaign, this would be my topic!