Each week, we’re asked hundreds of questions on our Slack channel, from “my test is failing, please help” to “I need help convincing my team.” Exhibit A:
We’ve already created a list of reasons to convince you that contract testing is a good idea, but today I’d like to take the opportunity to answer a question that comes up often – what is contract testing, and why should you consider adding it to your microservices testing approach?
In this post, we’ll cover:
- Integration testing and the challenges with end-to-end integrated tests
- Contract testing - what it is and why it helps
- Pact - how Pact works
- Demo of contract testing with Pact
You also can watch this as a recorded video series.
Integration testing
Before we talk about contract testing, we should talk about why contract testing exists. It exists to help with integration testing – the process by which we build confidence that a system works as a whole.
In a distributed system, integration testing is a process that helps us validate that the various moving parts that communicate remotely – things like microservices, web applications, and mobile applications – all work together cohesively.
There are many types of integration testing, but the most common approach is called “end-to-end integrated testing,” which involves all the components being deployed together in a real environment – one that closely resembles production – and running a battery of test scenarios against it.
While the concept of contract testing predates Pact, Pact exists because end-to-end integrated tests are a scam.
The problem with end-to-end integrated tests
While tests at the top of the pyramid more closely represent what the customer would experience, they have several painful drawbacks. They:
- are slow. Because they traverse multiple systems and generally must be run serially, each test may take several seconds to several minutes, especially if prerequisite setup (such as data preparation) must be performed.
- are hard to maintain. End-to-end tests require all systems to be in the correct state before they are run, including the correct version and data.
- can be unreliable or flakey. Because of the complexity in orchestrating a test environment, they can often fail, causing false-positives and becoming a distraction to the team. In many cases, they fail due to a configuration issue unrelated to any code change.
- are hard to fix. When an end-to-end test fails, debugging the issue is usually difficult because of the distributed and remote nature of the problem.
- scale badly. As more teams’ code gets tested, things get more entangled, test suites run exponentially slower, and releases get clogged in automation pipelines.
- find bugs too late in the process. Because of the complexity of running such testing suites, in many situations these tests are only run on CI after code has been committed – in many cases, by a separate testing team days afterward. This delay in feedback is extremely costly to modern, agile delivery teams.
Because of these properties, it is advised to keep end-to-end integrated tests to a minimum number to ensure key business transactions or features are covered.
See proving end-to-end tests are a scam to dive deeper into this.
What is contract testing?
Contract testing is a methodology for ensuring that two separate systems (such as two microservices) are compatible and can communicate with one other. It captures the interactions that are exchanged between each service, storing them in a contract, which then can be used to verify that both parties adhere to it. Contract testing goes beyond schema testing, requiring both parties to come to a consensus on the allowed set of interactions and allowing for evolution over time.
What sets this form of testing apart from other approaches that aim to achieve the same thing is that each system can be tested independently from the other and that the contract is generated by the code itself, meaning the contract is always kept up to date.
The following diagram shows the key steps in contract testing:
There are many other important properties that flow on from this, which we’ll discuss further.
How does Pact implement contract testing?
Pact is a code-first contract testing tool, which requires access to the code on both sides of an integration point. To write Pact tests, you need to be able to write a unit test of the consumer and be able to manipulate state (usually within the context of a unit test) on the provider side.
How does Pactflow implement contract testing?
While Pactflow supports Pact as its primary contract testing tool, it also enables a broader range of tools to be used in the contract testing process (such as Postman, Dredd, or other service virtualization and mocking tools), providing a way to “upgrade” them into a general contract testing capability. See our blog on bi-directional contract testing for more.
What is consumer-driven contract testing?
The “consumer-driven” prefix simply states an additional philosophical position that advocates for better internal microservices design by putting the consumers of such APIs at the heart of the design process. Provider-driven APIs tend to be biased toward the data that is being exposed and the system that is exposing it.
Pact and Spring Cloud Contracts are examples of frameworks that default to this kind of implementation, but this isn’t a requirement to get the key benefits of contract testing and both can be used to achieve the aims of contract testing.
What are the benefits of contract testing?
It’s helpful first to consider where contract testing sits within the context of a broader automation testing approach. When creating a test automation strategy, a good rule of thumb in how you should expend your effort is the approach advocated in Mike Cohn’s Test Pyramid:
Stick to the pyramid shape to come up with a healthy, fast and maintainable test suite: Write lots of small and fast unit tests. Write some more coarse-grained tests and very few high-level tests that test your application from end to end [3]
Contract tests fit in the Service Tests layer, as they execute quickly and don’t need to integrate to external systems to run. Their job is to give you confidence that the systems you integrate with are compatible with your code before you release.
NOTE: The UI Tests at the top of the pyramid here are often also referred to interchangeably with “integrated end-to-end (e2e) tests.”
The value of contract tests
Contract tests generally have the opposite properties to integrated e2e tests.
- They run fast, because they don't need to talk to multiple systems.
- They are easier to maintain. You don’t need to understand the entire ecosystem to write your tests.
- They are easy to debug and fix, because the problem is only ever in the component your testing – so you generally get a line number or a specific API endpoint that is failing.
- They are repeatable.
- They scale; because each component can be independently tested, build pipelines don’t increase linearly/exponentially in time.
- They uncover bugs locally on developer machines. Contract tests can and should run on developer machines before pushing code.
From a business point of view, it is well known that the later in a project life cycle that a bug is found, the costlier it is to fix.
By running most of your automation tests as fast unit or local integration tests, you can keep build speeds to a minimum and prevent queues building up in teams.
Other benefits
It turns out that contract tests also have other secondary yet positive side effects.
- The ability to develop the consumer (e.g., a React Web App) before the API.
- The ability to drive out the requirements for your provider first, meaning you implement exactly and only what you need in the provider.
- You get a set of well-documented use cases (“Given ... a request for ... will return ...”) that show exactly how a provider is being used.
- The ability to see exactly which fields each consumer is interested in, allowing unused fields to be removed and new fields to be added in the provider API without impacting a consumer.
- The ability to immediately see which consumers will be broken if a change is made to the provider API.
I hope this brief introduction to the topic showed you how you can use contract testing to replace many (or in some cases, all) of your end-to-end integrated tests to speed up your CI pipeline and increase your teams’ velocity.
Contract testing is not a silver bullet, and no two teams are the same, so think about your test strategy, how it applies to you, and what you need to get from it.
