Reading:
What is contract testing and why should I try it?

What is contract testing and why should I try it?

Matt Fellows
What is contract testing and why should I try it?

Each week we're asked 100's of questions on our Slack channel, from "my test is failing, please help" to "I need help convincing my team". Exhibit A:

Convince me!

We've already created a list of reasons to convince you that contract testing is a good idea, but today I'd like to take the opportunity to answer a question that comes up often - what is contract testing, and why should you consider adding it to your microservices testing approach?

In this post we'll cover the following, with accompanying videos:

  1. Integration testing and the challenges with end-to-end integrated tests
  2. Contract testing - what it is and why it helps
  3. Pact - how Pact works
  4. Demo of contract testing with Pact

Integration testing

Before we talk about contract testing, it's first appropriate to talk about why contract testing exists. They exist to help with integration testing - the process by which we build confidence that a system works as a whole.

In a distributed system, integration testing is a process that helps us validate that the various moving parts that communicate remotely - things like microservices, web applications and mobile applications - all work together cohesively.

There are many types of integration testing, but the most commonly relied on approach is what is called “end-to-end integrated testing”, which involves all of the components being deployed together in a real environment - one that closely resembles production - and running a battery of test scenarios against it.

Whilst the concept of contract testing predates Pact, Pact exists because end-to-end integrated tests are a scam.

The problem with end-to-end integrated tests

watch: the problem with end-to-end integrated tests

Whilst tests at the top of the pyramid more closely represent what the customer would experience, they have several painful drawbacks. They:

  • are slow; because they traverse multiple systems and generally must be run serially, each test may take several seconds to several minutes to complete, especially if pre-requisite setup (such as data preparation) must be performed.
  • are hard to maintain; end-to-end tests require all systems to be in the correct state before they are run, including the correct version and data.
  • can be unreliable or flakey: because of the complexity in orchestrating a test environment, they can often fail causing false-positives, becoming distractions to the team. In many cases, they fail due to a configuration issue unrelated to any code change.
  • are hard to fix: when an end-to-end test fails, debugging the issue is usually difficult, because of the distributed and remote nature of the problem.
  • scale badly; as more teams' code gets tested, things get more entangled, test suites run exponentially slower and releases get clogged in automation pipelines.
  • find bugs too late in the process: because of the complexity of running such testing suites, in many situations these tests are only run on CI after code has been committed - in many cases, by a separate testing team days afterwards. This delay in feedback is extremely costly to modern, agile delivery teams.

Because of these properties, it is advised to keep them to a minimum number e.g. to ensure key business transactions or features are covered.

See proving end-to-end tests are a scam to dive deeper into this.

What is contract testing?

watch: explanation of contract testing and how Pact works

Contract testing is a methodology for ensuring that two separate systems (such as two microservices) are compatible and are able to communicate with one other. It captures the interactions that are exchanged between each service, storing them in a contract, which can then be used to verify that both parties adhere to it. Contract testing goes beyond schema testing, requiring both parties to come to a consensus on the allowed set of interactions and allowing for evolution over time.

What sets this form of testing apart from other approaches that aim to achieve the same thing, is that each system is able to be tested independently from the other and that the contract is generated by the code itself, meaning the contract is always kept up to date with reality.

The following diagram shows the key steps in contract testing:

how contract testing works

There are many other important properties that flow on from this, which we'll discuss further below.

How does Pact implement contract testing

Pact is a code-first contract testing tool, that requires access to the code on both sides of an integration point. To be able to write Pact tests, you need to be able to write a unit test of the consumer, and to be able to manipulate state (usually within the context of a unit test) on the provider side.

How does Pactflow implement contract testing?

Whilst Pactflow supports Pact as its primary contract testing tool, it also enables a broader range of tools to be used in the contract testing process (such as Postman, Dredd or other service virtualisation and mocking tools), providing a way to "upgrade" them into a general contract testing capability. See our blog on bi-directional contract testing for more.

What is consumer-driven contract testing?

The "consumer-driven" prefix simply states an additional philosophical position that advocates for better internal microservices design by putting the consumers of such APIs at the heart of the design process. Provider-driven APIs tend to be biased towards the data that is being exposed and the system that is exposing it.

Pact and Spring Cloud Contracts are examples of frameworks that default to this kind of implementation, but this isn't a requirement to get the key benefits of contract-testing and both can be used to achieve the aims of contract testing.

What are the benefits of contract testing?

watch: contract-testing in the test pyramid and how to remove end-to-end integrated tests

It's helpful first to consider where contract testing sits within the context of a broader automation testing approach. When coming up with a test automation strategy, a good rule of thumb in how you should expend your effort is the approach advocated in Mike Cohn's "Test Pyramid":

Stick to the pyramid shape to come up with a healthy, fast and maintainable test suite: Write lots of small and fast unit tests. Write some more coarse-grained tests and very few high-level tests that test your application from end to end [3]
From the Practical Test Pyramid

Contract tests fit in the "Service Tests" layer, as they execute quickly and don't need to integrate to external systems to run. Their job is to give you confidence that the systems you integrate with are compatible with your code before you release.

NOTE: The "UI tests" at the top of the pyramid here are often also referred to interchangeably with "integrated end-to-end (e2e) tests".

The value of contract tests

Contract tests generally have the opposite properties to e2e integrated tests:

  • They run fast, because they don't need to talk to multiple systems.
  • They are are easier to maintain: you don't need to understand the entire ecosystem to write your tests.
  • They are easy to debug and fix, because the problem is only ever in the component your testing - so you generally get a line number or a specific API endpoint that is failing.
  • They are repeatable:
  • They scale: because each component can be independently tested, build pipelines don't increase linearly / exponentially in time
  • They uncover bugs locally, on developer machines: contract tests can and should run on developer machines prior to pushing code.

From a business point of view, it is well known that the later in a project lifecycle that a bug is found, the more costlier it is to fix.

From Beth's excellent talk on the topic

By running most of your automation tests as fast unit or local integration tests, you can keep build speeds to a minimum, and prevent queues building up in teams.

It turns out that contract tests also have other secondary, yet positive side-effects:

Other benefits

  • The ability to develop the consumer (eg. a React Web App) before the API
  • The ability to drive out the requirements for your provider first, meaning you implement exactly and only what you need in the provider.
  • You get a set of well documented use cases ("Given ... a request for ... will return ...") that show exactly how a provider is being used.
  • The ability to see exactly which fields each consumer is interested in, allowing unused fields to be removed, and new fields to be added in the provider API without impacting a consumer.
  • The ability to immediately see which consumers will be broken if a change is made to the provider API.

I hope this brief introduction to the topic showed you how you can use contract testing to replace many (or in some cases, all) of your end-to-end integrated tests to speed up your CI pipeline and increase your teams' velocity.

Contract testing is not a silver bullet and no two teams are the same: so think about your test strategy, how it applies to you and what you need to get from it.

Example contract test with Pact and Pactflow

watch: contract testing demo with NodeJS and Pact JS

Next Up

Our follow up article to this series is "Getting started with contract testing", where we help you answer key questions on navigating the challenges of make contract testing a success in your organisation.

Resources & Further Reading

arrow-up icon