It's helpful first to consider where contract testing sits within the context of a broader automation testing approach. When coming up with a test automation strategy, a good rule of thumb in how you should expend your effort is the approach advocated in Mike Cohn's "Test Pyramid":
Stick to the pyramid shape to come up with a healthy, fast and maintainable test suite: Write lots of small and fast unit tests. Write some more coarse-grained tests and very few high-level tests that test your application from end to end [1]
Contract tests fit in the "Service Tests" layer, as they execute quickly and don't need to integrate to external systems to run. Their job is to give you confidence that the systems you integrate with are compatible with your code before you release.
NOTE: The "UI tests" at the top of the pyramid here are often also referred to interchangeably with "integrated end-to-end (e2e) tests".
The problem with e2e integrated tests
Whilst tests at the top of the pyramid more closely represent what the customer would experience, they have several painful drawbacks. They:
- are slow; because they traverse multiple systems and generally must be run serially, each test may take several seconds to several minutes to complete, especially if pre-requisite setup (such as data preparation) must be performed.
- are hard to maintain; end-to-end tests require all systems to be in the correct state before they are run, including the correct version and data.
- can be unreliable or flakey: because of the complexity in orchestrating a test environment, they can often fail causing false-positives, becoming distractions to the team. In many cases, they fail due to a configuration issue unrelated to any code change.
- are hard to fix: when an end-to-end test fails, debugging the issue is usually difficult, because of the distributed and remote nature of the problem.
- scale badly; as more teams' code gets tested, things get more entangled, test suites run exponentially slower and releases get clogged in automation pipelines.
- find bugs too late in the process: because of the complexity of running such testing suites, in many situations these tests are only run on CI after code has been committed - in many cases, by a separate testing team days afterwards. This delay in feedback is extremely costly to modern, agile delivery teams.
Because of these properties, it is advised to keep them to a minimum number e.g. to ensure key business transactions or features are covered.
The value of contract tests
Contract tests generally have the opposite properties to e2e integrated tests:
- They run fast, because they don't need to talk to multiple systems.
- They are are easier to maintain: you don't need to understand the entire ecosystem to write your tests.
- They are easy to debug and fix, because the problem is only ever in the component your testing - so you generally get a line number or a specific API endpoint that is failing.
- They are repeatable:
- They scale: because each component can be independently tested, build pipelines don't increase linearly / exponentially in time
- They uncover bugs locally, on developer machines: contract tests can and should run on developer machines prior to pushing code.
From a business point of view, it is well known that the later in a project lifecycle that a bug is found, the more costlier it is to fix.
By running most of your automation tests as fast unit or local integration tests, you can keep build speeds to a minimum, and prevent queues building up in teams.
It turns out that consumer driven contract tests also have other secondary, yet positive side-effects:
Other benefits
- The ability to develop the consumer (eg. a React Web App) before the API
- The ability to drive out the requirements for your provider first, meaning you implement exactly and only what you need in the provider.
- You get a set of well documented use cases ("Given ... a request for ... will return ...") that show exactly how a provider is being used.
- The ability to see exactly which fields each consumer is interested in, allowing unused fields to be removed, and new fields to be added in the provider API without impacting a consumer.
- The ability to immediately see which consumers will be broken if a change is made to the provider API.
Resources & Further Reading
Next: What is the difference between consumer driven contract testing and Bi-Directional Contract Testing?
