Integration testing is hard. It remains one of the mostly costly aspects of development today, and it's hard to see that changing any time soon. Whilst tools like Pact have popularised the concept of contract testing, there are other approaches that can be employed to reduce the cost and complexity of this task.
In this 3 part series, I'll discuss the differences between schema testing, specifications and contract testing, and get into the trade-offs of each. We'll then look at most popular approaches to schema testing in part 2. We'll cap off the series in part 3 by demonstrating how you can use PactFlow to manage a schema-based contract testing workflow and talk about how we are looking to incorporate it into our tooling in the future to support a wider range of testing.
Introduction
We often talk about contract testing as an alternative to end-to-end integrated tests and generally refer to a specific type of contract testing that uses code-based automated tests to generate a contract from a unit-test of your consumer code, and verify against a running provider API. Pact and Spring Cloud Contract (SCC) are examples of tools that do it this way.
Recently, there has been a lot of interest in using schemas - such as a JSON schema - as a basis for contract testing. Interoperability with other tools (such as OAS v3) and a healthy ecosystem of tooling has drastically improved in recent times, making it a viable alternative.
What if PactFlow could provide the same collaboration tools and continuous delivery experience for schema-based contract testing?
PactFlow as a generalised contract-testing continuous delivery solution?
PactFlow is based on the open source Pact Broker, which is "an application for sharing of consumer driven contracts and verification results". It is deeply integrated into the Pact ecosystem but, perhaps surprisingly, cares very little for the contents of what contract it is given. It will actually accept any valid JSON document (but we'd like it to support others). So if you have a JSON schema or are willing to encode your contract into a JSON document, for example as a base64 encoded property, you could use all of the CI/CD tools that Pactflow has to offer - including the powerful can-i-deploy feature.
But before we get into how we might use PactFlow with schema tests, we need to talk about what they are and how they are different to contract tests.
PactFlow will accept any valid JSON document as a contract
Terminology
There are now multiple definitions of contract testing that can be confusing, so I'll start by defining them within the context of use cases around HTTP APIs and message based systems. Note that these terms aren't mutually exclusive; you can combine the type of check with your preferred method and strategy (more on this later).
Specification vs Schema vs Contract
We must start by making a distinction between a specification, a schema and a contract and how they relate to testing, because this is an area most people have difficultly disambiguating. For the purposes of this article, a specification and a schema are the same thing - both being a formal description of an API, in whole or part thereof.
For a much more detailed read on both the origins of consumer-driven contract testing and the differences between specifications (schemas) and contracts, see the canonical reference.
Let's extract some of wisdom in that post and summarise.
The first thing to note is that document schemas are only a portion of what a service provider has to offer consumers to enable them to exploit its functionality. We call the sum total of these externalized exploitation points the provider contract.
This is stating that a specification can be seen as a narrow view of a provider, and other components such as the "conversations" between consumers and providers as well as the consumer behaviour make up the full contract.
The second important consideration is the evolution component:
By implementing contract tests, the provider gains a better understanding of how it can evolve the structure of the messages it produces without breaking existing functionality in the service community
This is stating that a specification is a point in time view, and that contracts provide a way of making explicit the evolving needs of the service community, which can be either consumer or provider led.
So, in summary:
- Schema test - asserts that a single system is compatible with a schema (such as an OAS) at a point in time.
- Contract test - asserts that two systems are able to communicate by agreeing on what interactions (conversations) can be sent between them and providing concrete examples to test the agreed behaviour (how deep these tests go may be the subject of a separate post). Contract testing goes beyond schema testing, requiring both parties to come to a consensus on the allowed set of interactions allowing evolution over time.
Methods
- Code-based contract test - a method of generating contracts and testing that contracts are valid, using code-based automation tests (i.e. the tests must execute real application code on both sides of the interaction).
- Schema-based contract test - a method of testing that checks that a consumer communicates messages that match a (subset of a) given schema, and that a provider produces output that matches this schema. This may be produced by static analysis, via code generation or otherwise.
Strategy
- Consumer driven contract testing - an approach to contract testing that has consumers drive the requirements of a service provider by communicating their needs with the provider via a contract, and have the provider conform to the superset of the needs of all of its consumers.
- Provider driven contract testing - the inverse of consumer-driven, where a provider defines the contract and communicates its capabilities to consumers, who must each validate they conform to the spec.
- Provider contract test (provider schema test) - a provider contract test may be defined as a type of schema testing that ensures an API provider is compatible with its published schema (such as an OAS). Although it doesn't quite fit the definitions above, the term "contract" is used here for consistency with other references around the web for this popular form of testing.
Benefits and Trade-offs
Let's discuss the relative strengths and weaknesses of the two methods described above.
| # | Code-based contract tests: pros |
|---|---|
| 1 | No chance of implementation drift, as real application code is executed to produce the contract (consumer). Similarly, real code is invoked to verify the contract (provider). |
| 2 | Specification by example increases comprehension - contract tests use representative examples of interactions to ensure the system works appropriately. This provides useful "how to use" documentation, as well as "how it works" documentation. |
| 3 | Specification by example removes ambiguity that abstract specifications may create. For example, what input will produce this 400, what header is required to avoid a 401 ? |
| 4 | Strong verification guarantees - will catch more bugs because real code is executed (more on this below). |
| 5 | Provides a clear, first-class process for service evolution. |
However, code-based contract testing tools like Pact/SCC do have some downsides, that schema-based tests alleviate or remove entirely:
| # | Schema-based contract tests: pros |
|---|---|
| 1 | Simpler developer experience - most engineers understand the concepts of a schema verification, whereas tools like Pact can take some time to learn. |
| 2 | Faster to get started, lower maintenance - code-based contract testing requires writing and maintaining tests for all of the interactions in your system. If you can generate a schema from code, however, you can do schema-based contract testing much more efficiently. This may also make it attractive for retrofitting onto legacy systems where the code may not be as accessible to code-based testing. |
| 3 | Faster test runs - because the test phase is just a schema diff, they execute very quickly. |
| 4 | Reduced duplication - contract tests often overlap with things that have already been defined elsewhere. |
| 5 | Removes the problem of "test data" - in Pact, we solve this using provider states. Whilst this is a lot better than the de-facto standard of seeding e2e test environments, it is a source of confusion for newcomers and can be a frustration for provider teams as the number of consumers and interactions grow. |
| 6 | Broader tooling capability - as long as the schema is generated correctly (and is intrinsically linked to how your code actually behaves), it doesn't matter how you generate the contract. For example, if you have a statically typed language, you could generate a schema based on your DTO types. You could use a recording proxy to capture the interactions, or perhaps you could convert the output of an existing test suite. |
| 7 | Alleviates the problems with contract testing using UI tools (see this cypress testing post for background). |
| 8 | Broader audience - these checks may be black-box or white-box, opening up to a wider range of test authors. |
So this paints a very nice picture for schema-based contract testing, but there are a number of downsides:
| # | Schema-based contract tests: cons |
|---|---|
| 1 | Not all schemas capture key aspects of a contract. For example, there is no standard way to define the HTTP-level semantics - such as the verb, path, status code or headers - in a JSON schema. This may create a maintenance burden to ensure they are kept in sync, or create a gap in coverage if left untested. |
| 2 | The schema will likely still need to be hand-crafted and maintained (likely by the provider team), because inferring a schema from requests is very difficult to do reliably. At the very least, it would need reviewing by a human. |
| 3 | Schemas are abstract, and introduce ambiguity which can lead to misinterpretations. For instance, in an OAS you may define that an API can return a 400, a 403 or a 200, but you cannot say for certain which specific set of inputs will result in those status codes by looking at the specification alone. Similarly, certain requests may illicut varations in a response (think of the anyOf or oneOf logical operators in OAS). Schemas are deliberatively agnostic to such things. To drive home the point, schemas will often mark some fields as required and not others - this makes it unclear which fields a consumer actually can reliably expect to be present when used and in what situations. |
| 4 | Coverage - following from (3), it's easy to check if a system is compatible with a schema, but it's very difficult to be sure it fully implements the spec. For example, despite OAS being the most widely used API documentation tool, we know of no such tool that can currently do this. The aphorism that an API is "not incompatible with the spec" sums this up rather elegantly. |
| 5 | Depending on how you generate and validate your schemas, the level of guarantees you get will vary considerably. For example, if the provider only validates a subset of the schema and a consumer is compatible with a non-overlapping subset of the same schema, you will have a false sense of security. |
| 6 | Code <-> schema drift - poor projection of the contract may lead to bad results. For example, if using code generation tools and a field that isn't appropriately tagged for export is ignored, it won't appear in any generated artifact and show up as a red flag. |
| 7 | API surface area waste - if consumers generate code/types from schemas, they will naturally consume all of the available API surface area (requests, fields etc.), even if the application doesn't actually need all of them. This makes evolution more difficult, because you can't track the specific needs of each consumer: you must assume they use the entire set of features. |
| 8 | Evolution - if you want the broader guarantees and benefits of a "contract" as defined above, you need to find a reliable mechanism to introduce the evolution, collaboration and conversations (e.g. HTTP-level semantics) into the process. |
| 9 | Sharing - you need to find a way to reliably share, version and collaborate on the schema. Many teams adopt the rule of "no breaking changes" once a version is stable, to avoid this complexity. |
TL;DR - schema-based contract tests sacrifice a level of guarantees in favour of a simpler developer experience. Code-based contract tests provide a stronger set of guarantees but comes with the extra cost of maintaining tests.
Next time
In part 2, we'll discuss the most common schema testing approaches, demonstrate how you can use PactFlow to coordinate a schema testing workflow and talk about how we might be able to incorporate these ideas into our tooling in the future to support a wider range of testing.
