Schemas are not contracts

It is common to hear the phrase "we do contract first development", but what they really mean is that they define a schema for (or a specification of) their API. Schemas are useful, but they are not contracts, and in this short article I will try to explain the problem with schemas and why contract-testing can be used to address these challenges.

Prefer a video? Watch Matt present a wider version of this topic at YOW! Perth in 2022.

For a longer investigation on the topic, check out the schema-based contract testing series. If you want to know how you can make schemas work with contract testing, check out our bi-directional contracts feature (new live) or read the blog.

Schemas may take many shapes - protobufs, GraphQL, Avro etc., but for illustration purposes we'll use arguably the most common in API development: Open API Specification (OAS, previously Swagger).

These tools have many benefits, such as the ability to generate code stubs, and are well integrated into the ecosystem. The problem is, they can't necessarily be trusted as a replacement for contract testing...

Schemas and Contracts

Using the following definition as a guide, the differences between a schema and contract test can be summarised as follows:

1. Schema - using a generalised, often declarative notation, defines the data types and set of inputs/outputs that a single system supports (such as an OAS) at a point in time. In HTTP for example, these are the syntactical rules that requests and responses must follow.

Here is how JSON Schema defines a schema:

"You may have noticed that the JSON Schema itself is written in JSON. It is data itself, not a computer program. It’s just a declarative format for “describing the structure of other data”. This is both its strength and its weakness (which it shares with other similar schema languages). It is easy to concisely describe the surface structure of data, and automate validating data against it. However, since a JSON Schema can’t contain arbitrary code, there are certain constraints on the relationships between data elements that can’t be expressed. Any “validation tool” for a sufficiently complex data format, therefore, will likely have two phases of validation: one at the schema (or structural) level, and one at the semantic level. The latter check will likely need to be implemented using a more general-purpose programming language."

This second check is where other forms of testing can come into play, including contract testing. In fact, as we'll see below, schemas are not necessarily even sufficient to perform the first kind of check.

2. Contract - defines how two systems are able to communicate by agreeing on what interactions (conversations) can be sent between them and providing concrete examples to test the agreed behaviour. Contract testing goes beyond defining a schema, requiring both parties to come to a consensus on the allowed set of interactions allowing evolution over time.

Or put another way:

Schemas are abstract, contracts are concrete.

Matt Fellows

Problems using schemas for testing

To start to get a feel for the problem, Andras has neatly summed up the challenge:

With that context, let's drill into a number of common problems that contract tests are able to cover that schemas alone will not:

1. Not all schemas capture key aspects of a contract. For example, there is no standard way to define the HTTP-level semantics - such as the verb, path, status code or headers - in a JSON schema. This may create a maintenance burden to ensure they are kept in sync, or create a gap in coverage if left untested.
2. Schemas are abstract, and introduce ambiguity which can lead to misinterpretations. For instance, in an OAS you may define that an API can return a 400, a 403 or a 200, but you cannot say for certain which specific set of inputs will result in those status codes by looking at the specification alone. Similarly, certain requests may illicit variations in a response (think of the anyOf or oneOf logical operators in OAS). Schemas are deliberatively agnostic to such things. To drive home the point, schemas will often mark some fields as required and not others - this makes it unclear which fields a consumer actually can reliably expect to be present when used and in what situations.
3. Test Coverage - following from (2), it's easy to check if a system is compatible with a schema, but it's very difficult to be sure it fully implements the spec. For example, despite OAS being the most widely used API documentation tool, we know of no such tool that can currently do this. The aphorism that an API is "not incompatible with the spec" sums this up rather elegantly.
4. Variable guarantees - Depending on how you generate and validate your schemas, the level of guarantees you get will vary considerably. For example, if the provider only validates a subset of the schema and a consumer is compatible with a non-overlapping subset of the same schema, you will have a false sense of security.
5. Code <-> schema drift - poor projection of the contract may lead to bad results. For example, if using code generation tools and a field that isn't appropriately tagged for export is ignored, it won't appear in any generated artefact and show up as a red flag.
6. API surface area visibility / waste - if consumers generate code/types from schemas, they will naturally consume all of the available API surface area (requests, fields etc.), even if the application doesn't actually need all of them. This makes evolution more difficult, because you can't track the specific needs of each consumer: you must assume they use the entire set of features. With contract tests, it's not a breaking change if you remove an endpoint that no client uses, or remove a field that no client expects.
7. Evolution - if you want the broader guarantees and benefits of a "contract" as defined above, you need to find a reliable mechanism to introduce the evolution, collaboration and conversations (e.g. HTTP-level semantics) into the process.
8. Sharing - you need to find a way to reliably share, version and collaborate on the schema. Many teams adopt the rule of "no breaking changes" once a version is stable, to avoid this complexity.
9. Loose data types - for example in protobufs, there are only a few primitive data types that may be encoded in the protobuf definition. But in reality, fields will have stricter encodings. For example, knowing the uuid field requires an actual uuid rather than any string is useful from an integration contract perspective. The guarantees you get here will often vary depending on how specific the schema definition is for your use case (as a rule of thumb, the more generalised the schema, the less guarantees you'll get).

In conclusion

So how can you get the maximum value out of your API specification whilst simultaneously reaping the benefits of contract testing?

Tim Jones, one of the Pact maintainers, sums up what many teams already do:

A spec is not a contract. So, most people who are using OAS but not Pact also pair their OAS with eg a Postman collection of request/response examples. This pairing of spec + examples is a contract.

Tim Jones

Pairing your specification with a set of API tests gives you a crude contract. You still  need a mechanism to share and version these tests (8) and will lose out on the evolution (7) and surface area (6) benefits without your own custom tooling. The problem with this, is that you need these tests written from the perspective of the Consumer and they must be tied to your Consumer's actual behaviour (to prevent drift), not the Provider's, otherwise you're simply marking your own exam.

If you can solve that problem, you're well on your way to reaping the full benefits of both contract testing and API schemas.

Give schema contract testing with Pactflow a try

If you want to know how you can make schemas work with contract testing and don't want to build your own custom tooling, check out our Bi-Directional Contract Testing feature or read the blog.