Schema-based contract testing with JSON schemas and Open API (Part 3)

by Matt Fellows

Introduction

This is the third instalment of a three part series on schema testing. In part one of the series, we discussed the differences between schema testing, specifications and contract testing, and highlighted the trade-offs of each. In part two, we reviewed the most popular approaches to schema testing. In this final post, we demonstrate how you can use Pactflow to manage a schema-based contract testing workflow and talk about how we might be able to incorporate it approaches into our tooling in the future to support a wider range of testing.

UPDATE: Mar 2022

We now support OAS and schema based contract testing in Pactflow. See our post on bi-directional contracts for more on this.

Schema-based contract testing with Pactflow

So now that we have a grasp on the key concepts and trade-offs, let's see how we can integrate Pactflow with the code-generation approach using JSON Schema as the request/response body validator - note that as long as you can capture the contract in a file, any of the above approaches are also compatible.

I would class this as a consumer-driven, schema-based contract testing methodology, as it allows validating a contract in the provider using the same workflows you would with a standard Pact test.

JSON Schema + Pactflow testing workflow

Consumer Test

This phase is analogous to the Pact consumer unit testing phase, where the consumers' client code is tested and as an output produces a contract.

In our example, we are generating a JSON schema from our statically typed language (TypeScript) using the typescript-json-schema library. The process:

  1. Author the API consumer type (e.g. products.ts) (1)
  2. Generate the contract (using the TypeScript -> schema generation) (2)
  3. Publish the contract to Pactflow and run the  can-i-deploy check. (3)
  4. If successful, deploy to production and tag the application version as moved to prod

Provider Test

This phase is analogous to the provider verification step. It uses json-schema-diff to do a semantic comparison of contract submitted by the consumer against the JSON schema to check for backwards incompatible contract changes.4. Fetch the consumer contract

  1. Download the schema from Pactflow (4)
  2. Generate a contract from the provider's code base (5)
  3. Perform the diff using json-schema-diff. (6)
  4. Also compare against the current "golden" (published) schema (this is particularly useful for public APIs and ensure documentation doesn't drift) Download the schema from Pactflow (7)
  5. Send the verification results (pass/fail) back to Pactflow
  6. Run the can-i-deploy check
  7. If successful, "deploy" to production and tag the application version as moved to prod

Benefits

Using Pactflow this way resolves the evolution (8) and sharing (9) problems, whilst preserving many of the powerful workflow features evolved through the Pact ecosystem over the past 7 years.

Code

You can give the project a go to convince yourself whether or not it is a good idea. You will need a running Pact Broker or a free pactflow.io account to test it for yourself.

https://github.com/pactflow/pactflow-jsonschema-example

Swagger/OpenAPI Specification and Pactflow

As alluded to earlier, it is actually possible to combine the above strategies. In fact, Atlassian have created several useful tools to allow teams to pick and choose what works best for them. One example is their Swagger Mock Validator, which allows teams to write consumer Pact tests - getting all of usual benefits Pact provides - whilst allowing Providers who test against a Swagger/OAS spec to check whether or not they are compatible with consumers.

Pact + Swagger workflow

Consumer

It's the usual Pact story: the consumer authors a series of unit tests against their API client and publishes the resultant contract to the broker.

Provider

The process is similar to above, where the provider verification step simply performs a static schema comparison between the Pact file and the OAS specification held by the provider. If the pact is a subset of the OAS spec, the test will pass.

This alleviates several of the challenges mentioned above and really helps to decouple teams, whilst providing many of the guarantees. It does hinge on the ability to ensure your API does implement the OAS.

I've used this strategy in the past, as well as simply added it as an overlay to Pact tests to ensure any published swagger file is indeed consistent with the implementation.

You can check out the project here: https://bitbucket.org/atlassian/swagger-mock-validator/

Conclusion

All of the options have their tradeoffs, and you need to choose what makes sense for you. The "Pact workflow" can be more involved, but the schema-based tests provide weaker guarantees (and perhaps also will suffer at a certain scale anyway).

Regardless of the approach, however, there is always a need for some central repository to store and facilitate the evolution of the contract - in whichever form that takes. Pactflow is ideally suited to play this role.

I think an exciting possibility is to combine these two approaches: enhancing the code-based contract tests with schemas. By introducing tools for evolution (Pactflow) and capturing the supported conversations (specification by example) we can solve most of the issues described above, reduce duplication and coupling between teams, whilst preserving the benefits of the Pact approach. It also introduces the possibility of a more seamless specification-first approach to API design. It also allows for both: consumer or provider initiated 🤯.

This makes it a "choose your own adventure" to the testing approach, whilst enabling a consistent continuous delivery workflow.

In fact, we have just added support for what we are referring to as "bi-directional contracts" (enabling a provider-driven approach or consumer-driven approach to contract testing), which incorporates these ideas and allowing even more flexibility in the approach.

If you'd like to have early access to this exciting new feature please join our developer program.

Perhaps this is like the attempts to reconcile general and special relativity, and will result in a more complicated mess for everyone, but this author thinks it may just be the next major evolution in contract-testing (pun intended).