Pact is dead, long live Pact

In 1272, when Henry III died, his son Edward I was off fighting in the Crusades. To avoid any chance of a civil war erupting over the order of succession, the Royal Council proclaimed: "The throne shall never be empty; the country shall never be without a monarch" and established an immediate and automated succession process obviating any need for decision-making, based on the laws of primogeniture. But it was the French that first coined the phrase "Le roi est mort, vive le roi!" ("The King is dead, long live the King!"), using this precedence, declaring accession to the French throne of Charles VII after the death of his father Charles VI in 1422 (source: Wikipedia).

Since 2013 Pact has reigned supreme over the realm of contract-testing, but can it hold the throne against a confluence of new forces, and if not, who will rise up and claim it?

Hail to the King

Integration testing is hard, and it gets harder as the number of systems, teams and people get involved. At least, it does with end-to-end integrated testing.

Contract testing is a mathematically viable approach to scaling integration testing with the size of the system (see this great post by Elliot Murry from Infinityworks that breaks it down for you, and the original inspiration from J.B. Rainsberger).

It was Pact that combined the ideas from J.B. Rainsberger and Ian Robinson's consumer driven contracts pattern and made it into an easy to use framework. It was the first of its kind to do so, and it has been reducing the cost of integration testing for thousands of companies worldwide since 2013.

Challenges with consumer-driven contract testing and Pact (heresy)

But Pact is not without its flaws. As the first of its kind, it's not surprising there are areas that could be improved with lessons and feedback over the past ~7 years.

Our mission is to enable organisations to get more value from Pact and contract testing quickly and at scale.

☝️ If the success of Pactflow hinges on the success of Pact, it's crucial we address them.

Leaning into these challenges is uncomfortable, but important if we are to stay relevant.

Technical challenges:

1. It can take time to get value from Pact

There is no free lunch - Pact requires users to write tests 😱. Whilst this is stating the bleeding obvious, it does mean that the time-to-value for Pact is not always immediate - there is an incremental value to each integration that's tested and investment must be made. This is usually OK for new systems, because tests are added along with new functionality. But it is especially true when retrofitting Pact onto existing systems - you can't rely on Pact to gate a release until you have all of the interactions covered.

2. The consumer-driven approach does not scale to large numbers of consumers

Whilst Pact significantly improves coupling with respect to integrated tests, it does have a level of coupling between consumers and providers, primarily around provider states (data) and builds (CI). This coupling can get difficult at a certain ratio of consumers to providers.

3. BFF/API Gateways (e.g. GraphQL, AWS API Gateway, Kong)

These common architectural components are often pass-thru or transformation layers, with little business logic or persistence, but are required to partake in contract testing nonetheless. Whilst they can be tested with Pact, they tend to be rather cumbersome to do.

4. UI testing with Pact can lead to pain

Basically, you don't want to do this with Pact. We've written about this a lot.

Non-technical challenges:

1. It can be hard to get provider teams on board with Pact

There are myriad reasons for this, and it's so common we created a "convince me" page years ago to help combat the bad ones. But there are some good(er) ones also:

• Providers may already have good testing processes (e.g. tools and processes to ensure compliance with an OAS)
• They feel as though they are “doubling up” test effort
• Implementing provider states can be challenging
• Legacy systems that are rigid to change and therefore test in the manner Pact requires

2. Paradigm shift / steep learning curve 📈

The “pact” consumer-driven model can take time to comprehend and implement. This is true of most changes in approaches (TDD, Agile, DevOps etc.) but it is a fact, and accelerators and guardrails to keep teams on track could help address this.

3. Non-developers may be excluded from contributing

Lastly, Pact is a code-first tool that requires access to the underlying code base of the system under test. This means it is a true "shift left" tool for integration testing - you can't get any more left than when the feature is being written in the first place!

But not all teams work this way, and not all code bases are conducive to this without rearchitecting. There are good reasons to allow roles such as QA's, Automation Testers and SDETs to get involved in the contract testing process.

Disrupting Pact (a revolution is afoot)

To achieve our objectives of "do it faster" and "do it at scale", we need to force some significant changes in our ecosystem.

We often talk about "disrupting Pact", but what might a version 2.0 look like? Where does a commercial offering fit into this picture, and how do we ensure the longevity of Pact and our Open Source community? These questions have been at the core of Pactflow from the beginning.

Before we talk about the how, let's visit some of the criteria that must be met to achieve our vision:

Objectives

1. Expand the breadth of technology we can support (e.g. OAS, GraphQL, Protobufs, API Gateways etc.)
2. Reduce the time-to-value of contract testing
3. Simplify the adoption and scaling of contract testing
4. Expand the types of ways a contract may be generated or verified, and allow the use of BYO tools (e.g. record/replay or instrumentation approaches to creating contracts, or tools such Cypress, MSW, Mountebank, Wiremock, Hoverfly and other such mocking tools)
5. Expand the roles that can contribute to contract testing (such as Testers)

We are now at a crossroads where the key ingredients required to disrupt the status quo have emerged. These are plugins and bi-directional contracts, built upon the foundation of the universal contract specification at the heart of Pact.

Plugins

The first item on the agenda is to vastly expand the breadth and depth of technology supported by Pact contract-testing. If we are to solve the problem of integration testing in the enterprise, then we must cover as much of the technology footprint as we can.

Technology moves quickly, so what is today the standard (e.g. REST/HTTP) is tomorrow's legacy technology (hello SOAP/XML, CORBA, COBOL, ...). We must enable the rapid implementation and rollout of support for emerging standards.

Technology can also be specific to organisations. It's common in many industries such as banking, telecommunications and engineering to have proprietary protocols and systems to address local needs. It would be ideal if customers could create closed source tooling so that contract testing may also be applied in these situations.

Our approach to dealing with this, is to extend Pact with plugins, and make it available to all as Open Source.

Bi-directional contracts

Plugins address the need for rapid technological advancement, and expanding our ecosystem and community of contributors. But plugins are still rooted in the Pact consumer-driven methodology and its unique - and unabashed - approach to contract testing. This is not a bad thing, but it only solves one objective (#1) from above.

Bi-directional contracts is our approach to dealing with the gaps (objectives #2-5).

What is bi-directional contracts?

Bi-directional contracts, in essence, is the ability for both sides of an integration point to publish their own view of the integration - a consumer contract capturing the consumer needs and a provider contract publishing what the provider can do - and the evidence to prove it.

We then provide a mechanism within Pactflow to ensure both views of the contract are compatible with each other when using the can-i-deploy tool, a process we call cross contract verification. We do this whilst preserving all of the powerful workflows currently supported in the process.

Importantly, it doesn't imply the use of Pact to generate or verify the contract.

NOTE: for our first version of this feature, we only support Pact based consumer contracts  and Open API Spec (OAS/Swagger) provider contracts, but we plan to expand the options on both sides over time.

To illustrate this flexibility, you might already be heavily invested in tools like Cypress for your front end testing and leveraging its handy route stubbing feature - this is a candidate to convert into a pact contract. Or you might already have a functional testing suite for your provider, using tools like Postman or Dredd. If you have confidence you can remain compatible with the OAS file then Pactflow can essentially convert these tools into a powerful contract-testing workflow.

Pactflow can take your existing tools and "upgrade" them into a powerful contract-testing workflow.

How it works

Following from above, the picture looks like this:

1. The consumer is tested against a mocking tool of your choosing (such as Pact or Cypress).
2. The test produces a pact file and is uploaded to Pactflow. Where Pact is not used as the mock, you will need to convert your mocks into a pact file (see the example below for how to do this. NOTE: we plan to produce adapters to capture contracts from common tools in due course).
3. The Provider starts with their contract (such as an OAS) and writes tests using a tool of their choosing to ensure it is compliant.
4. The Provider verifies this contract using this tool, and publishes the contract and the verification results to Pactflow (NOTE: we also plan on producing a tool for this purpose).
5. During the CI/CD pipeline for either build, when can-i-deploy is run, the cross contract validation step is performed within Pactflow, which checks to see if the consumer contract is a compatible subset of the provider contract.

Once a consumer has produced their consumer contract, their CI/CD pipeline might look something like this - note how the pipeline looks the same as it does in the standard Pact flow (except for the fact that the can-i-deploy step performs an additional server side check at Pactflow):

The provider pipeline is slightly different, because it now publishes its own contract, rather than verifies one or more Pact contracts:

Benefits

Aside from the immediate value of integrating OAS into our workflow, this approach provides compelling evidence for addressing the remaining objectives described above:

1. Contract initiation can occur from either side of the contract (consumer first or provider first). This model maps more naturally to the way many enterprises design and build their systems. It also reduces the friction in retrofitting onto pre-existing systems.
2. New consumer changes (e.g. a new response field expectation, or a new endpoint that was previously unused by that consumer) that are compatible with the provider, are immediately deployable. There is no complicated build orchestration required to sequence the change.
3. Because we now compare contracts, rather than replay them, provider teams no longer need to worry about Provider States or having too many examples to test against with specific data requirements.
4. Consumers can now violate our previous guidance on creating contracts from UI or generated tests, and can be allayed of concerns about creating an explosion of examples, for the same reasons.
5. The result of (3) and (4) is that consumer and provider teams are fully decoupled, and the flow is much simpler to implement (except of course for the pesky runtime dependency!).
6. It allows teams to BYO tools and extract more value from existing investments - if you already have tools and processes that are capable of producing a contract (a process we are calling contract capture) or verifying a contract (contract verification) you can combine them with Pactflow to get a "contract testing upgrade", for immediate value.
7. Following from (4), this means there are myriad more ways to capture a contract and verify one, including making it easier for roles such as Testers to get involved from outside of the codebase, using tools they are comfortable with.

Example Project

If by this point, you are convinced and want to have a try. We have put together an example solution to show it all in action.

In our example consumer project we show the use of Pact as the mock, but also how to BYO tool using nock's record and replay feature. We just rely on converting the nock fixtures into a Pact file compliant with the specification (about 25 lines of code), and Pactflow functionality for the usual CI/CD workflows.

In our example provider project, we use Dredd to ensure the provider is compatible with the OAS.

Why start with OpenAPI Specification?

You might at this point be wondering why we chose to start with Open API Specification (OAS/Swagger)? Aside from being a popular question on our forums, there are a number of reasons why it's a good place to start, and a number of problems with OAS that we'd like to address.

Reason #1

OAS has won the battle of the API documentation tools, has greatest market penetration and there is a lot of mature tooling. So we believe we can solve for more customers, and also get more feedback on our approach quickly.

Reason #2

In our experience, OAS hasn’t resulted in less production issues regarding breaking changes. An OAS document doesn’t guarantee that there won’t be integration bugs because OAS is a documentation tool, not a testing tool.

Reason #3

The approach and tooling we’re using has been shown to be effective at Atlassian

Problem #1

No tool can tell you your API implements the spec, they only tell you what you’re doing is not incompatible with the spec.

Problem #2

Versioning and collaborating on the same document can be a challenge - often teams will have different copies of the same document leading to issues. Hands up if you have ever received a version of your API “source of truth” via an email or a Jira ticket? That's what I thought.

A shift to democracy

At this point, it should be clear that Pact is here to stay, and that we plan significant investments for it. But Pact alone won't realise the entirety of our vision.

We plan to transition contract-testing into something more akin to a democracy, one where multiple tools and techniques can work together to form a coherent and vibrant ecosystem all with the common goal of solving the challenges of integration testing.

One with Pact at the centre of this movement, but rather than be a winner-take-all dictator, it takes on a role more like a Prime Minister (or President), working together with group of elected advisors to support a diverse set of needs.

Further, we see Pactflow as coordinating and providing oversight, much like a government, ensuring policies (such as standards) and infrastructure (such as tools) are available for the community to thrive.

If you've read this far, and the painful analogy with Kings, feudalism, revolution and democracy haven't sent you running for the hills, then you should definitely join our Developer Preview Program and stay up to date with new features like this before it's made public - join me, will you?