The Case for Contract Testing: Cutting Through API Integration Complexity

Modern API Development is complex, to say the least. In a world driven by fast-paced, feature-focused delivery, teams often fall into the trap of relying too heavily on end-to-end (E2E) tests. These tests can provide high levels of confidence when they pass, but the vast scope of the tests can also make it difficult to diagnose failures.

This blog marks the beginning of our three-part “AI Automation” series, where we’ll explore how Generative AI is transforming contract testing, enabling developers and testers to address the challenges of complex API integrations with greater efficiency and accuracy.

Here’s what you can expect from this three-part series:

• Part One: Tackling API integration complexities with contract testing
• Part Two: How AI transforms contract testing and alleviates traditional challenges
• Part Three: Building the future of API contract testing with AI automation in PactFlow

Isolated Testing Leads to Redundancy

Tests are often spread across the full stack, with teams working in silos. When this happens, developers and testers often focus on different levels of testing, leading to overlap and duplication without improving code quality.

This issue is exacerbated by the growing focus on quality, with techniques like Test-Driven Development (TDD) and Behavior-Driven Development (BDD) becoming commonplace. While these approaches lead to more tests being written than ever before, a holistic view is integral to achieving quality at each layer of the testing pyramid.

Understanding the Testing Pyramid

To understand how contract testing can address these issues, let's consider a scenario where a "Collaborator" is a component responsible for communicating with another system. This could be tested in multiple ways:

• Component Integration Test: At the unit test level
• System Integration Test: At the integration test level
• End-to-End Test: Including UI tests and subcutaneous tests (which bypass the UI and directly access APIs)

Component Integration Test

In a component integration test, the external service is mocked. This allows the consuming team to build their application even when the external service is unavailable. These tests are run in isolation at a unit testing level, ensuring fast feedback with minimal dependencies. However, keeping the mock representation of the external service in sync with the real provider is challenging. This leads us to the next testing technique, system integration testing.

System Integration Test

A system integration test is traditionally conducted in a shared environment where both the consuming application and the external service must be developed and deployed. This coordination requires time and effort, as both services must be available in the same environment and may be influenced by external factors beyond the test’s control. If a test fails here, it could be because the mock used in the component integration test is out of date, or the provider has diverged from its initial representation.

End-to-End Test

Not all businesses view the APIs they create as the product they sell; rather, these APIs may power business-focused journeys that traverse multiple systems.  End-to-end (E2E) tests provide the highest level of confidence as they focus on user behaviors and ensure the entire system behaves as expected. However, since the system behaves as a black box, diagnosing the root cause of failures becomes extremely difficult. Designing systems with observability in mind can help with fault diagnosis, which is crucial for identifying production failures.

With APIs, it’s vital to know what is in use within your service catalog and how they operate in unison to deliver business value. Workflow specifications, such as Arazzo, can help tie business outcomes to the API journeys that deliver them. Subcutaneous testing can also be applied to concentrate on the API calls, bypassing the UI, which is often the slowest part of E2E testing.

Evolving Systems Gracefully

Contract testing is a technique that incorporates many of the advantages offered by the testing practices mentioned above. We can employ TDD techniques to write component integration tests, which create contracts between the API collaborator and the external service as a byproduct. These contracts clearly define the expectations of the consumer and can be shared with the external services the consumer depends on.

External services can use these contracts to ensure their applications honor the expectations set by the consumer. This approach helps prevent stale component integration testing mocks and offers providers valuable insights into which parts of their APIs are being consumed by whom.

To maximize the benefits and allow systems to evolve gracefully, contract generation and validation should be tied to continuous integration (CI). Consuming applications should publish new contracts, while providing applications must ensure that updated code honors existing contracts with service consumers.

This last point is critical, as it addresses a significant challenge: versioning. Versioning is both difficult and burdensome for service consumers and providers. Without a clear view of API surface area consumption, one must assume everything is in use, leading to bloated response schemas, multiple code paths to support versioned APIs, and customer confusion about what to use.

By tying contract generation to CI systems and tracking the code version that created it, we can monitor its deployment through various environments. This allows service provider developers to validate service consumer contracts across all current environments, enabling them to propose an evolution path for changes as the code is being written.

It's important to note that contract testing is a technique, and there are many ways to achieve it. Various tools offer contract testing capabilities, each with its own learning curve, which can sometimes detract from the true benefits of the approach.

The Complexities of Delivery

APIs rarely exist in isolation and have seen explosive growth due to the shift from monolithic to microservices architectures and the increasing number of protocols for message transmission.

According to industry research:

• 61% of API growth comes from microservices
• 81% of companies operate in a multi-protocol environment
• 57% use three or more protocols

APIs have become pervasive in modern life, powering everything from ride-share applications to social media platforms and government services. Yet, even mature organizations struggle with the impact of microservice sprawl.

Significant barriers still exist when implementing microservices or effective API-driven systems, such as:

1. Lack of experience or skills
2. Ever-increasing complexity of systems
3. Growing demands for faster delivery
4. Limited time due to heavy workloads

The Challenges of Contract Testing

Steep Learning Curve

Contract testing involves multiple layers and requires teamwork between API consumers and providers. It can replace existing tests, affecting ROI and requiring new skills. Since it’s a technique, not a tool, teams might need to create or adopt a solution, which could mean adjusting workflows, tests, or even the application. If done poorly, contract testing can do more harm than good.

Impact on the Delivery Lifecycle

Contract testing may require a fundamental shift in how teams develop today, which can initially impact delivery velocity. It also requires buy-in from consumer and provider teams and project stakeholders.

Scalability

As teams benefit from contract testing, they often become advocates within their organizations. However, spreading these practices consistently across the organization takes time and effort.

Creating and Maintaining Effective Tests

Creating an initial contract test can be difficult, especially as users may not know where to start. Ineffective tests can provide a false sense of confidence by either testing too little or testing the wrong parts of your code. At worst, they could bind a service provider to providing specific fields or endpoints that a consumer does not need.

It's also easy to miss key touchpoints for your code (e.g., point of change, pre- and post- deployment) for contract testing checks or to implement them incorrectly. This can cause updated consumer contracts to block service provider pipelines.

Beyond Breaking Changes: How Contract Testing Improves Software Design

Contract Testing and Methodologies

Contract testing aligns well with agile methodologies and is analogous to TDD for your APIs. TDD follows a pattern of red, green, refactor: write a failing test, write code until the test passes, and then refactor the code, knowing the test protects it. Contract testing follows this pattern by allowing a consumer to write a test specifying how they want a provider to behave in each situation. The consumer then writes code to implement that contract, ensuring they’ve issued the expected request. The mock provider returns a result according to that contract, allowing the developer to continue with unit testing assertions on their API collaborator.

Service providers can apply the same TDD concept on their side. They have a set of contracts showing the consumers’ expected behavior. The test framework should replay the requests to the provider’s code, which must satisfy each contract.

At this point, the developer can update the service provider’s code and replay the consumer contracts. Any discrepancies are identified, and the developer has the information needed to construct an upgrade path for consumers, avoiding breaking changes.

Agile & DevOps methodologies align neatly with this approach. On one hand, we advocate for small incremental changes to deliver value quickly; on the other, we want to ensure continuous integration and delivery, so we can push changes quickly to end users.

Contract testing complements this by providing fast feedback loops on incremental changes. This allows us to confidently manage and deploy these changes across environments with the knowledge from testing various combinations. These insights help keep system integration and E2E tests leaner, reducing the mean time to change in terms of delivering something to production.

Improved Development Workflows

Fast Feedback Loops on Coordinated Changes

Using a Pact Broker, teams can leverage webhooks to ensure provider verification is triggered when a consumer contract changes. The broker contains a list of all previous contracts and verification results and will pre-verify them if the contents haven’t changed.

If a change is detected, multiple webhooks are fired for each environment the provider resides in, guaranteeing that consumers get the quickest possible feedback. This instills the confidence to deploy at the point of commit, rather than waiting until after deployment to a system-integrated environment. This approach can be a significant time and cost saver when multiple teams or developers contend over a shared resource.

Can I Deploy

A queryable matrix is held in a Pact Broker to check whether contract tests have passed before deploying to a particular environment. This matrix can be set up to poll the results. When a new contract is published, and verification is pending, users can instantly deploy their applications once they are notified of the provider verification result that was automatically triggered.

Expand & Contract

Providing insights into field-based usage of an API gives developers an understanding of the impact of renaming a field, which might affect three out of six consumers.

From there, they can add an additional field and communicate to the three consumers to switch over. These consumers can publish their new contracts, which will then be verified, allowing them to deploy to production.

The developer can then run checks again and confirm that all consumers have stopped using the old field. This process allows it to be safely deprecated and omitted without requiring a version bump of the API or maintenance of two endpoint versions.

This information is invaluable in improving API comprehension and reducing field confusion for end users.

Bi-Directional Contract Testing

In traditional contract testing, there is an assumption that a service provider exists, and the consumer contracts are replayed against it, providing the highest level of confidence. This belief places a significant burden on providers to verify these contracts outside of the existing functional tests in the codebase. In some cases, the provider application is inherently untestable – perhaps a legacy application – or the codebase may not be accessible.

An alternative mechanism is bi-directional contract testing, where the consumer contract is statically compared against an API description document. This approach significantly simplifies the provider verification workflow and expands use cases for providers over whom you have no control, such as third-party APIs.

While this method offers fewer guarantees (since the static document may not fully align with the codebase it represents – a phenomenon known as provider drift), this risk can be mitigated to varying degrees depending on the approach taken.

A provider contract, as we see it, consists of two parts: the API description and the provider verification result (and associated documentation). This verification is primarily linked to the exit code of the provider’s test suite, but there is flexibility.

One major advantage of the bi-directional comparison provided by PactFlow is the ability for a provider team to transition from pr1esenting a provider contract to performing a traditional provider verification against their running application. This has proven particularly beneficial for organizations new to contract testing, especially those that have not yet convinced provider teams to fully integrate contract testing into their workflows.

Looking Ahead: The AI Advantage in Contract Testing

Contract testing is a powerful approach to managing the complexities of API integration, significantly reducing the risk of breaking changes and dramatically improving software quality. However, as we’ve explored, contract testing comes with its own challenges, particularly in scaling and maintaining effective tests. In the next installment of our series, we’ll explore how integrating AI into contract testing can overcome these hurdles, enhancing efficiency, accuracy, and overall software quality.