Reading:
Contract Testing Vs Integration Testing

Contract Testing Vs Integration Testing

Alicia Foreman

One of the most critical factors affecting the quality of your software and the experience it provides your end customers is testing. Throughout the software development lifecycle (SDLC), development teams employ numerous techniques to ensure quality outcomes.

SmartBear offers testing solutions for every phase of the SDLC and in this blog I will focus in on contract testing - which can be orchestrated with Pactflow and Pact the open source Pact framework - and how this compares to integration testing as well as unit and end-to-end (E2E) testing.

Contract testing is a technique coined well over a decade ago by Martin Fowler that has grown in popularity with the rise of APIs and microservices architectures and associated complexities, emerging as a simple and effective control measure to calm the chaos caused by the sprawl, reduce the reliance on more E2E integrated testing and prevent bugs in production.

What is contract testing and where does it fit into my test strategy?

Contract testing is a technique coined well over a decade ago that has grown in popularity with the rise of APIs and microservices architectures and associated complexities, emerging as a simple and effective control measure to calm the chaos caused by the sprawl, reduce the reliance on more E2E integrated testing and prevent bugs in production.

The goal of contract testing is aligned with the “shift left” mindset. You want fast and reliable feedback on an independent API integration earlier in the SLDC. This makes it easier to test and debug, which reduces reliance on more complex testing later in the cycle.

Contract testing allows teams to move faster. It removes low-value bugs earlier in the SDLC so they don’t cause holdups later. The way it works is to borrow elements of both unit and integration testing. This provides valuable validation of an API integration.

Note the test pyramid. It’s a heuristic, first proposed by Mike Cohn, which helps software development and testing teams guide their allocation of time and effort on various types of testing.

Figure 1: An unbalanced test pyramid, without contract testing 

As you move up the pyramid, the different tests give more confidence, but at the price of slower feedback and higher effort (costs and time).

The test pyramid is a guide to spending more time using testing techniques at the base and less at the top. In figure 2, we see it rebalanced. Here, there is more reliance on the base and mid-level test techniques. You will note that this doesn't remove the need for E2E tests, but reduces the reliance on them, making execution faster.

Figure 2: Rebalanced pyramid, including contract testing 

Explanation of API testing techniques

The following chart aims to provide more context as an aid to explain the the balance of figure 2. It covers the main API testing techniques, the scope, who completes them, the benefits/limitations, and the tools SmartBear offers to help you and your team deliver.

Unit Test

What is it?

A unit test is at the lowest level. It provides a foundational quality gate.

It's a way of testing a piece of code that can be logically isolated in a system that works as expected.

What constitutes a unit is ill-defined. It often refers to the testing of a single function, but can be broader than that.

What’s the goal of this type of test?

The goal of unit testing is to deliver relevant information about the smallest components of your software quickly and frequently, sometimes many times per day.

It's a key practice in TDD (test-driven development). It’s crucial in the ability to refactor (change) code safely, and therefore supports the long-term maintenance of a system.

Unit tests also provide important visibility to new contributors to a code base, because they double as a form of living documentation.

How does it work?

With unit testing, you can isolate and test small components of code at a time.

By limiting the scope of the exercised software to a portion of the system under test, you improve the quality at a foundational level.

Unit testing an API endpoint is normally done by mocking other components or systems, in order to isolate the part of the API you are focused on testing.

What is the scope of the test?

A unit test focuses on a small amount of code, such as a single function.

At what stage of the SDLC does this test occur?

Pre-commit, during build

Who is responsible for this type of test?

Developers

What are the benefits of this test?

Unit tests are extremely fast.

They detect issues earlier to avoid the costs of fixing later in the SDLC.

They’re an efficient means of testing, as it can be done at any time. There’s no reliance on others.

Unit tests focus on individual functionality

They understand the impact of development changes with ease

They assist with code refactoring and reduce long-term maintenance costs.

What are the limitations/challenges of this test?

Unit testing does not do any of the following:

Consider integration issues between services.

Catch complex errors that arise from multiple services.

Consider non-functional aspects, including usability and performance.

Provide guarantees for functional operation or compliance with requirements.

Additionally, unit testing of private/internal functions instead of externally visible functions can lead to a higher maintenance cost. Some schools of thought encourage more “integration test” (sociable) unit tests, which test externally visible behaviour, rather than internal implementation details.

Unit testing is known as a white box testing technique where are you are aware of the underlying structure and code.

What tools can be used for this test?

There are different unit testing frameworks, depending on the type of code you're working with.

Contract Test

What is it?

A contract test is a technique for testing software application interfaces and integrations.

Contract testing is a non-functional technique that bridges the gap between unit and integration tests. It takes the benefits of each and ensures two systems communicate.

It's non-functional, because it doesn’t consider side effects.

What’s the goal of this type of test?

The goal of contract testing is to get fast feedback on systems that communicate over APIs, removing unnecessary bugs.

Contract testing helps remove the dependency on end-to-end integration tests and reduces the need to maintain complex test environments.

How does it work?

Contract testing ensures that a pair of applications will work together correctly. It checks each application in isolation to ensure the messages it sends or receives conforms to a shared understanding. This is documented in a “contract.”

See how it works.

What is the scope of the test?

Similar to a unit test, a contract test focuses on the external collaborator. This is the API client interface on a consumer side, and the appropriate receiver on the provider side, such as a Controller.

At what stage of the SDLC does this test occur?

Pre-commit, during build

Can also occur post-commit.

Who is responsible for this type of test?

During the build: Developers/SDETs

Post-commit: QA/Testers/SDETS

Bi-Directional Contract Testing gives team members without access to the source code a way to orchestrate contract testing, repurposing existing mocks from other commonly used tools.

What are the benefits of this test?

Contract tests are fast.

They focus on testing a single integration at a time without the need to deploy

With contract tests, there’s no need for dedicated test environments

They provide fast, reliable feedback, that is easier to debug..

Tests scale linearly with the number of integrations, instead of exponentially.

They allow you to deploy services independently.

There’s reduced communication and coordination.

The provider knows who the consumers are and how the consumers use.

Read: "Proving E2E Tests are a Scam"

What are the limitations/challenges of this test?

A contract test is completed in isolation, between only one API consumer and one provider. It doesn't check for side effects, or the behavior of the overall system.

For example, imagine an endpoint for a collection of /orders that accepts a POST request to create a new order. A contract test ensures the consumer and provider have a shared and accurate understanding of the request and response required to create an order. A functional test for the provider would ensure that when a given request was made, an Order with the correct attributes persisted in the datastore. Contract testing is known as a white box testing technique where are you are aware of the underlying structure and code.

What tools can be used for this test?

Pactflow is the complete contract testing solution. It allows teams to orchestrate contract testing at scale.

With Pactflow, teams get fast and reliable feedback on the reliability of a single integration between an API provider and consumer earlier in the software development lifecycle. This reduces the reliance on complex E2E tests.

Create a free Pactflow account to get started

Integration Test

What is it?

Integration tests look for information on how two or more systems work together.

They’re a form of functional testing that ensures all features continue to work as expected after a change has been introduced.

Integration testing is the most often used form of API testing, as APIs are at the centre of most integrations between internal or third-party services.

Integration testing catches a class of issues that unit tests can’t pick up – bugs that lurk between components.

What’s the goal of this type of test?

The goal of integration testing is to consider the bigger picture and prevent regressions at a higher level; how the features work together to make a product, and how that product will work with other software.

Unlike unit tests, which cover how an individual component works in isolation, integration tests must consider multiple components working together.

How does it work?

An integration test usually consists of many requests or steps that simulate real user behavior. At each step, assertions check that the service works as expected and returns the required response.

Note that “integration testing” is a vague term, encompassing anything from sociable unit tests that span multiple components or functions in a single code base, to real-life systems interacting in a deployed environment.

What is the scope of the test?

Integration tests look at how two components work together independently of the system as a whole, to accomplish a specific task or deliver a specific outcome.

An integration test can span multiple functions, classes, components, or even systems.

At what stage of the SDLC does this test occur?

Pre-commit, during build, after build in deployment environment

Who is responsible for this type of test?

Devs and QA/Testers

What are the benefits of this test?

Integration tests ensure two or more modules work together and can deliver the desired outcome aligned to requirements

They lay the foundations for the next stage of testing.

What are the limitations/challenges of this test?

Integration tests can be difficult to perform compared to unit and contract testing approaches, which are more black box style.

It's time-consuming and resource intensive to test all connected modules of a system.

Additional effort is required to create stubs and drivers, and this needs to be done accurately to avoid ineffective testing.

What tools can be used for this test?

ReadyAPI allows teams to create, manage, and execute automated functional, security, and performance tests in one centralized interface. This accelerates API quality for Agile and DevOps software teams. Get started by importing API definitions like OpenAPI/Swagger or AsyncAPI, testing and recording live API traffic, or virtualizing web services to remove pipeline dependencies.

Learn more about ReadyAPI

End-to-end test

What is it?

End-to-end tests are a functional testing methodology. They test the entire functionality and performance of an application under production-like circumstances and data to replicate live settings.

End-to-end tests are valuable, but are the most complicated tests to set up and maintain.

There are two main types of E2E tests:

E2E API (backend)

E2E UI (frontend) / UI)

E2E API tests – also called E2E tests – consider the integrity of the entire system.

E2E UI testing extends end-to-end testing, with the addition of testing behavior through a user interface such as a mobile application or web interface.

 

What’s the goal of this type of test?

The goal of end-to-end testing is to ensure the entire software application is production-ready and avoid risks post-release.

UI testing ensures the end user’s experience is delivered as expected

How does it work?

An API E2E testing process simulates a real user scenario.

This not only validates the system under test, but ensures its sub-systems work and behave as expected.

 

End-to-end tests extend integration testing to test multiple applications at once, usually deployed to a test environment.

As with the API E2E test, a E2E UI test specifically considers the frontend user experience.

What is the scope of the test?

The entire system (excluding the UI).

A UI test considers all components that make up the end user’s experience.

At what stage of the SDLC does this test occur?

Post-commit, after deployment into a test environment

Who is responsible for this type of test?

Testers/QA/ Automation Engineers

All forms of E2E testing are normally conducted by testers to ensure software is production ready. This avoids risks post-release.

What are the benefits of this test?

The main benefit of E2E testing is ensuring that the whole system and subsystems work as intended. You can catch potential user experience issues from the connected systems.

This confirms your application is functional at every level – from the front to back-end and on multiple systems – and provides perspective on its performance across different environments.

E2E testing also provides expanded test coverage by creating additional test cases that may have not previously been considered.

It detects bugs and increases application productivity.

Read more

What are the limitations/challenges of this test?

E2E tests of any form can be slow. This creates bottlenecks, especially when there are too many E2E tests.

They can also be fragile, due to fluctuating data, wrong versions of code, or wrong configurations.

They can be challenging and time consuming to debug, and require complex environment and data management.

They can make it difficult to assess coverage.

And they come with an expensive overhead – coordinating large all-in-one deployments. With E2E tests, most of the cost is in maintenance, which is why there are the least number of UI tests, with a focus on covering the most crucial workflows.

The impact of this:

  • Teams can be left idly waiting
  • Tests result in sequential flows
  • Testing may result in Late Go/No-Go decisions

What tools can be used for this test?

For UI E2E testing, TestComplete from SmartBear is a UI functional test automation platform for web, desktop, and mobile applications. You can script your tests or use the scriptless record and replay function.

Learn more about TestComplete

Using contract testing and integration testing together

In summary, as we saw in figure 2 - the rebalanced test pyramid - there is not one single step of the testing process is more important than the other. Each step plays a critical role in delivering quality software. SmartBear proudly supports all stages.

Numerous Pactflow customers praise the addition of contract testing in their test automation strategies:

Including contract testing with Pactflow in M1 Finance's test automation strategy has meant teams can simply reveal both the provider and the consumer side of the contract and move forward with confidence when they see validation.
Contract testing with Pactflow was implemented as part of an app modernisation project for a major US-based mortgage lender resulting in a 70% reduction in manual testing costs.

Create your free Pactflow account to test it out for yourself.

The pyramid is conceptual, but can help individual development teams determine the importance and weight of specific testing techniques in their ecosystem. This adds layers of quality and assurance at the most valuable places in the SDLC.

Now we've taken you through the path to quality delivery via an efficient SDLC, how do you ensure your application stays running, even when you are asleep? Well, you can take a look at BugSnag by SmartBear.

arrow-up icon