We often get asked how does Pact differ from tools like Postman, Insomnia and other REST API testing tools.
In this article, we discuss the differences in approach and the pros and cons of using Pact and Postman for various API testing related scenarios.
Postman is a collaboration platform for API development, it has a fantastic IDE-like GUI that makes it easy to explore and navigate an API and a scripting language in NodeJS that you can use to automate checks against an API Provider that you can run in your CI pipelines.
Postman (and Newman) do not require direct access to the code for the System under Test (SUT), as they test it from the outside in a black-box style test. It's ideally suited to exploring and collaborating on APIs and running functional API tests.
Postman is purely a Provider-only testing tool, in that it can not test how a consumer (e.g. a Web application or another API) interacts with a Provider.
Pact is a distributed systems testing framework and importantly, it is a code-first tool for testing HTTP and message integrations using an approach known as contract tests. When combined with Pactflow (or a Pact Broker), it becomes a powerful continuous-delivery tool.
Pact's main purpose is to ensure that a consumer (such as a Web App or another API) and a provider API are compatible with each other - without ever having to test them together using an end-to-end integrated test suite.
Pact tests are written like any other unit test, and requires access to the code base of the System under Test to ensure that any expectations are based on how the consumer and provider behave, so that the tests can't drift from the actual implementation.
Here is an at-a-glance comparison of the tools:
|Use Cases||HTTP APIs||HTTP APIs, Message APIs and Web Apps|
|Usage||GUI, CLI (Newman)||Code / IDE / CLI|
|Language||NodeJS (scripting)||All major languages (11 in total), CLI tools|
|Testing Style||Black box (only tests from the "outside")||White box (needs access to code)|
|Test Author||Anyone (e.g. from another team)||Developers|
|System Coverage||API Providers||API Consumers and API Providers|
|Integration compatibility checks||No||Yes|
Pros and Cons
- Easy to get started and requires little to no coding ability
- Ability to import schemas (such as OAS, JSONSchema and GrahpQL) and use them as a basis for tests and API exploration
- Postman collections make it easy to share and collaborate with others
- Provider centricity - does not consider the consumer of the API
- Encourages a separate test suite that requires maintenance outside of the code it is testing
- Delay in feedback on API issues (due to a separate testing suite)
- Does not provide tools to orchestrate complex CI/CD pipelines
- Provides guarantees consumers and providers work together, and fast developer feedback
- Tests are always valid and maintained, because they are generated by the code with native integration in the language of the System under Test
- CLI tools explicitly designed for continuous-delivery pipelines and enable independently releasable software components
- Can be more complicated to get started
- Requires specialist coding skills
- Not suited to external testing teams who are not part of the product team or don't have access to source code
When to use
Postman is ideally suited to the following tasks and activities:
- API exploration and discovery
- Functional/E2E API tests
- External testing teams who don't have access to the source code
Pact is ideally suited to:
- Preventing breaking changes between API consumers and providers (contract tests)
- Orchestrating continuous delivery pipelines for microservices and web applications
- Developers with access to source code
Can you do contract testing with Postman?
It is possible to do a pseudo contract-test using Postman, by making assumptions about the consumer code base and how it makes API calls, storing those requests as Postman collections specific to that consumer and making the provider test against that collection as part of its CI process. But any tool that is not tied to the actual implementation (code) will not have the same value proposition that something like Pact can guarantee.
Because it does not have access to the code, the test author will have to manually setup expectations that will likely both drift from implementation over time and have broader expectations on a provider. For example, you won't be able to say which specific fields you are interested in with any degree of confidence. If the consumer code changes how it interacts with the API Provider, the Postman suite would need to be manually updated or risk devolving into a false representation of the real consumer.
Because you are unlikely to have access to the source code for this sort of test, you are unlikely to be able to stub out data and dependencies easily so they can run as isolated unit tests - so it's likely that you would need to do this against a real, deployed version of the Provider in an E2E style test.
Postman favours E2E functional tests, whereas Pact is a contract-testing tool.
Tom Hombergs has an excellent article discussing the relative strengths and weaknesses of these approaches, which can be summarised below. Ultimately, you need to choose the tradeoffs that suit your situation best, which may include a combination of the two:
Pact and Postman work in similar spaces but have different objectives in mind and can even be used as complimentary tools. For example, a colleague is currently using Postman to manually test and explore an API under development for our Teams and Permissions management feature.
Postman is a functional API testing tool and Pact is a contract testing tool. Using Pact, you could provide guarantees that your systems are compatible with one another using fast, isolated and reliable tests. Combined with Postman, you could have a collaborative developer experience that makes it easy to explore APIs, as well as a functional API testing suite for your provider.