Contract testing with the OSS Pact framework and PactFlow is often compared to tools like Postman, Insomnia and other REST API testing tools.
In this article, we discuss the differences in approach and the pros and cons of using the Pact consumer-driven contract testing framework and Postman for various API testing-related scenarios.
Postman
Postman is a collaboration platform for API development, it has a fantastic IDE-like GUI that makes it easy to explore and navigate an API and a scripting language in NodeJS that you can use to automate checks against an API Provider that you can run in your CI pipelines.
Postman (and Newman) do not require direct access to the code for the System under Test (SUT), as they test it from the outside in a black-box style test. It's ideally suited to exploring and collaborating on APIs and running functional API tests.
Postman is a provider-only testing tool, in that it does not test how a consumer (e.g., a web application or another API) actually interacts with a provider. Postman testers can set up test suites (in the form of collections) that contain consumer expectations; however, these are not connected to real-world consumer behavior and must be maintained manually.
Pact - the most broadly adopted contract testing specification
Pact is an open source consumer driven contract testing framework designed to test distributed systems. It is a code-first tool for testing HTTP and message integrations using an approach known as contract tests. When combined with PactFlow (or a Pact Broker), it becomes a powerful continuous-delivery tool.
The main purpose of Pact is to ensure that a consumer (such as a Web App or another API) and a provider API are compatible with each other - without ever having to test them together using an end-to-end integrated test suite.
Pact tests are written like any other unit test, and requires access to the code base of the System under Test to ensure that any expectations are based on how the consumer and provider behave, so that the tests can't drift from the actual implementation.
PactFlow - contract testing for teams at scale
PactFlow is the complete contract testing platform, allowing teams and enterprises to test API integrations earlier in the software development lifecycle.
PactFlow works with the Pact open source consumer-driven contract testing framework and adds:
- Enterprise-ready security, user management, and collaboration features
- Exclusive bi-directional contract testing framework
- An enhanced UI
Learn more about PactFlow’s features.
Comparison: Postman, Pact and PactFlow
Here is an at-a-glance comparison of the tools:
| Postman | Pact | PactFlow | |
|---|---|---|---|
| Use Cases | HTTP APIs | HTTP APIs, Message APIs and Web Apps | Same as Pact + API gateways, third-party API testing |
| Usage | GUI, CLI (Newman) | Code / IDE / CLI | Code/IDE/CLI / Autogenerated |
| Language | NodeJS (scripting) | All major languages (11 in total), CLI tools | As per Pact |
| Testing Style | Black box (only tests from the "outside") | White box (needs access to code) | White box and black box |
| Test Author | Anyone (e.g. from another team) | Developers | Developers (Pact) Testers (bi-directional contract testing) |
| System Coverage | API Providers | API Consumers and API Providers | API consumers and API providers |
| Functional Testing | Yes | No | No |
| Contract Testing | No | Yes | Yes |
| Integration compatibility checks | No | Yes | Yes |
Pros and Cons
Postman
Pros
- Easy to get started and requires little to no coding ability.
- Ability to import schemas (such as OAS, JSONSchema, GrahpQL, and gRPC), and use them as a basis for tests, mocks, and API exploration.
- Postman collections make it easy to share and collaborate with others.
Cons
- Provider centricity – does not consider the consumer of the API.
- Encourages a separate test suite that requires maintenance outside of the code it is testing.
- Delay in feedback on API issues (due to a separate testing suite).
Pact
Pros
- Provides guarantees consumers and providers work together, and fast developer feedback.
- Tests are always valid and maintained, because they are generated by the code with native integration in the language of the System under Test.
- CLI tools explicitly designed for continuous-delivery pipelines and enable independently releasable software components.
Cons
- Can be more complicated to get started.
- Requires specialist coding skills.
- Not suited to external testing teams who are not part of the product team or don’t have access to source code.
When to use
Postman is ideally suited to the following tasks and activities:
- API exploration and discovery
- Functional/E2E API tests
- External testing teams who don't have access to the source code
Pact is ideally suited to:
- Preventing breaking changes between API consumers and providers (contract tests)
- Orchestrating continuous delivery pipelines for microservices and web applications
- Developers with access to source code
Can you do contract testing with Postman?
It is possible to do a pseudo contract-test using Postman, by making assumptions about the consumer code base and how it makes API calls, storing those requests as Postman collections specific to that consumer and making the provider test against that collection as part of its CI process. But any tool that is not tied to the actual implementation (code) will not have the same value proposition that something like Pact can guarantee.
Because Postman does not have access to the code, the test author will have to manually setup expectations that will likely both drift from implementation over time and have broader expectations on a provider. For example, you won't be able to say which specific fields you are interested in with any degree of confidence. If the consumer code changes how it interacts with the API Provider, the Postman suite would need to be manually updated or risk devolving into a false representation of the real consumer.
Additionally, with this type of test, it's unlikely you will have access to the source code therefore unlikely you will be able to stub out data and dependencies easily so they can run as isolated unit tests. In this case you may need to do this against a real, deployed version of the Provider in an E2E style test.
Postman favours E2E functional tests, whereas Pact is a contract-testing tool.
Tom Hombergs has an excellent article discussing the relative strengths and weaknesses of these approaches, which can be summarised below. Ultimately, you need to choose the tradeoffs that suit your situation best, which may include a combination of the two:
Conclusion
Both the Pact framework and Postman work in similar spaces but have different objectives in mind and can even be used as complimentary tools. For example, a colleague is currently using Postman to manually test and explore an API under development for our Teams and Permissions management feature.
Postman is a functional API testing tool and Pact is a specialty contract testing framework. Using the Pact framework, you could provide guarantees that your systems are compatible with one another using fast, isolated and reliable tests. Combined with Postman, you could have a collaborative developer experience that makes it easy to explore APIs, as well as a functional API testing suite for your provider.
Interested to try out Pact with minimal set up and effort? Create a free PactFlow Starter Plan.
