Cypress + Pact: Front End Testing with Confidence

by Matt Fellows

One of the biggest challenges with UI tests is that you almost always have to trade off speed and flakiness for reliability and validity.

You either stub out a backend to make the tests less flakey, losing confidence as you risk it not being a real representation of reality. Or, you use a real backend environment to ensure things actually work they way they should, which sucks, because managing end-to-end test environments is hard.

We recently launched our hosted stubs feature that can be used as a backend replacement for such tests. They solve many of the problems with end-to-end UI tests because they are fast, deterministic and can be relied upon as a validated replacement of the real thing.

This requires, however, that you've already created your contracts in a separate testing layer - for example by writing pact tests to unit test your API client.

Update - April 2022

Our new feature—Bi-Directional Contract Testing—is now live. If your Provider uses this feature, you can now generate as many interactions as you please from your Cypress tests, without fear of creating your own disastrous tale of UI testing, and turning your API providers against the whole idea. Simply convert your Cypress stubs to a pact file, and upload those instead of writing Pact tests. See the docs for how, and read on for more about how Cypress and Pact can work together.

The case for creating Pacts in UI tests

But this approach is not always desirable—perhaps you don't have access to the code to write the tests, or perhaps the code is not in a state that lends itself to this sort of testing—and it would be nice to be able to generate the contracts from the UI tests directly. Here are some of the reasons we think it's worth pursuing further:

1. Improving the Cypress testing experience

Cypress has good support for dealing with network requests, and the documentation contains an excellent guide on the tradeoffs of stubbing vs e2e tests, summarised in the table below:

Real Server:

Pros Cons
More likely to work in production Requires seeding data
Test coverage around server endpoints Much slower
Great for traditional server-side HTML rendering Harder to test edge cases

Stubbing:

Pros Cons
Control of response bodies, status, and headers No guarantee your stubbed responses match the actual data the server sends
Can force responses to take longer to simulate network delay No test coverage on some server endpoints
No code changes to your server or client code Not as useful if you’re using traditional server side HTML rendering
Fast, < 20ms response times -

Similarly, in the Pact docs, we talk about the level in which Pact tests should be authored, stating that you should "avoid using Pact for tests that involve the UI":

Pact docs and why you should avoid using Pact in UI Tests

Whilst the second point may be addressed in the v4 Pact specification (see the RFC), we've seen some really bad Pact roll outs over the years that eventually led to this FAQ being written, and so we do recommend exercising caution. What we don't want to see is a poorly designed test suite that is hard to maintain. More on this later.

However being able to reliably produce stubs that match how the real API behaves and importantly - adapt when your tests adapt - would be a real win.

2. Reducing "duplication" for teams already using Pact

Teams writing Pact tests and similar API tests in Cypress could reduce the overlapping test areas and corresponding maintenance if the two tools were better integrated.

3. Enabling a broader audience to get the benefit from Pact

Currently, Pact tests are written as "white box" style tests, meaning they are generally authored by developers who maintain the code base. The Cypress experience is suited to a wider range of test authors who may benefit from writing and contributing to Pact tests.

4. Retrofitting "legacy" code bases with tests

Sometimes old code bases are a bit harder to pick apart and test at the "right layer", making unit testing much more difficult. So difficult that you just don't do it. You know the kinds of problems I'm talking about. Having a Pact test run "from the outside" has significant advantages for these use cases.

5. The community wants it!

It's been a feature request for a while now and the #cypress channel in our Slack workspace is making more noise. The request is neatly summed up a community member:

I want to run Cypress tests against mocked data and I want to have contracts between the frontend and backend. I think it would be great if those contracts could be used to supply mocks or equivalently, if the test mocks could be used to generate contracts.
Jacob Raihle

So we were convinced...

🎉 cypress-pact

To further convince ourselves, we created a demo app using a custom Cypress plugin and commands that shows how Pact + Cypress could be used together, and the results were impressive.

We were able to seamlessly integrate it into the tooling and have it work with the usual networking commands such as cy.server and cy.route as well as cy.wait leveraging aliases.

Here is a video that shows Cypress running in GUI mode as we test a product catalog React app:

  • On the first run, we navigate to / which calls out to a Product listing API at /products .
  • The second run extends the test to click through to a product page which needs to use the /products/:id endpoint. The test fails, because the interaction hasn’t been configured in Cypress or Pact - we caught a potentially important contract detail!
  • Add the product/:id interaction in and voila!
Example Cypress and Pact run

The next step is to extract this into a separate plugin and publish it to https://github.com/pact-foundation/cypress-pact

Benefits & Cautions

By integrating Pact with Cypress, there is an opportunity to reduce some of the downsides of stubbing in Cypress, namely:

  • Providing guarantees that request/responses will be supported by the provider
  • Coverage of all server endpoints and interactions required by the (web) application

Additionally, we can bring in new test authors to the Pact ecosystem, and reduce duplication for teams already using both tools, thereby saving time to value and reducing maintenance costs.

In summary we think it has great potential:

  • It's great developer experience - Cypress is a fantastic and easy-to-use tool
  • Can be a good option to retrofit testing - sometimes it's harder to refactor code and get all of the tests in place. Doing UI e2e (but not integrated) tests can be a quick way to build confidence
  • Opens up the use of Pact to a broader audience (because tests can be written at a higher level, and don't need direct access to the code base)
  • Reduces concerns around "duplication", where tests at multiple levels have significant overlap, and may make adoption harder in teams due to perceived or actual maintenance costs

It's not all roses and unicorns, however, and we think a word of caution is also warranted. The usual rules of Pact tests still apply.

If possible, write Pact tests as isolated unit tests that test the API client directly (i.e. avoid doing it through tools like Cypress)

If you can't do this, here are some guidelines that should help you avoid common pitfalls:

  • Do focus on testing the integration point itself, not the UI
  • Do keep the testing scope to as few HTTP interactions in each test as possible, ideally a single call
  • Do try to keep the number of times a particular interaction is executed as close to 1 as possible
  • Don't use Pact to test the UI itself
  • Don't try to test all of the variations of data in Pact tests (if they don't add any new information to a contract)
  • Don't try and test all the validation failures - focus on what the response looks like when the validation fails, not on the implementation of the validation rules themselves.
The aim with using Pact in Cypress tests should be to test the integration point through the UI, and not testing the UI itself.

It's worth keeping in mind that there are two sides to the contract, and if the generated contracts are bloated with many tests of the same endpoint with minor variations that don't significantly improve the actual API test coverage, it will create a maintenance burden and frustrations for your provider team.

Summary of approaches

Approach Tradeoff
Create pacts the usual way, and use the generated contracts for stubs instead of Cyrpress Safest way for most, least nice experience for Cypress users
Do Pact testing within Cypress tests (as demonstrated here) Good compromise: the best safety guarantees + simple Cypress experience
Use Cypress stubs + serialise them to a pact file Do not do this, unless you are using the bi-directional feature.

Sold on the idea?

We'd love to hear from you, if you'd like to know more here is how you can get involved: