A disastrous tale of UI testing with Pact

by Beth Skurrie

I recently wrote a post entitled "Why Pact implementations fail and how you can avoid it" based on my own experiences and conversations I've had with people at different companies that have used Pact over the years. I want to do a deep dive into the experience of one particular company that ended up removing their Pact tests because of the pain they were causing - I hope it will help others to avoid the same pain.

Ron Holshausen and I caught up with a couple of people from this company years after their failed Pact implementation. We'd heard via the Melbourne grapevine that they'd had issues, and we wanted to get to the root of them. Over a cup of coffee they listed their woes to us.

Basically, their Pact tests were painful. On the consumer side, when the tests failed it was really hard to tell what caused the failures. The tests were really slow and hard to debug. They failed a lot of the time without helping them find any bugs. On the provider side, it seemed like there were so many interactions that needed to be verified, and many of them seemed like they were just slight variations of each other. Again, they seemed to fail often just because of the data set up, not because of any API incompatibilities.

As we listened to this description of their problems, I had a suspicion I knew what the cause was.

I drew them a diagram of their integration point, and asked them to mark on it what parts of the consumer was being covered by the Pact tests. Their answer confirmed my suspicion. Rather than generating their pacts via tests that were focussed on the integration itself, they were generating their pacts using tests that were designed to do feature level testing of the entire consumer codebase though the UI.

The dangers of testing through the UI

The proliferation of browser based, full stack integration tests over the years since tools like Selenium have become mainstream has created many problems. Yes, they tantalisingly promise safety, confidence, and bug free releases. But they are also flakey, and time consuming to maintain, debug and execute. At the last client I worked at, it took me almost a year of my "free time" at work to get one suite of flakey full stack Cucumber tests running reliably! UI driven, "scenario style" application tests are high value, but also high cost, and they cause a lot of pain when testing low level concerns like field names in HTTP requests and responses (which is exactly what Pact is meant for).

When executing scenarios through the UI that have multiple pages and steps involved, the likelihood of failure increases, as each step relies on the success of the previous steps. You're also more likely to have scenarios with significant portions of overlapping concern as you test different variations of the flow. This means your tests are increasing in execution and maintenance burden without a proportional increase in useful coverage.

Imagine using Pact to support some scenario style tests through the UI for a 4 step page flow. Each of the steps has 3 options. If tested though the UI, this could generate up to 81 Pact interactions (3*3*3*3) that then need to be verified on the provider side. If tested using unit tests that focussed only on the code that was responsible for making the call to the provider, it would only generate 12 (3+3+3+3). This is an overly simplistic example, however, it gives you an idea of the rate at which complications can increase when using an inappropriate "scope" of testing for a particular concern.

What to do instead

Despite their high cost, a small number of carefully chosen scenario style tests that are driven through the UI can provide good confidence that an application is working as intended. Ideally, these should use stubbed providers to avoid test time dependencies on downstream applications. How can we use the confidence that Pact tests give us in our integrations, in combination with the confidence that UI driven tests give us in our application as a whole?

The safest way to avoid the combinatorial explosion problem that we explored above is to use focussed unit tests to generate the contract files, and then use the contract files to seed stubs for the UI tests. This means that you have much greater control over what gets included in the contract, while still providing reliable stubs for your UI tests. Pactflow provides hosted stubs for every pact published, or you can use a local Pact stub server. You can read more about your options in this page in the Pact docs.

Another approach is to use shared data fixtures that are used in both the Pact tests, the UI component tests, and in the UI driven scenario tests. This means that you are essentially using "validated fixtures" all over your codebase. Be aware that you are now coupling your tests via these fixtures though, so any changes you make to them may require updates in many places.

You may have seen Matt's recent post on testing with Cypress and Pact together, and be wondering why we're exploring the concept of Pact+UI tests while simultaneously warning against using them. The answer is that the first solution that I suggested above isn't always a workable option, for various reasons that Matt touches on in his post. Writing Pact tests requires a high level of development expertise, which can sometimes make it difficult for testers without a coding background to contribute. Using Pact with Cypress opens up Pact tests to a wider community and reduces the barrier to entry.

BUT! With great power comes great responsibility! If you are going to use the UI in your Pact tests (despite all our warnings!), how can you avoid making the same mistakes that the ill-fated company I've talked about in this post did?

The answer is:

Remember that you are writing tests for the integration point that happen to go through the UI.
You are not using Pact to write tests for the UI.

Let's go over what that looks like in concrete terms.

  • The warning from before still holds true - do not use Pact for your scenario based, multi screen, multi-step tests.
  • Keep the testing scope to as few HTTP interactions in each test as possible, ideally a single call.
  • Try to keep the number of times a particular interaction is executed as close to 1 as possible.
  • Don't use Pact to test the UI itself - but you can use the generated pacts to provide fixture data for UI component tests that don't require a browser driver.
  • Don't try to test all of the variations of data in Pact tests (if they don't add any new information to a contract)
  • Don't try and test all the validation failures - focus on what the response looks like when the validation fails, not on the implementation of the validation rules themselves.

Conclusion

We all love to hear a good horror story about how someone else did something wrong. If we haven't made the same mistake, we get to feel smug; and if we have made the same mistake, it's nice to know that at least someone else has done the same thing! I hope that this post can keep more people in the "smug" category, and out of the "good company" category. Remember - if you can, avoid using the UI in your Pact tests. If you can't, then remember to test the integration point - not the UI. Make sure you don't end up being part of the company that rips all their Pact tests out again!

UPDATE (Feb 2021)

If you use Pactflow's bi-directional contracts capability, this problem disappears. Read more or contact us to find out more about this feature.