Reading:
Dealing with Hypermedia formats with Pact tests

Dealing with Hypermedia formats with Pact tests

Ronald Holshausen
Updated

Hypermedia based APIs are becoming much more prevalent as the format used for web application APIs. We use HAL for our APIs here at Pactflow, and the Pact Broker has been using HAL for it's APIs since the beginning.

There are lots of benefits for using hypermedia. Formats like HAL provide a structured way to organise related resources and a way at runtime to detect what options are available. And if the consumers of an API always use the links to interact with the API, the API endpoints can be moved around with out any impact.

Other formats like Siren take things even further by defining the actions a consumer can do with a resource as a list of action links.

The big issue with these types of API formats is that where the consumer needs to navigate a series of links, the Pact mock server will have to provide all the links in the responses. This is ok for the consumer test, but the resulting Pact file will have links relative to the mock server from the consumer test, and these will never match what the provider responds with. The provider could be mounted on a different root path and, more likely, could be running on a different host or port. The other issue is that IDs used as path variables could also be different.

To overcome this issue, a consumer test DSL function was added to match URLs. You can see an example of that in our sample Siren project.

// https://github.com/pactflow/example-siren/blob/master/consumer/src/__tests__/delete-order.spec.js#L70
{
  links: [
    {
      "rel": [ "self" ],
      "href": url("http://localhost:9000", ["orders", regex("\\d+", "1234")])
    }
  ],
}

NOTE: This function is only currently available with Pact-JVM, Pact-Rust, Pact-JS V3 Beta and Pact-C++ (as of November 2020).

This function works by specifying a base URL (which can change), and then a list of path fragments which can either be a string value or a matching function (the only really useful one to use is the regular expression matcher).

So, the example from above url("http://localhost:9000",["orders",regex("\\d+","1234")]) will create the URL http://localhost:9000/orders/1234 for use in the consumer test, but will generate a regular expression matcher ".*\\/orders\\/\\d+$" to match the URLs from the provider.

The second big problem is that these formats provide list of links to other resources and to the actions that can be executed on any resource. In some cases there would be more links than needed by what the consumer is doing in the test, and it would be tedious to have to provide them all for no benefit and if one was removed (even with the consumer not using it), the Pact verification would fail.

The other issue is that the consumer test would be tied to the order that these links are in the list, and if the provider changed the order, any Pact test would fail.

One of the new enhancements we have been working on that can solve this issue is the "array contains" matcher.

Matching items in an array against a list of required variants

The new matcher works by providing a list of required items (which can be totally different objects), and it will match the actual list from the provider if all the required variants match at least one item in the list.

Again, you can see this in our example Siren project. In the consumer test, we have:

// https://github.com/pactflow/example-siren/blob/master/consumer/src/__tests__/delete-order.spec.js#L73
  "actions": arrayContaining(
    {
      "name": "update",
      "method": "PUT",
      "href": url("http://localhost:9000", ["orders", regex("\\d+", "1234")])
    },
    {
      "name": "delete",
      "method": "DELETE",
      "href": url("http://localhost:9000", ["orders", regex("\\d+", "1234")])
    }
  )

This will match the actions for the order resource if they contain the "update" and "delete" actions. Note that the items in the list are also using the url matching function to match the links for the actions.

The actual provider returns something like:

      "actions": [
        {
          "name": "update",
          "method": "PUT",
          "href": "http://localhost:8080/orders/6774860028109588394"
        },
        {
          "name": "delete",
          "method": "DELETE",
          "href": "http://localhost:8080/orders/6774860028109588394"
        },
        {
          "name": "changeStatus",
          "method": "PUT",
          "href": "http://localhost:8080/orders/6774860028109588394/status"
        }
      ]

Here the changeStatus action link will be ignored.

You can also test this by commenting out the delete endpoint (or the update one) of the order controller in the provider project and then running the Pact verification for the provider.

For example, if we make the following change to the OrderController (provider/src/main/java/io/pactflow/example/sirenprovider/controllers/OrderController.java):

  @GetMapping
  public ResponseEntity<RepresentationModel<?>> orders() {
    Long id = Math.abs(new Random().nextLong());
    Order order = new Order(id);
    Link selfLink = actions(order);
    EntityModel<Order> model = EntityModel.of(order, selfLink);
    RepresentationModel<?> orders = CollectionModel.of(model);
    orders.add(linkTo(methodOn(OrderController.class).orders()).withSelfRel());
    return ResponseEntity.ok(orders);
  }

  @GetMapping(value = "/{id}")
  public ResponseEntity<EntityModel<Order>> order(@PathVariable(value = "id", required = true) Long id) {
    Order order = new Order(id);
    Link selfLink = actions(order);
    EntityModel<Order> model = EntityModel.of(order, selfLink);
    return ResponseEntity.ok(model);
  }

  @PutMapping("/{id}")
  public EntityModel<Order> update(@PathVariable(value = "id", required = true) Long id, Order order) {
    Link selfLink = actions(order);
    return EntityModel.of(order, selfLink);
  }

  private Link actions(Order order) {
    return linkTo(methodOn(OrderController.class).order(order.getId())).withSelfRel()
      .andAffordance(afford(methodOn(OrderController.class).update(order.getId(), null)))
//      .andAffordance(afford(methodOn(OrderController.class).delete(order.getId())))
      .andAffordance(afford(methodOn(OrderController.class).changeStatus(order.getId(), null)));
  }

  @PutMapping("/{id}/status")
  public EntityModel<Order> changeStatus(@PathVariable(value = "id", required = true) Long id, String status) {
    Order order = new Order(id);
    Link selfLink = actions(order);
    return EntityModel.of(order, selfLink);
  }

//  @DeleteMapping("/{id}")
//  public ResponseEntity<Void> delete(@PathVariable(value = "id", required = true) Long id) {
//    return ResponseEntity.ok().build();
//  }

will result in the following failure:

$ ./gradlew pactverify

> Task :startServer

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::        (v2.3.4.RELEASE)

2020-11-09 14:53:32.046  INFO 39485 --- [           main] i.p.e.s.SirenProviderApplication         : Starting SirenProviderApplication on ronald-P95xER with PID 39485 (/home/ronald/Development/Projects/Pact/example-siren/provider/build/libs/siren-provider-0.0.1.jar started by ronald in /home/ronald/Development/Projects/Pact/example-siren/provider)
2020-11-09 14:53:32.048  INFO 39485 --- [           main] i.p.e.s.SirenProviderApplication         : No active profile set, falling back to default profiles: default
2020-11-09 14:53:32.797  INFO 39485 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 8080 (http)
2020-11-09 14:53:32.808  INFO 39485 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
2020-11-09 14:53:32.808  INFO 39485 --- [           main] org.apache.catalina.core.StandardEngine  : Starting Servlet engine: [Apache Tomcat/9.0.38]
2020-11-09 14:53:32.870  INFO 39485 --- [           main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
2020-11-09 14:53:32.870  INFO 39485 --- [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 759 ms
2020-11-09 14:53:33.071  INFO 39485 --- [           main] o.s.s.concurrent.ThreadPoolTaskExecutor  : Initializing ExecutorService 'applicationTaskExecutor'
2020-11-09 14:53:33.221  INFO 39485 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8080 (http) with context path ''
2020-11-09 14:53:33.229  INFO 39485 --- [           main] i.p.e.s.SirenProviderApplication         : Started SirenProviderApplication in 1.53 seconds (JVM running for 1.903)
java -jar /home/ronald/Development/Projects/Pact/example-siren/provider/build/libs/siren-provider-0.0.1.jar is ready.

> Task :pactVerify_Siren_Order_Provider FAILED

Verifying a pact between Siren Consumer and Siren Order Provider
  [Using File /home/ronald/Development/Projects/Pact/example-siren/consumer/pacts/Siren Order Provider-Siren Order Service.json]
  get root
    returns a response which
      has status code 200 (OK)
      has a matching body (OK)
  get all orders
    returns a response which
      has status code 200 (OK)
      has a matching body (FAILED)
  delete order
    returns a response which
      has status code 200 (FAILED)
      has a matching body (OK)

NOTE: Skipping publishing of verification results as it has been disabled (pact.verifier.publishResults is not 'true')


Failures:

1) Verifying a pact between Siren Consumer and Siren Order Provider - get all orders

    1.1) body: $.entities.0.actions Variant at index 1 ({"href":http://localhost:9000/orders/1234,"method":DELETE,"name":delete}) was not found in the actual list

        [
          {
        -    "href": "http://localhost:9000/orders/1234",
        +    "href": "http://localhost:8080/orders/7779028774458252624",
            "method": "PUT",
            "name": "update"
          },
          {
        -    "href": "http://localhost:9000/orders/1234",
        -    "method": "DELETE",
        -    "name": "delete"
        +    "href": "http://localhost:8080/orders/7779028774458252624/status",
        +    "method": "PUT",
        +    "name": "changeStatus"
          }
        ]


    1.2) status: expected status of 200 but was 405



FAILURE: Build failed with an exception.

* What went wrong:
There were 2 non-pending pact failures for provider Siren Order Provider

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.6.1/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 4s
8 actionable tasks: 6 executed, 2 up-to-date

In particular, the first error above is $.entities.0.actions Variant at index 1 ({"href":http://localhost:9000/orders/1234,"method":DELETE,"name":delete}) was not found in the actual list

This is exactly what we want from a contract test.

Lists with very different content

The "array contains" matcher has been designed for the case where the items in the list have a totally different structure. In our example, the actions had different values but the structure was the same. But it need not be.

Examples of were this could be GraphQL union types or even "eventing" type systems where the API would return a list of events that have occurred with a resource. Different events would have a different structure. The other example I've seen is a document cataloguing system. A query to the API for the catalogue could return a list of matching documents with different structures depending on the types of the documents.

Conclusion

Hypermedia formats are becoming more popular for web application APIs, and using contract tests to verify them can be really useful to check that the required links and actions that a consumer is dependent on are always there, even when they move around. Pact allows you to do this using some of the new features we've been working on.

Five Reasons Why Your Contract Testing Initiative Could Fail (And How to Avoid Them)
7 July 2023

Five Reasons Why Your Contract Testing Initiative Could Fail (And How to Avoid Them)

Set yourself up for contract testing success. Joe Joyce - Senior Solutions Engineer at SmartBear - explains the five most common reasons a contract testing initiative could fail and how to avoid them.

4 min read

Proving E2E tests are a Scam
18 March 2021

Proving E2E tests are a Scam

"We can see that in this example Contract Testing requires approximately a tenth of the compute resources yet provides twice the number of test fixtures. An important takeaway for me was that the cost of testing your contracts doesn’t depend on the size of your system."

14 min read

arrow-up icon