Dealing with Hypermedia formats with Pact tests
Hypermedia based APIs are becoming much more prevalent as the format used for web application APIs. We use HAL for our APIs here at Pactflow, and the Pact Broker has been using HAL for it's APIs since the beginning.
There are lots of benefits for using hypermedia. Formats like HAL provide a structured way to organise related resources and a way at runtime to detect what options are available. And if the consumers of an API always use the links to interact with the API, the API endpoints can be moved around with out any impact.
Other formats like Siren take things even further by defining the actions a consumer can do with a resource as a list of action links.
Dealing with links in Pact tests
The big issue with these types of API formats is that where the consumer needs to navigate a series of links, the Pact mock server will have to provide all the links in the responses. This is ok for the consumer test, but the resulting Pact file will have links relative to the mock server from the consumer test, and these will never match what the provider responds with. The provider could be mounted on a different root path and, more likely, could be running on a different host or port. The other issue is that IDs used as path variables could also be different.
To overcome this issue, a consumer test DSL function was added to match URLs. You can see an example of that in our sample Siren project.
// https://github.com/pactflow/example-siren/blob/master/consumer/src/__tests__/delete-order.spec.js#L70
{
links: [
{
"rel": [ "self" ],
"href": url("http://localhost:9000", ["orders", regex("\\d+", "1234")])
}
],
}
NOTE: This function is only currently available with Pact-JVM, Pact-Rust, Pact-JS V3 Beta and Pact-C++ (as of November 2020).
This function works by specifying a base URL (which can change), and then a list of path fragments which can either be a string value or a matching function (the only really useful one to use is the regular expression matcher).
So, the example from above url("http://localhost:9000",["orders",regex("\\d+","1234")]) will create the URL http://localhost:9000/ for use in the consumer test, but will generate a regular expression matcher orders/1234".*\\/orders\\/\\d+$" to match the URLs from the provider.
Lists of actions or links
The second big problem is that these formats provide list of links to other resources and to the actions that can be executed on any resource. In some cases there would be more links than needed by what the consumer is doing in the test, and it would be tedious to have to provide them all for no benefit and if one was removed (even with the consumer not using it), the Pact verification would fail.
The other issue is that the consumer test would be tied to the order that these links are in the list, and if the provider changed the order, any Pact test would fail.
One of the new enhancements we have been working on that can solve this issue is the "array contains" matcher.
Matching items in an array against a list of required variants
The new matcher works by providing a list of required items (which can be totally different objects), and it will match the actual list from the provider if all the required variants match at least one item in the list.
Again, you can see this in our example Siren project. In the consumer test, we have:
// https://github.com/pactflow/example-siren/blob/master/consumer/src/__tests__/delete-order.spec.js#L73
"actions": arrayContaining(
{
"name": "update",
"method": "PUT",
"href": url("http://localhost:9000", ["orders", regex("\\d+", "1234")])
},
{
"name": "delete",
"method": "DELETE",
"href": url("http://localhost:9000", ["orders", regex("\\d+", "1234")])
}
)
This will match the actions for the order resource if they contain the "update" and "delete" actions. Note that the items in the list are also using the url matching function to match the links for the actions.
The actual provider returns something like:
"actions": [
{
"name": "update",
"method": "PUT",
"href": "http://localhost:8080/orders/6774860028109588394"
},
{
"name": "delete",
"method": "DELETE",
"href": "http://localhost:8080/orders/6774860028109588394"
},
{
"name": "changeStatus",
"method": "PUT",
"href": "http://localhost:8080/orders/6774860028109588394/status"
}
]
Here the changeStatus action link will be ignored.
You can also test this by commenting out the delete endpoint (or the update one) of the order controller in the provider project and then running the Pact verification for the provider.
For example, if we make the following change to the OrderController (provider/src/main/java/io/pactflow/example/sirenprovider/controllers/OrderController.java):
@GetMapping
public ResponseEntity<RepresentationModel<?>> orders() {
Long id = Math.abs(new Random().nextLong());
Order order = new Order(id);
Link selfLink = actions(order);
EntityModel<Order> model = EntityModel.of(order, selfLink);
RepresentationModel<?> orders = CollectionModel.of(model);
orders.add(linkTo(methodOn(OrderController.class).orders()).withSelfRel());
return ResponseEntity.ok(orders);
}
@GetMapping(value = "/{id}")
public ResponseEntity<EntityModel<Order>> order(@PathVariable(value = "id", required = true) Long id) {
Order order = new Order(id);
Link selfLink = actions(order);
EntityModel<Order> model = EntityModel.of(order, selfLink);
return ResponseEntity.ok(model);
}
@PutMapping("/{id}")
public EntityModel<Order> update(@PathVariable(value = "id", required = true) Long id, Order order) {
Link selfLink = actions(order);
return EntityModel.of(order, selfLink);
}
private Link actions(Order order) {
return linkTo(methodOn(OrderController.class).order(order.getId())).withSelfRel()
.andAffordance(afford(methodOn(OrderController.class).update(order.getId(), null)))
// .andAffordance(afford(methodOn(OrderController.class).delete(order.getId())))
.andAffordance(afford(methodOn(OrderController.class).changeStatus(order.getId(), null)));
}
@PutMapping("/{id}/status")
public EntityModel<Order> changeStatus(@PathVariable(value = "id", required = true) Long id, String status) {
Order order = new Order(id);
Link selfLink = actions(order);
return EntityModel.of(order, selfLink);
}
// @DeleteMapping("/{id}")
// public ResponseEntity<Void> delete(@PathVariable(value = "id", required = true) Long id) {
// return ResponseEntity.ok().build();
// }
will result in the following failure:
$ ./gradlew pactverify
> Task :startServer
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v2.3.4.RELEASE)
2020-11-09 14:53:32.046 INFO 39485 --- [ main] i.p.e.s.SirenProviderApplication : Starting SirenProviderApplication on ronald-P95xER with PID 39485 (/home/ronald/Development/Projects/Pact/example-siren/provider/build/libs/siren-provider-0.0.1.jar started by ronald in /home/ronald/Development/Projects/Pact/example-siren/provider)
2020-11-09 14:53:32.048 INFO 39485 --- [ main] i.p.e.s.SirenProviderApplication : No active profile set, falling back to default profiles: default
2020-11-09 14:53:32.797 INFO 39485 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8080 (http)
2020-11-09 14:53:32.808 INFO 39485 --- [ main] o.apache.catalina.core.StandardService : Starting service [Tomcat]
2020-11-09 14:53:32.808 INFO 39485 --- [ main] org.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.38]
2020-11-09 14:53:32.870 INFO 39485 --- [ main] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext
2020-11-09 14:53:32.870 INFO 39485 --- [ main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 759 ms
2020-11-09 14:53:33.071 INFO 39485 --- [ main] o.s.s.concurrent.ThreadPoolTaskExecutor : Initializing ExecutorService 'applicationTaskExecutor'
2020-11-09 14:53:33.221 INFO 39485 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (http) with context path ''
2020-11-09 14:53:33.229 INFO 39485 --- [ main] i.p.e.s.SirenProviderApplication : Started SirenProviderApplication in 1.53 seconds (JVM running for 1.903)
java -jar /home/ronald/Development/Projects/Pact/example-siren/provider/build/libs/siren-provider-0.0.1.jar is ready.
> Task :pactVerify_Siren_Order_Provider FAILED
Verifying a pact between Siren Consumer and Siren Order Provider
[Using File /home/ronald/Development/Projects/Pact/example-siren/consumer/pacts/Siren Order Provider-Siren Order Service.json]
get root
returns a response which
has status code 200 (OK)
has a matching body (OK)
get all orders
returns a response which
has status code 200 (OK)
has a matching body (FAILED)
delete order
returns a response which
has status code 200 (FAILED)
has a matching body (OK)
NOTE: Skipping publishing of verification results as it has been disabled (pact.verifier.publishResults is not 'true')
Failures:
1) Verifying a pact between Siren Consumer and Siren Order Provider - get all orders
1.1) body: $.entities.0.actions Variant at index 1 ({"href":http://localhost:9000/orders/1234,"method":DELETE,"name":delete}) was not found in the actual list
[
{
- "href": "http://localhost:9000/orders/1234",
+ "href": "http://localhost:8080/orders/7779028774458252624",
"method": "PUT",
"name": "update"
},
{
- "href": "http://localhost:9000/orders/1234",
- "method": "DELETE",
- "name": "delete"
+ "href": "http://localhost:8080/orders/7779028774458252624/status",
+ "method": "PUT",
+ "name": "changeStatus"
}
]
1.2) status: expected status of 200 but was 405
FAILURE: Build failed with an exception.
* What went wrong:
There were 2 non-pending pact failures for provider Siren Order Provider
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.6.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 4s
8 actionable tasks: 6 executed, 2 up-to-date
In particular, the first error above is $.entities.0.actions Variant at index 1 ({"href":http://localhost:9000/orders/1234,"method":DELETE,"name":delete}) was not found in the actual list
This is exactly what we want from a contract test.
Lists with very different content
The "array contains" matcher has been designed for the case where the items in the list have a totally different structure. In our example, the actions had different values but the structure was the same. But it need not be.
Examples of were this could be GraphQL union types or even "eventing" type systems where the API would return a list of events that have occurred with a resource. Different events would have a different structure. The other example I've seen is a document cataloguing system. A query to the API for the catalogue could return a list of matching documents with different structures depending on the types of the documents.
Conclusion
Hypermedia formats are becoming more popular for web application APIs, and using contract tests to verify them can be really useful to check that the required links and actions that a consumer is dependent on are always there, even when they move around. Pact allows you to do this using some of the new features we've been working on.