Should I write contract tests for API clients and SDKs?

When creating and scaling out microservices across teams, one common approach is for API providers to create client libraries or SDKs for their services. This can be a convenient mechanism to increase adoption and reduce implementation effort for consumers. We are often asked if it's necessary to write contract tests in this case, and if so, how would you go about doing it.

The reasoning is that because the client SDK was created from a sanctified, trusted source, that the contract won't or can't be violated.

NOTE: This article applies equally to GraphQL and protocol buffers, which are essentially a Type system deployed over the network.

As an exercise, check to see if any of the following statements are not true:

1. My API never changes
2. Our consumers are released at the same time as our API providers

If one these conditions is false, then you need to find a way to ensure that in the time-span between the provider API being released and your consumer being released, that there is no possibility of the consumer having different expectations on the provider.

Put another way - can you guarantee that the client SDK version your consumer is using, is exactly the version the provider is currently built against?

This is where contract testing can help.

Old problems with a new name

Before RESTful APIs were a thing, way back when SOAP was the dominant species on earth, tools like Apache CXF and its predecessors reigned supreme. They were really convenient, except when they weren't - deployment time!

What we found was that you had to release all consumer changes (web application, API, whatever) at the same time as the API deployment. The sequence was something like this:

1. Put all systems into "maintenance mode"
2. Release Provider API
3. Release Consumers A...Z
n. Disable maintenance mode

This worked well, until you had to roll back one of the systems due to an incompatibility, at which point rollback was as painful as you might expect. The best of breed tools at the time created client SDKs for their SOAP APIs.

There was really tight release-time coupling. Coupling of APIs to consumers is unavoidable, but we'd prefer to "shift it left" to earlier in the pipeline, if not the Developer's machine. Runtime is too late.

Wash your hands of SOAP

Option 1: Write contract tests using the SDK

OK, so we've established that Client SDKs and similar abstractions such as GraphQL can be useful, but introduce a class of problems we'd like to avoid. Can we use contract tests, though?

Of course! Albeit the main caveat to point out, is that because the SDK may expect all possible attributes to be present, you may lose out on one benefit of contract testing - which is to know which specific attributes are actually required (you'll still get the benefits of which API endpoints are in use). This just means it will be harder for you to remove fields, because you won't truly know which fields are in use by the clients (because all clients expect all fields)

An example in Go

It is very common among Go microservices to do this sort of thing, so we built it into our Pact Go Workshop. Here, we have a client graciously provided from our User API team that does just this (NOTE: in the workshop the consumer/provider are in the same code base for convenience):

package client

import (
	"bytes"
	"encoding/json"
	"errors"
	"fmt"
	"io"
	"net/http"
	"net/url"

	"github.com/pact-foundation/pact-workshop-go/model"
)

// Client is our consumer interface to the Order API
type Client struct {
	BaseURL    *url.URL
	httpClient *http.Client
}

// GetUser gets a single user from the API
func (c *Client) GetUser(id int) (*model.User, error) {
	req, err := c.newRequest("GET", fmt.Sprintf("/users/%d", id), nil)
	if err != nil {
		return nil, err
	}
	var user model.User
	_, err = c.do(req, &user)

	if err != nil {
		return nil, ErrUnavailable
	}

	return &user, err

}

// GetUsers gets all users from the API
func (c *Client) GetUsers() ([]model.User, error) {
	req, err := c.newRequest("GET", "/users", nil)
	if err != nil {
		return nil, err
	}
	var users []model.User
	_, err = c.do(req, &users)

	return users, err
}

func (c *Client) newRequest(method, path string, body interface{}) (*http.Request, error) {
	rel := &url.URL{Path: path}
	u := c.BaseURL.ResolveReference(rel)
	var buf io.ReadWriter
	if body != nil {
		buf = new(bytes.Buffer)
		err := json.NewEncoder(buf).Encode(body)
		if err != nil {
			return nil, err
		}
	}
	req, err := http.NewRequest(method, u.String(), buf)
	if err != nil {
		return nil, err
	}
	if body != nil {
		req.Header.Set("Content-Type", "application/json")
	}
	req.Header.Set("Accept", "application/json")
	req.Header.Set("User-Agent", "Admin Service")

	return req, nil
}

func (c *Client) do(req *http.Request, v interface{}) (*http.Response, error) {
	if c.httpClient == nil {
		c.httpClient = http.DefaultClient
	}
	resp, err := c.httpClient.Do(req)
	if err != nil {
		return nil, err
	}
	defer resp.Body.Close()
	err = json.NewDecoder(resp.Body).Decode(v)
	return resp, err
}

var (
	ErrUnavailable = errors.New("api unavailable")
)

The key things to note here are the GetUser and GetUsers operations where they marshal a JSON response into a User model, which looks like:

type User struct {
	FirstName string
	LastName  string
	Username  string
	Type      string
	ID        int
}

Hypothetically, for our use case we might only care about the User's ID property, so we could write a Pact test as follows:

func TestClientPact_GetUser(t *testing.T) {
	t.Run("the user exists", func(t *testing.T) {
		id := 10

		pact.
			AddInteraction().
			Given("User sally exists").
			UponReceiving("A request to login with user 'sally'").
			WithRequest(request{
				Method: "GET",
				Path:   term("/users/10", "/users/[0-9]+"),
			}).
			WillRespondWith(dsl.Response{
				Status: 200,
				Body: map[string]interface{}{
					"ID": 10,
				},
				Headers: commonHeaders,
			})

		err := pact.Verify(func() error {
			user, err := client.GetUser(id)

			// Assert basic fact
			if user.ID != id {
				return fmt.Errorf("wanted user with ID %d but got %d", id, user.ID)
			}

			return err
		})

		if err != nil {
			t.Fatalf("Error on Verify: %v", err)
		}
	})
}

Notice that in our WillRespondWith.Body property (lines 15-17), we only request the ID object even though the Model has several others. In our case, our client isn't so brittle so we still retain the benefits of knowing which properties are actually in use by our consumers. In many cases, this is not possible, and you'll need to expect all of the JSON attributes even though you don't actually need them. This is an acceptable trade-off in our view, given the alternative.

Option 2: Ship pre-made contracts with the SDK

Another idea that we can borrow from the past, is to ship a technology compatibility kit or TCK (made famous by the Java JSR process) along with an SDK, to ensure it is compatible with the host environment.

So as an alternative to the above solution, this approach would be to ship pre-made contract tests or simply the contracts themselves, along with the client SDK. The provider team is responsible for creating an exhaustive contract testing suite that works with the specific version of the SDK/library being shared. The consumers could selectively enable the tests that are useful to them, or simply upload the contract in its entirety.

When this package is updated in the consumer code base, the new version of the contract is uploaded and associated with the current version of the client application. This has the drawback of not knowing which fields and endpoints are being used (if the full contract is uploaded), but does give you the guarantees around compatibilities and removes the need to synchronise deployments. It also has the benefit of being a much simpler workflow for the provider, and could get you doing a form of contract testing in a short amount of time.