Extending Pact with Plugins

UPDATE : Join the Pact Plugin Framework launch webinar on Dec 1: https://docs.pact.io/events/plugins-framework-launch 🚀.

Background

Pact was created initially to support the rise of RESTful microservices and has grown to be the de-facto API contract testing tool.

One of the strengths of Pact is its specification, allowing anybody to create a new language binding in an interoperable way. Whilst this has been great at unifying compatibility, the sprawl of languages makes it hard to add significant new features/behaviour into the framework quickly (e.g. GraphQL or Protobuf support).

Update for 2022: https://github.com/pact-foundation/pact-plugins is now a thing!🚀

The "shared core"

We have attempted to combat this time-to-market problem, by focussing on a shared implementation (the "shared core") in many of the languages. We initially bundled Ruby, because it was convenient, but have been slowly moving to our Rust core which solves many of the challenges that bundling Ruby presented.

It is worth noting that the "shared core" approach has largely been a successful exercise in this regard. There are many data points, but the implementation of WIP/Pending pacts was released (elapsed, not effort) in just a few weeks for the libraries that wrapped Ruby. In most cases, an update of the Ruby "binaries", mapping flags from the language specific API to dispatch to the underlying Ruby process, a README update and a release was all that was required. In many cases, new functionality is still published with an update to the Ruby binary, which has been automated through a script.

Moving beyond HTTP

But, the industry has continued to innovate since Pact was created in 2013, and RESTful microservices are only one of the key use cases these days - protocols such as Protobufs and Graphql, transports such as TCP, UDP and HTTP/2 and interaction modes (e.g. streaming or server initiated) are starting to become the norm. Standards such as AsyncAPI and CloudEvent are also starting to emerge.

For example, Pact is still a rather HTTP centric library, and the mixed success in retrofitting "message support" into all languages shows that extensions outside of this boundary aren't trivial, and in some respects are a second class citizen.

The reason is simple: HTTP doesn't change very often, so once a language has implemented a sensible DSL for it and integrated to the core, it's more a matter of fine tuning things. Adding message pact is a paradigm shift relative to HTTP, and requires a whole new developer experience of authoring tests, integrating to the core and so on, for the language author to consider.

Being able to mix and match protocol, transport and interaction mode would be helpful in expanding the use cases.

Further, being able to add custom contract testing behaviour for bespoke use cases would be helpful in situations where we can't justify the effort to build into the framework itself (custom protocols in banking such as AS2805 come to mind).

To give some sense of magnitude to the challenge, I put this table together in 2019 that shows some of the Pact deficiencies across popular microservice deployments. In my consulting career (which not-so-coincidentally also aligns quite closely with my Pact maintainership) I've encountered all of those technologies in one form or another.

The "shared core" approach can only take us so far, and we need another mechanism for extending behaviour outside of the responsibilities of this core. This is where I see a plugin approach working with our "shared core" model.

Proposal

Adding plugin support represents an inflection point, forging a new direction for contract-testing and Pact

NOTE: you can read the full proposal, comments and contribute here.

Objectives

1. Increase the capability and richness of the Pact ecosystem
2. Reduce time-to-market for a new feature
3. Reduce the barrier of entry to creating new features (previously, to have a broad impact you had know one of 2 fairly obscure languages: Ruby or Rust)
4. Increase the number of contributors making new features for Pact (should mostly flow from [3])
5. Make it easy to use a new feature

Proposal

The current proposal would involve:

1. Creating an HTTP (or RPC style such as gRPC) based plugin infrastructure in the Pact Reference library that was plugin aware and could communicate to a user-configured plugin (I have spiked this with golang already)
2. Updating each implementation to support a generic plugin type (potentially namespaced by the plugin name)
3. Supporting serialising of arbitrary interaction types in the pact file
4. (eventually) creating a rich library (probably an extension of one of the existing crates such as libmatching) that can help reduce boilerplate for each plugin (e.g. for flexible matching)

Example serialised pact file:

{
  "consumer": {
    "name": "TCPConsumer"
  },
  "provider": {
    "name": "TCPProvider"
  },
  "interactions": [
    {
      "type": "tcp",
      "description": "a hello request",
      "request": {
        "message": "hello"
      },
      "response": {
        "message": "world!"
      }
    }
  ],
  "metadata": {
    "pactSpecification": {
      "version": "4.0.0"
    },
    "plugin": {
      "name": "pact-foundation/tcp",
      "version": "1.0.0",
      "delimiter": "\r\n"
    }
  }
}

A type attribute could be added to interactions (see #79) to denote that this is a non-standard interaction (there may need to be other discriminating information).

A separate section of the metadata could be used to store plugin specific configuration.

Pros/Cons

The benefit of this approach, would be that from a framework perspective, a single plugin infrastructure could be created once and any number of plugins could then leverage the framework.

It could also open up a much richer contributor community, as plugins could be written once in any language of the contributors choosing, and contribute a new feature to the entire framework in a single go.

The main downside is that because it's not part of the framework, it may suffer from not being a "first class citizen".

I see the plugin approach as a way of assessing product viability - if a plugin gains popularity/momentum, it could be a candidate for incorporating into the framework proper.

Caveats

• Different plugin types may have different needs, and therefore the HTTP based approach may not be suitable for all plugin types (e.g. for matching, it might be better served via a shared lib).
• The current philosophy is to embrace the polyglot nature of the toolchain (benefits described above). This implies plugins should be able to be written in any language/runtime the author chooses. This will almost certainly lead to portability issues and differences in how processes are managed (e.g. an author creating a plugin specific to their immediate use case), but that's an acceptable tradeoff I think as we can always look to improve the situation.
• Specifically on the "matching" plugin type, one wonders if that is still best contributed directly to the core, rather than via an extension (for the reasons described above). We could create an FFI approach for contributors to create new matching types that can be used both by core Pact libraries, and also any plugin itself. I'm keen to hear your thoughts, because mine are not very fleshed out in this regard.
• On the ability to have more modularity in the plugin ecosystem (e.g. separate plugins for transport, protocol, matching etc. that can then be mixed and matched) I've not got a good answer to this. The "considered alternatives" touches on some of the issues/constraints that I can into, and why I think the HTTP provider is a good starting point. I do have some ideas, but none of them have passed just a small amount of reasoning about. At this stage, I'm of the view that "perfect is the enemy of the good" and will violate one of the stated objectives.

Design

Plugin Design - Consumer

High Level Summary

1. User is responsible for starting the plugin following plugin specific documentation. The plugin must start an administration HTTP server, which will be used by the framework to communicate instructions for each Test Session
2. Pact is given plugin specific configuration - including the administration API details - which is then sent to the administration server to initialise a new test session. This step should result in a new service being started for use by the test code (e.g. a TCP socket or a protobuf server) and a unique session ID returned. Each session must be thread safe and isolated from any other sessions
3. The Pact framework will maintain the details of the TestSession - including interactions, failures, logs etc.
4. The calling code is now able to add Interactions to the plugin, which are stored by the framework and registered with the plugin. The plugin is responsible for defining what an Interaction looks like and how it should be passed in for its specific combination of protocol, payload, transport and interaction type.
5. During Test Execution, the calling code communicates directly to the Mock Service provided by the plugin. The Mock Service is responsible for handling the request, comparing the request against the registered interactions, and returning a suitable response. It must keep track of the interactions that were matched during the test session.
6. After each individual Test Execution, verify() is called to see if the expected Interactions matched the actual Interactions. Any mismatches are retrieved from the plugin and returned to the caller.
7. If the Test Session was successful, write_pact() is called which will write out the actual pact file.
8. The plugin is shutdown by the User code.

Consumer Sequence Diagram

Example consumer test

Here is an example for a raw "hello world" TCP provider. It should respond with "world!" if "hello" is sent:

func TestPluginPact(t *testing.T) {
	// Start plugin
	go startTCPPlugin()

	provider, err := v3.NewPluginProvider(v3.PluginProviderConfig{
		Consumer: "V3MessageConsumer",
		Provider: "V3MessageProvider", // must be different to the HTTP one, can't mix both interaction styles
		Port:     4444,                // Communication port to the provider
	})

	if err != nil {
		t.Fatal(err)
	}

	type tcpInteraction struct {
		Message   string `json:"message"`   // consumer request
		Response  string `json:"response"`  // expected response
		Delimeter string `json:"delimeter"` // how to determine message boundary
	}

	// Plugin providers could create language specific interfaces that except well defined types
	// The raw plugin interface accepts an interface{}
	provider.AddInteraction(tcpInteraction{
		Message:   "hello",
		Response:  "world!",
		Delimeter: "\r\n",
	})

	// Execute pact test
	if err := provider.ExecuteTest(tcpHelloWorldTest); err != nil {
		log.Fatalf("Error on Verify: %v", err)
	}
}

Plugin Design - Provider

High Level Summary

1. User is responsible for starting the plugin following plugin specific documentation. The plugin must start an administration HTTP server, which will be used by the framework to communicate instructions for each Test Session
2. Pact is given plugin specific configuration - including the administration API details - which is then sent to the administration server to initialise a new provider Test Session.
3. The user starts the Provider Service, and runs the verify() command
3. Pact fetches the pact files (e.g. from the broker), including the pacts for verification details if configured, and stores this information.
5. For each pact, the framework will be responsible for configuring provider states, and sending each interaction from the pact file to the plugin. The plugin will then perform the plugin-specific interaction, communicating with the Provider Service and returning any mismatches to the framework. This process repeats for all interactions in all pacts.
6. The Pact framework will maintain the details of the TestSession - including pacts, interaction failures, pending status, logs etc.
4. Pact calculates the verification status for the test session, and optionally publishes verification results back to a Broker
8. The Pact client library then conveys the verification status, and the User terminates all process.s

Provider Sequence Diagram

Example provider test

Here is an example for a raw "hello world" TCP provider test.

func TestV3PluginProvider(t *testing.T) {
	go startTCPPlugin()
	go startTCPProvider()

	provider, err := v3.NewPluginProvider(v3.PluginProviderConfig{
		Provider: "V3MessageProvider",
		Port:     4444, // Communication port to the provider
	})

	verifier := v3.HTTPVerifier{
		PluginConfig: provider,
	}

	if err != nil {
		t.Fatal(err)
	}

	// Verify the Provider with local Pact Files
	err = verifier.VerifyPluginProvider(t, v3.VerifyPluginRequest{
		BrokerURL:      os.Getenv("PACT_BROKER_URL"),
		BrokerToken:    os.Getenv("PACT_BROKER_TOKEN"),
		BrokerUsername: os.Getenv("PACT_BROKER_USERNAME"),
		BrokerPassword: os.Getenv("PACT_BROKER_PASSWORD"),
		PublishVerificationResults: true,
		ProviderVersion:            "1.0.0",		
		StateHandlers: v3.StateHandlers{
			"world exists": func(s v3.ProviderStateV3) error {
				// ... do something
				return nil
			},
		},
	})

	assert.NoError(t, err)
}

Plugin Design - plugin discovery, initialisation & dependencies

Another challenge is how we can reliably discover, install and initialise plugins, and also how one plugin may depend on another. For example, a protobuf plugin may want to use gRPC as a transport, which might be provided by a separate plugin.

Considered alternatives

The bulk of this thinking was done over the last year, whilst considering how to achieve a gRPC/Protobufs integration. It's a good candidate, because it has new interaction styles (e.g. streaming, server push), new transport (http/2) and different protocols (Protobuf, JSON).

Option 1. Build a shared library and link to Rust engine

Rust (the core Pact engine) is famously not dynamic, and very much likes to know about all code that can run in advance. Whilst libraries can be linked, integrating them at runtime as would be required by a general user-defined plugin system is not easily supported (and certainly not recommended).

Option 2: Don't build a plugin ecosystem, just do it in the core

Supporting a generic protobuf server suffers similar issues to (1) - (need for reflection), and the gRPC/protobuf ecosystem in Rust is fairly poor compared to other languages. So any attempt to do it directly in the Rust core would likely come up short.

I spiked creating a shared library in Golang that could be linked at compile time, which has great support for both gRPC and protobufs. Whilst I demonstrated that linking this library to Rust would work, I realised that every single language that wanted protobufs support would then need significant changes to add support for it in this way. Ditto for every other change.

Given how long it's taken to replace the core library to Rust in several languages thus far, this option seemed the least likely to succeed.

Conclusion

My Pactflow co-founders and I are committed to Open Source ❤️ and ensuring Pact remains the de-facto contract testing tool - it's literally why we created Pactflow in the first place. We're excited to see plugins land in Pact and the impact it will have on our global users and so have committed to delivering this enhancement through our continuing partnership with the community.

Join the launch 🚀

Join the Pact Plugin Framework launch webinar on Dec 1: https://docs.pact.io/events/plugins-framework-launch.