API contract testing for docs: proving your OpenAPI spec matches the API

Most writing on API contract testing is aimed at QA engineers and microservice teams. We want to make a narrower case for anyone responsible for documentation: once AI agents and code generators are reading your docs, your OpenAPI spec stops describing the documentation and becomes the documentation. If it is wrong, every generated client and every agent-written integration inherits the error.

In our agent-ready docs checklist this check is called “The spec matches reality”. It sits in the TRUST layer at Critical priority, and this post explains why.

Your spec is now your most-read doc page

The OpenAPI Initiative lists the uses of a spec plainly: you can use it to “configure infrastructure, generate client code and create test cases for your APIs” (OpenAPI Initiative, What is OpenAPI?). Tools like OpenAPI Generator produce “API client libraries (SDK generation), server stubs, documentation and configuration automatically given an OpenAPI Spec” (OpenAPI Generator README).

Coding agents work the same way. They read the spec, not the guide pages around it, and write integration code straight from it. Nobody reads the spec the way a person reads a tutorial, pausing at a step and thinking “that doesn’t look right”. Whatever the spec says is what gets built.

We cover the wider picture in what agent-ready API docs require. This post deals with one guarantee only: that the spec is true. For the broader choice of description formats, see modern API documentation approaches.

What OpenAPI spec drift looks like

Apidog describes API drift as APIs and their documentation falling “out of sync”, and names one root cause directly: “A lack of comprehensive testing against API specifications creates opportunities for drift” (Apidog, Understanding the root causes of API drift).

OpenAPI spec drift usually looks mundane:

• A new required field is added in code but not in the spec. Generated clients don’t send it, and every request fails validation.
• The wrong error code. The endpoint returns 422 where the spec says 400. Client code that branches on 400 never handles the real error.
• An undocumented response. A 409 or 429 happens in production but appears nowhere in the spec. Generated clients have no type for it, and agents write no retry or conflict handling.
• A nullability mismatch. A field documented as non-nullable comes back null. Strictly typed clients throw on deserialisation.

Each of these is a documentation bug that ships as code. None is visible from the rendered reference page.

Contract testing turns the spec into a test

The mechanism is simple. A tool reads the spec, generates requests for every operation, sends them to a running API, and checks every response against what the spec promised.

Schemathesis is the tool our checklist names. Its quick start says it “generates inputs for each operation in your schema: schema examples, boundary values, and randomized data” (Schemathesis quick start). This is property-based API testing: rather than hand-writing cases, you state the properties every response must satisfy and let the tool search for counterexamples.

The Schemathesis checks reference lists what it verifies. In plain terms, a few that matter most for docs:

• status_code_conformance: the API returned a status code the spec doesn’t document for that operation.
• response_schema_conformance: the response body doesn’t match its schema.
• content_type_conformance: the Content-Type header doesn’t match the spec.
• negative_data_rejection: the API accepted input the spec says is invalid.
• ignored_auth: authentication the spec declares isn’t actually enforced.

The Schemathesis README states support for OpenAPI 2, 3.0 and 3.1 plus GraphQL, and the project is MIT licensed. The same README says it is “Used by teams at Spotify, WordPress, JetBrains, Red Hat, and dozens of other companies.”

As a sense of scale, the quick start’s own demo against its sample API generated 59 test cases and reported 20 unique failures. That is the tool’s demo, not a benchmark, but it shows how quickly mismatches surface.

Here is why this belongs in a docs checklist and not only a QA plan. Some failures will be API bugs and some will be spec bugs. Either way, the published contract is wrong, and either fix makes the docs true again. Someone has to decide which side moves, and that decision is a documentation decision as much as an engineering one.

Not the same as consumer-driven contract testing

Many leaders hear “contract testing” and think of Pact. Pact defines contract testing as “checking each application in isolation to ensure the messages it sends or receives conform to a shared understanding that is documented in a ‘contract’” and calls itself “a code-first consumer-driven contract testing tool” (Pact docs): the contract is generated from the consumers’ own tests.

Pact’s docs draw the line themselves: “Unlike a schema or specification (eg. OAS), which is a static artefact that describes all possible states of a resource, a Pact contract is enforced by executing a collection of test cases, each of which describes a single concrete request/response pair - Pact is, in effect, ‘contract by example’.”

In our view the two answer different questions. Consumer-driven tests protect known consumers, usually your own services. Spec-based tests protect the published spec that unknown consumers, including agents, rely on. A public API needs the second at minimum.

Older guides often recommend Dredd for spec-based testing. Its repository was “archived by the owner on Nov 8, 2024. It is now read-only” (Dredd on GitHub), so we wouldn’t start new work on it.

How to check it

This is the part to forward to an engineer. The Schemathesis quick start runs with one command:

uvx schemathesis run https://your-api.example.com/openapi.json

Three points to go with it:

1. Run it against staging or a test environment, not production. It sends deliberately invalid and randomised data.
2. Read the failures before you gate on them. The first run tells you where the spec and the API disagree; triage which side is wrong.
3. Then run it in CI on every change to the API or the spec, so drift is caught at merge time.

If your team’s API testing today means clicking “Try it out” in Swagger UI, that is still useful, but it only checks the calls you think of. Generated tests check the ones you didn’t.

Test the spec like code

The spec is the contract agents sign on your behalf, so test it like code. If your agents chain calls across endpoints, the same thinking extends to workflow descriptions with Arazzo. For the human-facing side of the same reference, see our REST API documentation best practices.

Get the full checklist

“The spec matches reality” is one of 16 checks in our agent-ready docs checklist, in the TRUST layer at Critical priority. The other checks cover how agents find, read and understand your docs. Get the free checklist.