Markdown for LLMs: serving .md pages and content negotiation to AI agents

Offering a markdown version of your docs is the easy part of making them readable by AI agents. The harder part is making sure agents actually receive it. Agents arrive by two different doors, many of them won’t find the markdown unless something points them to it, and the markdown you serve can quietly say less than the page it came from.

This post is about what an agent gets when it opens a single page. The index file that tells an agent which pages exist is a separate topic, covered in our post on llms.txt for developer documentation.

Why agents want markdown, not your HTML page

A rendered docs page is mostly navigation, scripts, styles and footer links. Vercel measured this on one of its own posts: “The HTML version of this page is around 500KB. The markdown version is 3KB, a 99.37% reduction in payload size.”

Size is only half of it. When an agent fetches HTML, its tooling converts the page to text before the model reads it, and the Agent-Friendly Documentation Spec puts it bluntly: “The HTML-to-markdown conversion in web fetch pipelines is lossy and unpredictable.” If you serve markdown yourself, you decide what the agent reads.

Door one: .md URLs

The llms.txt proposal sets out a simple convention: provide a clean markdown version “at the same URL as the original page, either with .md appended (page.html.md) or with the extension replaced by .md (page.md)”, using index.html.md or index.md for URLs without a file name.

The spec’s markdown-url-support check passes when .md URLs return valid markdown with a 200 status, warns when only some pages support it, and fails when they return errors or HTML.

Door two: content negotiation

Content negotiation means the same URL returns different formats depending on what the request asks for. An agent that sends Accept: text/markdown is saying “markdown please”. The spec’s content-negotiation check passes only when the server returns markdown with Content-Type: text/markdown. Markdown served with the wrong content type is a warning.

Who sends that header? Ben Word’s agent markdown support matrix (last updated 22 June 2026) lists Claude Code, Cursor, GitHub Copilot (Chat and CLI), Microsoft Copilot, OpenCode and OpenClaw as sending text/markdown. Codex CLI is listed as partial support: it fetches HTML first, then looks for a <link rel="alternate" type="text/markdown"> link and makes a second request for the markdown. ChatGPT browse, Claude.ai, Gemini, Perplexity, Windsurf, Cline, Aider and others are listed as fetching only HTML.

That is fewer than half of the 20 agents in the matrix.

Why you need both doors, and a signpost

The header-sending agents get markdown automatically if your server honours the request. Everyone else needs a .md URL and some way of learning it exists. The agent-docs-spec README notes that most agents don’t know about llms.txt or .md URL variants unless told.

There are three practical signposts:

• Link .md URLs from your llms.txt. That file is the index agents can be pointed at.
• Add a directive at the top of each markdown page. The spec describes Anthropic’s Claude Code docs, which include “a blockquote at the top of every markdown page telling agents to fetch the documentation index at llms.txt”. Its directive checks warn when the directive is buried “past 50% of content, where it may be past truncation”.
• Add rel="alternate" type="text/markdown" to the HTML head. The llms.txt proposal recommends it, and it is exactly what Codex CLI looks for.

The gotcha that breaks negotiation silently

According to the matrix, Claude Code, Copilot CLI and Microsoft Copilot send Accept: text/markdown, text/html, */*. Markdown and HTML are listed at equal priority.

A bug report on the roots/post-content-to-markdown plugin shows what happens next. The plugin only served markdown when it was strictly preferred over HTML, so on a tie these agents got HTML. The same issue notes the acceptmarkdown.com readiness checker gave the affected site a perfect score, because its test sent text/html;q=1.0, text/markdown;q=0.5, “which is a different question”.

Our view: when you test negotiation, use a header copied from a real agent, not a tidy one you wrote yourself.

Check two: tabs flatten cleanly

Language tabs and accordions are browser UI. In markdown there are no tabs, only sequential content, so what the agent gets depends on how your pipeline serialises them.

Size compounds the problem. The spec’s tabbed-content-serialization check gives an example: “A tutorial with 11 language variants serializes into a single massive document where an agent might see only the first 1-3 variants.” It passes tabbed content that serialises to under 50,000 characters, warns at 50,000 to 100,000, and fails above 100,000. The Agent Reading Test includes a tabbed-content page that checks whether an agent reads all 8 language variants, not just the first few.

What clean looks like in the markdown:

• every tab is present, not just the default one
• each variant is preceded by a heading or label naming the language
• each code block is fenced with a language tag

Markdown that says less than the page

When markdown is generated separately from the HTML, the two can drift, and an agent reading the markdown has no signal that the HTML says more. The spec’s markdown-content-parity check compares the two versions: it passes when under 5% of content segments are missing from the markdown, warns at 5 to 20%, and fails at 20% or more. Run it alongside the tabs check.

Build or buy

You have three broad options.

• A hosted docs platform that does it already. Mintlify, for example, serves markdown when you add .md to a page URL or send Accept: text/markdown or Accept: text/plain, and generates llms.txt and llms-full.txt.
• Conversion at the edge. Cloudflare’s Markdown for Agents “converts HTML to Markdown at the edge” when the request asks for markdown, adds an x-markdown-tokens header with an estimated token count, and is available on Pro, Business and Enterprise plans. The maximum supported origin response is 2 MB. Edge conversion inherits whatever your HTML contains, so tabs and parity still need checking.
• Your own build step plus a rewrite rule. This is Vercel’s approach: a rewrite detects Accept: text/markdown and routes the request to a handler that returns markdown with the right content type.

llms-full.txt, the whole site in one file, is a different option again. Mintlify itself describes it as a “large file”, and large files bring their own problems for agents.

If your docs are already built through CI, emitting markdown is a natural extra output. Our post on CI workflows for documentation covers where that kind of step fits.

How to check it

Forward this to an engineer, swapping in a real page:

# Door one: does the .md variant exist and say it is markdown?
curl -sI https://docs.example.com/quickstart.md | grep -iE "^HTTP|content-type"

# Door two: content negotiation, using the header Claude Code actually sends
curl -s -o /dev/null -w "%{content_type}\n" \
  -H "Accept: text/markdown, text/html, */*" https://docs.example.com/quickstart

# Tabs: every language label should appear in the markdown
curl -s -H "Accept: text/markdown" https://docs.example.com/quickstart | grep -nE "Python|JavaScript|Go"

Pass means a 200 status and text/markdown on both doors, and every tab label present. For a fuller run, afdocs, the companion tool to the spec, runs the spec’s markdown checks across a site with npx afdocs check https://docs.example.com.

Once your build emits markdown, these checks cost little to keep passing, and they decide whether the agents that matter to your users see the page you wrote. For the wider argument on writing for both audiences, see why the choice between humans and AI agents is false, and for API references specifically, what agent-ready API docs require.

Get the full checklist

This post covers two of the 16 checks in our agent-ready docs checklist, both in the Read layer: “Markdown on request” is rated Important, and “Tabs flatten cleanly” is rated Optimise. Get the free checklist.