How do I serve a markdown version of a page?

Serve the same content as text/markdown at the page path with a .md suffix, and also when the request sends Accept: text/markdown. Advertise it with a rel=alternate link or an RFC 8288 Link header pointing at the twin.

Does serving markdown to bots count as cloaking?

No. Cloaking means showing different content to crawlers than to users. A markdown twin is the same content in a different format, which is content negotiation and is what Accept headers are for.

Which user agents should get markdown automatically?

The AI crawlers and answer engines: GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended and similar. Keep the list in one place, because it changes.

Reference · updated 2026-09-16

How to serve markdown to AI agents (and why .md matters)

A markdown twin is a plain-markdown version of a page, served at the same path with a .md suffix, to a request sending Accept: text/markdown, or to a recognised AI crawler asking for HTML. It is the cleanest possible read for an agent: no navigation, no layout, no JavaScript, just the content and a frontmatter block carrying title, description, canonical and last-updated.

Why it beats letting them parse your HTML

An agent extracting facts from rendered HTML has to guess which parts are content and which are chrome. Every navigation item, cookie banner and footer link is noise it has to discard, and some of it gets discarded wrongly. A markdown twin removes the guessing.

It also survives the case where your content only exists after hydration. Many agents fetch without executing JavaScript, and a client-rendered page gives them an empty shell. A markdown twin is always server-rendered by definition.

Three ways in, and you want all three

The .md suffix is the discoverable one: an agent that has your URL can try it without being told. Accept: text/markdown is the polite one, for clients that negotiate. And serving markdown to a known AI crawler that asked for HTML is the proactive one, which some scanners reward explicitly.

Advertise the twin either with a rel=alternate link in the head or an RFC 8288 Link response header. Scanners check that the advertised target actually serves markdown, so advertising a path that 404s is worse than advertising nothing.

The bug that cost us a week of wrong data

Ours returned the homepage for every page on the site. Two causes stacked, and neither was visible without testing each path individually.

First, the route generating the markdown was marked force-static, which freezes one response at build time, so the requested path was never read. Second, and more subtly, we passed the path as a query parameter on a rewrite. On a rewrite the route handler still sees the original request URL, not the rewrite target, so the parameter never arrived and the handler fell back to its default. Pass the path as a request header instead; those do survive.

The lesson is narrow and worth having: test every twin, not one. A single working /index.md hides a completely broken system, and every AI crawler on your site was reading the wrong page in the meantime.

What goes in the frontmatter

Title, description, canonical and last-updated, at minimum. The point is that an agent gets metadata without scraping for it, and knows which URL to cite rather than citing the .md path.

A markdown twin with the frontmatter scanners look for

---
title: Commercial refrigeration installation
description: What Acme installs and services, and where.
canonical: https://acme.example/services
last-updated: 2026-09-16
---

# Commercial refrigeration installation

Acme specifies, installs and services refrigeration for commercial
kitchens across the UK and Ireland.

## What we do

- Survey and specification
- Installation and commissioning
- Servicing and compliance sign-off

Does your site have this?

The free checker scores your site against Markdown twins and everything else on this list, out of 100, in about ten seconds. It names what is missing rather than handing you a number.

Run the free check→

Common questions

How do I serve a markdown version of a page?: Serve the same content as text/markdown at the page path with a .md suffix, and also when the request sends Accept: text/markdown. Advertise it with a rel=alternate link or an RFC 8288 Link header pointing at the twin.
Does serving markdown to bots count as cloaking?: No. Cloaking means showing different content to crawlers than to users. A markdown twin is the same content in a different format, which is content negotiation and is what Accept headers are for.
Which user agents should get markdown automatically?: The AI crawlers and answer engines: GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended and similar. Keep the list in one place, because it changes.

Sources

01RFC 8288, Web Linking, IETF