Should I block AI crawlers in robots.txt?

It depends on whether being cited is worth more to you than the content being used for training, and those are separable: training collectors and search fetchers have different user agent names. The scored behaviour is stating a policy, not any particular policy.

Which AI crawler user agents should I name?

At minimum GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot and Google-Extended. The list changes, so keep it somewhere you will actually update.

What is the Schemamap directive, and should I use one?

A robots.txt line pointing at structured data feeds, read by NLWeb clients. We would not add one. Google's robots.txt report and every other mainstream validator call it an unknown directive and mark the whole file invalid, and the audience that reads it is small. Publish your catalog at /.well-known/ard.json instead, which agents check without being told and costs no warning.

Reference · updated 2026-09-16

How to write robots.txt for AI crawlers

Naming AI crawlers individually in robots.txt states a policy rather than leaving one to be inferred. A blanket User-agent: * with Allow: / technically permits everything, but it was almost certainly written before these crawlers existed, and scanners read it as absence of a policy rather than as consent. Listing GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot and the rest explicitly makes the decision legible either way.

Allowing and blocking are both valid, silence is not

This is not an argument for allowing AI crawlers. Plenty of businesses have good reasons to block training collection while permitting search crawling, and those are different crawlers with different names, which is exactly why the distinction is worth writing down.

What costs you is saying nothing. An agent reading a bare wildcard cannot tell whether you considered the question. Naming the crawlers answers it in one fetch.

Training versus search is a real distinction

GPTBot and ClaudeBot collect for training. OAI-SearchBot, Claude-SearchBot and PerplexityBot fetch to answer a live query and cite you. Google-Extended governs Gemini training separately from Googlebot's search indexing.

Most businesses want the second group and are ambivalent about the first. You can express that, and the fact that most robots.txt files do not is the gap.

The directive not worth adding

A Schemamap directive pointing at your structured data feeds is a real convention, but only inside one small ecosystem. Every mainstream validator, Google's robots.txt report included, treats it as an unknown directive and reports the whole file as invalid because of it.

We ran one here until September 2026 and removed it. Ours pointed at sitemap.xml, which is a list of pages rather than a structured data feed, so it was handing the few clients that read it the wrong thing while costing a validation error on every check. If you want an agent to find your catalog, publish it at /.well-known/ard.json, a path agents already look at without being told.

Naming crawlers, with a comment stating the policy

User-agent: *
Allow: /
Disallow: /api/

# AI crawlers are allowed deliberately, not by default.
# Retrieval and citation permitted. Attribution requested.

User-agent: GPTBot
Allow: /
Disallow: /api/

User-agent: ClaudeBot
Allow: /
Disallow: /api/

User-agent: PerplexityBot
Allow: /
Disallow: /api/

Sitemap: https://acme.example/sitemap.xml

Does your site have this?

The free checker scores your site against robots.txt for AI and everything else on this list, out of 100, in about ten seconds. It names what is missing rather than handing you a number.

Run the free check→

Common questions

Should I block AI crawlers in robots.txt?: It depends on whether being cited is worth more to you than the content being used for training, and those are separable: training collectors and search fetchers have different user agent names. The scored behaviour is stating a policy, not any particular policy.
Which AI crawler user agents should I name?: At minimum GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot and Google-Extended. The list changes, so keep it somewhere you will actually update.
What is the Schemamap directive, and should I use one?: A robots.txt line pointing at structured data feeds, read by NLWeb clients. We would not add one. Google's robots.txt report and every other mainstream validator call it an unknown directive and mark the whole file invalid, and the audience that reads it is small. Publish your catalog at /.well-known/ard.json instead, which agents check without being told and costs no warning.

Sources

01GPTBot documentation, OpenAI