Can I trust the AI crawler user agent strings I see in my logs?

No. A user agent is self-reported and trivially spoofed. Allowing or blocking on that basis affects well-behaved clients only. Verification requires a signature you can check against a published key.

Do I need Web Bot Auth if my site is read-only?

Almost certainly not yet. If an agent can only read pages a browser could also read, identity buys you analytics rather than security. It starts to matter when agents can act, spend, or consume something metered.

Reference · updated 2026-09-16

What is Web Bot Auth? Letting good agents identify themselves

Web Bot Auth is an approach to bot identity built on HTTP Message Signatures (RFC 9421), where an agent signs its requests with a key published in a directory you can fetch and verify. It exists because a user agent string is a claim, not a credential: anything can send `User-Agent: ClaudeBot`. A signature is checkable, which lets you admit the agents you want without admitting everything that impersonates them.

The problem it solves

Every decision you make about AI crawlers today rests on a header the caller controls. Allow GPTBot in robots.txt and you have allowed anything claiming to be GPTBot. Block it and you have blocked only the polite. The asymmetry is uncomfortable for anyone rate limiting, metering, or serving different content to agents.

Signed requests move the claim from something asserted to something verifiable. You fetch the directory, you check the signature, and you know who is calling.

Why it matters more as agents start acting

While agents only read, identity is mostly an analytics problem. The moment they can act, through WebMCP tools or an API, it becomes an authorisation problem. You will want to treat a verified assistant operating on a user's behalf differently from an unverified script, and you cannot do that on a user agent string.

This is the direction the whole agent web is moving: from 'can it read me' to 'do I know who it is'. Most sites are not there yet, and that is fine, but it is worth knowing which problem you are solving when you get there.

What this means if you are a small site

Probably nothing yet, and we would rather say so. If everything you expose is public and read-only, there is nothing an unverified agent can do that a browser could not, and the honest answer is to publish auth.md saying access is open and revisit this when you expose a write surface.

The point at which it starts mattering is the point at which an agent can spend your money, consume metered capacity, or take an action on a customer's behalf.

Does your site have this?

The free checker scores your site against Web Bot Auth and everything else on this list, out of 100, in about ten seconds. It names what is missing rather than handing you a number.

Run the free check→

Common questions

Can I trust the AI crawler user agent strings I see in my logs?: No. A user agent is self-reported and trivially spoofed. Allowing or blocking on that basis affects well-behaved clients only. Verification requires a signature you can check against a published key.
Do I need Web Bot Auth if my site is read-only?: Almost certainly not yet. If an agent can only read pages a browser could also read, identity buys you analytics rather than security. It starts to matter when agents can act, spend, or consume something metered.

Sources

01RFC 9421, HTTP Message Signatures, IETF