Robots.txt Rules for Search and AI Crawlers: A Working Guide

A robots.txt file asks crawlers which paths they may fetch. It follows the Robots Exclusion Protocol, published as RFC 9309 in September 2022. It controls crawling, not indexing and not access, and AI companies now publish separate crawler names for search, training and user-triggered fetching. This guide covers the rules, the current tokens (as of 13 November 2026) and a safe way to write the file.

Key Takeaways

• RFC 9309 says robots.txt rules are "not a form of access authorization". Anything private needs a login or server rule, not a Disallow line.
• Google's robots.txt documentation supports four fields: user-agent, allow, disallow and sitemap. Crawl-delay is not supported by Google.
• A blocked URL can still be indexed without a snippet. Google's robots.txt introduction says to use noindex, a password or removal to keep a page out of results.
• OpenAI documents separate tokens: OAI-SearchBot for search, GPTBot for training, ChatGPT-User for user-initiated actions. Each is set independently, according to OpenAI's crawler page.
• Anthropic documents ClaudeBot (training), Claude-SearchBot (search) and Claude-User (user requests) in its crawler help article.
• Perplexity says PerplexityBot is for search and is not used for training, while Perplexity-User generally ignores robots.txt because a person requested the fetch, per its bot documentation.
• A new individual IETF draft from an Anthropic author proposes a robots.txt token for AI training preferences. It is not a standard, and Google says it intends to support it in early 2027.

How the protocol actually works

The rules below come from RFC 9309 and Google's documentation. Where they differ, I say so.

Two points in that table cause most real mistakes.

First, a specific group does not inherit from the * group. If you write a group for one crawler, repeat every rule you want it to follow. A crawler named in its own group ignores the general group.

Second, a robots.txt that returns a server error is not the same as a missing file. A 5xx can halt crawling, while a 404 allows everything. If your CDN or application starts failing on /robots.txt during a deploy, crawling can stop for reasons that look like a ranking problem. Check this early when you triage a sudden drop; the order is in Technical SEO Triage Order.

Which token controls what

The tokens below are what each company documents today. Names, purposes and behavior change, so recheck the vendor page before you edit a live file.

Google's overview of its crawlers says common crawlers always respect robots.txt for automatic crawls, while user-triggered fetchers and special-case crawlers can behave differently. That distinction shows up again in the AI vendors' pages: search and training crawlers follow robots.txt, and user-triggered fetchers may not.

The practical split is purpose. A site that wants to appear in AI search answers but does not want its content used for training would allow the search tokens and disallow the training tokens. OpenAI says this is supported because each setting is independent. For the full OpenAI picture, see OAI-SearchBot vs GPTBot.

The new training-preference signal (as of 13 November 2026)

Anthropic's help article says that as of October 30, 2026 it has begun recognizing a machine-readable signal for "no AI training", based on an IETF draft, and that rules naming ClaudeBot directly take priority. The linked draft is draft-kelley-aipref-training-preference-signal-00, authored by Kevin Kelley of Anthropic and dated October 30, 2026.

What the draft proposes, as listed on the IETF datatracker page: a robots.txt group whose User-Agent line is the policy token __usage_ai-training, followed by ordinary Allow and Disallow rules. A Disallow: / expresses a preference against training on the whole site. Wildcard User-Agent: * groups do not count for this token.

What other vendors have said is mostly secondhand, so here is how I would sort it:

• Google (primary source opened): the common crawlers page says Google intends to support the signal in early 2027. That page describes Google-Extended as a standalone token for Gemini training and grounding and does not say how the two will relate.
• Microsoft and OpenAI (reported, not confirmed by me): PPC Land reports that Microsoft will honor training and grounding preferences from February 1, 2027 and that OpenAI aims to implement the signal "in the coming months". I could not open the original Bing and OpenAI posts, so treat those dates as reported.
• Anthropic (primary source opened): supports it now, with the caveat that its handling may change.

It is still an individual Internet-Draft, not an RFC, and the datatracker page says the IETF has not endorsed it. The IETF's AI Preferences working group is a separate, chartered effort on vocabulary and attachment mechanisms, including robots.txt and HTTP headers.

My read: adding the token is cheap and low risk, but until more crawlers confirm they honor it, keep your named-crawler rules (GPTBot, ClaudeBot, Google-Extended) as the source of truth.

A separate Google control for AI search features

Google's Search Console help page says that as of August 31, 2026 a Search generative AI control has rolled out to all websites worldwide. It covers AI Overviews, AI Mode and generative AI features in Discover, and it does not affect AI training, which Google points to Google-Extended for. Excluding content also means no traffic or impressions from those features, so this is a business decision, not a hygiene setting.

A safe method to write the file

1. Write down the policy first. For each purpose (search, AI search, training, user fetching), decide allow or block. A single sentence per purpose is enough.
2. List what must never be crawled. Carts, internal search results, staging paths. Remember that robots.txt does not secure them.
3. Write one group per crawler you name. Repeat the shared rules inside each named group.
4. Keep noindex out of it. Use a meta robots tag or X-Robots-Tag header, and leave the page crawlable. Google's block indexing documentation says that if robots.txt blocks the page, the crawler never sees the noindex rule.
5. Test with the live URL. Fetch /robots.txt, check the status code is 200, and check the content type. Then check the file at the CDN edge, not only at the origin.
6. Watch the logs. Confirm that the crawlers you allowed are actually arriving. The AI bot log file walkthrough shows how.

A hypothetical file for a content site that wants search visibility but opts out of training:

User-agent: *
Disallow: /cart/
Disallow: /internal-search

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

This is an illustration, not a recommendation. OAI-SearchBot has its own group, so it does not inherit the * disallows; add the cart lines to its group if you want it to skip them.

What this means for marketers and founders

• Check your CDN before your file. OpenAI recommends allowing OAI-SearchBot and its published IP ranges, and Google recommends allowing crawling at the CDN layer too. A bot protection rule can block what robots.txt allows. See Cloudflare, Googlebot and Default Blocks.
• Decide training and search separately. Blocking a training crawler and allowing a search crawler are different business choices.
• Do not use robots.txt for secrecy. The file is public by design, so it also reveals the paths you list.

Common mistakes

• Blocking a URL in robots.txt and expecting it to leave the index.
• Adding a named group and forgetting it no longer inherits the * rules.
• Using Crawl-delay for Googlebot. Google does not support it; Anthropic's article says it supports the extension for its own bots.
• Deploying a robots.txt that returns 5xx for a few hours.
• Copying a "block all AI bots" list from a blog without checking each token's purpose. Some tokens are search crawlers, and blocking them removes you from that product's results.

FAQ

What is robots.txt used for?

It tells crawlers which parts of a site they are asked not to fetch. It is a voluntary protocol defined in RFC 9309 and it does not provide security. Pages you list can still be reached by anyone with the URL.

Does blocking a page in robots.txt remove it from Google?

No. Google says a disallowed URL can still be indexed if other sites link to it, and it may show without a description. Use a noindex directive on a crawlable page, a password, or a removal.

What is the difference between GPTBot and OAI-SearchBot?

GPTBot is documented for content that may be used to train OpenAI models. OAI-SearchBot is for surfacing sites in ChatGPT search. OpenAI says the two settings are independent, so you can allow one and block the other.

Should I block AI crawlers?

It depends on whether you value appearing in AI search more than keeping content out of training. A common approach is to allow search crawlers and decide on training crawlers separately. Check each vendor's page for what each token does.

Do all AI bots obey robots.txt?

Vendors say their search and training crawlers do. User-triggered fetchers such as ChatGPT-User and Perplexity-User may not, because a person asked for the page. Third-party scrapers that do not identify themselves will not follow it at all.

How long do robots.txt changes take to work?

RFC 9309 says crawlers should not use a cached copy for more than 24 hours unless the file is unreachable, and Google says caching is generally up to 24 hours. OpenAI says search-related changes can take about 24 hours.

Want help with this?

I'm Younus Fardeen, an organic growth, SEO and AEO strategist with 4+ years of marketing experience across edtech and startup teams. If you want help auditing your robots.txt and crawler policy for search and AI visibility, see my work and reach me through the contact form at younusfardeen.com.