OAI-SearchBot is the crawler that determines whether your pages can be surfaced in ChatGPT search answers; GPTBot only collects data for model training. Blocking GPTBot does not remove you from ChatGPT search results, and blocking OAI-SearchBot does not protect your content from training. Most robots.txt files I audit get this backwards, which means teams are either losing visibility they wanted or believing they have protection they do not have.
Key Takeaways
- OpenAI documents three crawlers with separate jobs: OAI-SearchBot (search eligibility), GPTBot (training), and ChatGPT-User (user-initiated fetches). A newer OAI-AdsBot validates ad landing pages.
- Per OpenAI's documentation, sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers, though can still appear as navigational links."
- Blocking GPTBot is a training decision, not a visibility decision. It has no effect on whether you appear in ChatGPT search answers.
- ChatGPT-User handles fetches a user explicitly triggers and is, in OpenAI's words, "not used to determine whether content may appear in Search."
- On 8 August 2026, Search Engine Journal reported that site-scoped (
site:) queries inside ChatGPT's fan-out jumped from roughly 0.37% to 16.8%, a structural tilt toward domains the model already knows by name. - The practical move is to write robots.txt around intent, search-in/training-out, everything-out, or everything-in, rather than copy-pasting a block list from a blog post.
The Confusion Is Costing Real Visibility
In the last year I have reviewed robots.txt files for edtech companies, early-stage SaaS brands and a few marketplaces. The single most common finding is not a technical SEO problem in the classic sense. It is a team that read "block AI crawlers" advice in 2023, added a Disallow for every OpenAI user agent they could find, and then wondered a year later why they never show up when someone asks ChatGPT for recommendations in their category.
The reverse error is just as common and more consequential from a legal or brand perspective: a team blocks GPTBot, tells leadership "we've opted out of AI," and remains fully crawlable by every other model provider, and fully eligible for ChatGPT search citations anyway.
These are two different decisions. OpenAI's own documentation treats them as two different decisions. Your robots.txt should too.
What OpenAI Actually Documents
OpenAI publishes its crawler documentation at developers.openai.com, and it is unusually clear compared to most model providers. There are four user agents worth knowing.
OAI-SearchBot, the search eligibility crawler
This is the one that matters for visibility. OAI-SearchBot builds and maintains the corpus ChatGPT search draws from. OpenAI's documentation states that sites opted out of this crawler "will not be shown in ChatGPT search answers, though can still appear as navigational links."
That caveat is worth reading twice. Opting out does not make you invisible in ChatGPT: a link to you can still appear as a plain navigational result. But you lose the thing that actually drives consideration: being cited inside the answer, with your framing, your numbers and your language attached to the recommendation.
GPTBot, the training crawler
GPTBot collects publicly available content that may be used to train future models. That is its entire scope. Blocking GPTBot is a defensible policy decision, plenty of publishers and course businesses have made it, but it is a decision about model training data, not about being found.
If you block GPTBot and nothing else, you remain fully eligible for ChatGPT search citations.
ChatGPT-User, the on-demand fetcher
ChatGPT-User fires when a user or a custom GPT explicitly asks ChatGPT to visit a URL. OpenAI states it is "not used to determine whether content may appear in Search." Blocking it mostly means that when a real human pastes your URL into ChatGPT and asks "summarise this page," ChatGPT cannot.
I have yet to hear a good business reason for blocking this one. It is the closest thing in the AI stack to a person clicking your link.
OAI-AdsBot, the ad landing page validator
The newest of the four. OAI-AdsBot checks ad landing pages for policy and quality compliance. OpenAI states it does not feed training. If you run, or plan to run, ads in OpenAI's surfaces, blocking it will cause problems. If you do not, it is largely irrelevant to you.
Exact robots.txt Configurations by Intent
Here are the four configurations that cover almost every real business case. Pick the one that matches what you actually want.
Intent 1: Be visible in ChatGPT search, opt out of training
This is the configuration most commercial sites want and almost none have.
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: GPTBot
Disallow: /Intent 2: Full participation
If you have no training concerns and want maximum surface area, the honest answer is that you do not need OpenAI-specific rules at all, default-allow is the default. But being explicit documents the decision for whoever inherits the file.
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Allow: /
User-agent: ChatGPT-User
Allow: /Intent 3: Full opt-out
User-agent: OAI-SearchBot
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: OAI-AdsBot
Disallow: /Understand the trade: you accept losing in-answer citations across ChatGPT entirely.
Intent 4: Selective: protect gated or commercial paths, expose the rest
The configuration I recommend most often for edtech and SaaS. Your curriculum pages, pricing and comparison content stay crawlable; your paid course material, internal tooling and user-generated areas do not.
User-agent: OAI-SearchBot
Disallow: /app/
Disallow: /dashboard/
Disallow: /course-content/
Allow: /
User-agent: GPTBot
Disallow: /The Verification Step Everyone Skips
Write the rules, then confirm they do what you think. Three checks, in order:
- Fetch your own robots.txt as plain text. Not through a plugin preview.
curl https://yourdomain.com/robots.txtand read what actually ships. - Check for conflicting wildcard rules. A broad
User-agent: *block earlier in the file, or a CDN-level bot rule, can override your intent invisibly. - Check your WAF and CDN. This is the one that bites people. Cloudflare, Akamai and similar services ship bot-management rules that block AI crawlers at the edge regardless of what robots.txt says. Your robots.txt can say
Allowall day; if the edge returns 403, you are blocked.
Point three has been the actual root cause in the majority of "why aren't we cited?" audits I have run. Robots.txt was fine. The WAF was not.
The 8 August 2026 Retrieval Shift and Why It Matters Here
Search Engine Journal reported that on 8 August 2026, OpenAI changed how ChatGPT retrieves. Site-specific queries, the ones scoped with a site: operator to a particular domain, went from about 0.37% of fan-out queries to 16.8%.
That is a roughly 45x increase in the share of retrieval that targets a named domain directly.
What this changes strategically
When a meaningful slice of retrieval is domain-scoped, the model has to already know your domain name to scope a query to it. That structurally favours brands with existing name recognition, and it makes brand-name search demand an AEO input, not just a vanity metric.
What this does not change
It does not mean unbranded topical content stopped working: the majority of fan-out queries are still topical. And a single reported metric shift, however large, is a snapshot of a system OpenAI can and does change again. Treat it as directional evidence about where the platform is heading, not as a permanent law.
What I would actually do about it
Two things, both boring and both durable. First, make sure your brand name and your category are strongly associated in your own content: comparison pages, "alternatives to" pages, founder and practitioner bylines. Second, keep OAI-SearchBot unblocked, because domain-scoped retrieval is worth nothing if the crawler cannot read the domain when it gets there.
How This Fits Alongside Other AI Crawlers
OpenAI's split is not unique. Anthropic separates ClaudeBot, Claude-User and Claude-SearchBot along similar lines. Google's Google-Extended governs training and grounding in other Google products rather than Search AI features. Perplexity runs PerplexityBot against its own index.
The pattern across all of them: training crawlers and retrieval crawlers are separate, and the retrieval one is the one that affects whether you get cited. If you internalise nothing else from this post, internalise that.
Google's guidance on crawler control lives in Google Search Central, and it is worth reading alongside OpenAI's for the contrast in how the two companies frame the same choice.
A Practical Audit You Can Run This Week
Half a day of work, and I have never run it without finding something.
Step one: inventory what you currently block
Pull your live robots.txt, your CDN bot rules and any security plugin settings into one document. Most teams have never seen these three side by side.
Step two: write down the intent, in a sentence
"We want to be cited in ChatGPT answers but not used for training." Or whatever yours actually is. Get a decision-maker to sign off on that sentence before touching config.
Step three: reconcile config to intent
Rewrite robots.txt from the intent, not from the existing file. Then fix the edge rules to match.
Step four: log the change and set a review date
Crawler names change. OAI-AdsBot did not exist when many of these files were written. Six-monthly review is enough.
Frequently Asked Questions
Does blocking GPTBot remove me from ChatGPT search results?
No. GPTBot is the training crawler. Search eligibility is governed by OAI-SearchBot. This is the single most common misunderstanding in the space.
If I block OAI-SearchBot, do I disappear from ChatGPT entirely?
No. OpenAI's documentation says opted-out sites "will not be shown in ChatGPT search answers, though can still appear as navigational links." You lose in-answer citation, not all presence.
Is there a single user agent that blocks all OpenAI crawling?
Not one that covers every use. You have to list the agents individually, OAI-SearchBot, GPTBot, ChatGPT-User and OAI-AdsBot, because they serve different systems.
Do these crawlers respect robots.txt?
OpenAI states they do, and publishes IP ranges so you can verify requests are genuine. Verification matters because user-agent strings are trivially spoofed by unrelated scrapers.
Should I block ChatGPT-User?
Rarely. It fires when a person deliberately asks ChatGPT to read a URL. Blocking it degrades a genuine user action without protecting anything meaningful.
What is OAI-AdsBot for?
It validates advertising landing pages for policy and quality compliance and, per OpenAI, does not feed training. Relevant if you advertise on OpenAI surfaces; otherwise low priority.
Will unblocking OAI-SearchBot get me cited quickly?
Unblocking makes you eligible. It does not make you cited. Citation still depends on whether your content is the best available answer for the queries the model fans out. Eligibility is the floor, not the ceiling.
How do I know if my content is being cited in ChatGPT today?
There is no first-party OpenAI reporting equivalent to Search Console. You are left with manual prompt testing, referral traffic segmented by chatgpt.com referrer, and third-party tracking tools. Bing Webmaster Tools offers first-party citation data, but only for Microsoft Copilot.
Does the August 2026 site-scoped query shift mean small brands cannot compete?
It makes brand recognition more valuable, not competition impossible. Domain-scoped queries are a large minority of fan-out, not the whole of it. Topical authority still gets you retrieved on unbranded questions, which is how most people discover a brand they did not previously know.
How often should I revisit this configuration?
Twice a year is enough for most sites, plus any time a provider announces a new crawler. Set a calendar reminder; these files rot quietly.
If you are working through what your own crawler setup should say, or you would rather someone audited it with you than hand you another checklist, I would be glad to talk. I have spent 4+ years in marketing helping edtech and startup brands grow organically, most recently taking Masai School's Instagram from 26K to 117K and LinkedIn from 50K to 160K. You can see more of that work and reach me through the contact form at younusfardeen.com. No pitch, just a conversation about what is actually happening on your domain.