Which crawlers are you deciding about?
Each company names its own, and each name has one job. The job is what you are deciding about, so here they are by company, with the job beside each name. The ones that serve answers are the AI search crawlers; the others gather pages for training.
- OpenAI: OAI-SearchBot surfaces websites in ChatGPT search, and GPTBot crawls content that may be used to train its models (OpenAI).
- Anthropic: Claude-SearchBot improves Claude's search results, Claude-User fetches a page when someone asks Claude a question, and ClaudeBot collects training data (Anthropic).
- Perplexity: PerplexityBot surfaces and links websites in Perplexity's search results, and Perplexity says it is not used to crawl content for AI foundation models (Perplexity).
- Google: Google-Extended is a robots.txt token that manages whether content Google crawls may be used to train future Gemini models and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI (Google).
What happens if you block the search crawlers?
You put that assistant's search answers at risk. OpenAI is plain about it: sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers (OpenAI). Anthropic is more careful with its words, and says disabling Claude-SearchBot or Claude-User may reduce a site's visibility in its search results (Anthropic). Keep that "may" in mind; it is a possibility, stated as one.
PerplexityBot is the crawler Perplexity names for surfacing and linking websites in its results (Perplexity). A robots.txt file tells crawlers which URLs they may access (Google Search Central), so a line that refuses PerplexityBot asks the crawler doing that job to stay away.
Allowing them is no promise either. An open door lets a crawler read your pages. What an assistant then says, and whether it names you, is decided by the company that runs it.
What does blocking the training crawlers change?
It is a decision about training. OpenAI says GPTBot crawls content that may be used to train its models, and that its settings are independent of each other (OpenAI). Anthropic says restricting ClaudeBot excludes a site's future materials from training (Anthropic). Note the word future: read a block as a choice going forward.
There is no wrong answer here. If you are comfortable with your public pages being used for training, leave GPTBot and ClaudeBot alone. If you are not, refuse them by name. Either way, set the search crawlers on their own lines, so one decision does not quietly make the other.
Does blocking Google-Extended take you out of Google?
No. Google says Google-Extended does not affect inclusion in Google Search and is not a ranking signal (Google). It is about Gemini training and grounding, as the list above says, and you can refuse it without affecting whether you are included in Google Search.
Googlebot is a different name with a different job, and a block on it is a different matter. A robots.txt that disallows Googlebot on key URLs is a high severity check in our free scan (T1). When you edit the file, keep the two names apart.
Can robots.txt keep every AI visit out?
No. Two fetchers that act on a person's request may not follow it. OpenAI says ChatGPT-User visits pages on user request, that robots.txt rules may not apply to it, and that it does not decide what appears in search (OpenAI). Perplexity says Perplexity-User fetches pages on user request and generally ignores robots.txt (Perplexity).
And the file was never a lock. Google describes robots.txt as a way to tell crawlers which URLs they may access, mainly to avoid overloading a site. It says the file is not a way to keep a page out of Google, that a blocked page can still be indexed if other pages link to it, and that noindex or a password is the way to do that (Google Search Central). Anything that must stay private belongs behind a password.
Who sets your robots.txt, and how do you check it?
Find out before you change it. Google says site builders often manage the file themselves (Google Search Central), so a line in yours may be one nobody at your business chose, and the place to change it may be a setting in the builder.
The free scan reads your robots.txt first. If the file asks crawlers like ours to stay out, the scan stops there. Otherwise A1 reports a block on OAI-SearchBot, Claude-SearchBot, Claude-User or PerplexityBot as a finding, one of the technical SEO problems it looks for. A block on GPTBot or ClaudeBot (A1T) or on the Google-Extended token (A1G) is a note that adds nothing to the estimate. That one is your call.
If you want the line changed for you, Fix Sprint is up to 12 hours of the fixes your scan found, shipped in 10 business days with before and after evidence. Every change waits for your approval before it goes live on your website.
Reviewing your robots.txt, counted
Here is the time a review of your robots.txt takes, using the names this post covers. The minutes are made-up round numbers for the example, so put in your own.
- AI search crawlers to look for: OAI-SearchBot, Claude-SearchBot, Claude-User, PerplexityBot
- 4
- Names for AI training and Gemini use: GPTBot, ClaudeBot, Google-Extended
- 3
- Googlebot, to make sure it is not refused on key pages
- 1
- Names to check in the file
- 4 + 3 + 1 = 8
- Minutes per name, to find its lines and decide
- 3
- One review
- 8 names at 3 minutes each: 24 minutes
- Reviews a year: once a year, and once after a change of site builder or plugin
- 2
- Total
- 2 reviews at 24 minutes: 48 minutes a year
Under an hour a year. Write your choice beside each of the eight names the first time, so the next review is a comparison and not a fresh decision.
If one of these companies names a new crawler, it adds 3 minutes to a review and 6 minutes to the year.
Round numbers for the arithmetic, not a benchmark and not a forecast. Put in your own.
Questions
Will allowing AI search crawlers get my business named in ChatGPT?
Not by itself. Allowing OAI-SearchBot lets it read your site, and OpenAI says sites that opt out of it are not shown in ChatGPT search answers (OpenAI). Whether an answer names you is up to OpenAI.
Can I allow ChatGPT search and refuse AI training at the same time?
Yes. OpenAI says each setting is independent, so a site can allow OAI-SearchBot and disallow GPTBot (OpenAI). With Anthropic, ClaudeBot is the training crawler, Claude-SearchBot and Claude-User serve search, and each honors robots.txt (Anthropic).
Will blocking Google-Extended hurt how I show up in Google Search?
Google says no: the token does not affect inclusion in Google Search and is not a ranking signal (Google). A block on Googlebot is the one to avoid on pages you want found.
Can I block AI crawlers to keep a page private?
Not reliably. Google says robots.txt is not a way to keep a page out of Google, and points to noindex or a password instead (Google Search Central). Perplexity-User generally ignores the file (Perplexity). Put a private page behind a password.