Before Googlebot, ChatGPT, Claude or Perplexity touch a single page, they fetch
/robots.txt. Twelve lines decide whether you are indexed, cited in AI answers,
or used for training — and most sites get at least one of those wrong.
The same file now governs classic search, AI answer engines and autonomous agents.
Controls crawl budget, keeps faceted and session URLs out of the crawl, and hands search engines your sitemap.xml without a Search Console login.
Generative Engine Optimization starts with access. If an AI search crawler is disallowed, you cannot be cited in its answers — no matter how good the content is.
OAI-SearchBot · Claude-SearchBotAgents fetching a page on a user's behalf check robots.txt first, then look for the files it points to: sitemaps, and increasingly llms.txt.
The major AI companies now run separate crawlers for separate jobs, each with its own user-agent token. You can opt out of training and still be cited in AI answers.
Bulk collection for future models. Blocking them is a policy choice with no direct visibility cost.
Build the index behind AI answers. Block these and you drop out of that engine's citations.
Fetch one URL because a person asked. Providers treat these differently from crawlers — check each one's docs.
| User-agent token | Operator | Role | If you Disallow it |
|---|---|---|---|
GPTBot |
OpenAI | Training | Opts content out of future model training. |
OAI-SearchBot |
OpenAI | AI search | Removes you from ChatGPT search answers. |
ChatGPT-User |
OpenAI | User fetch | Stops fetches a user asked ChatGPT to make. |
ClaudeBot |
Anthropic | Training | Opts content out of model training. |
Claude-SearchBot |
Anthropic | AI search | Removes you from Claude search results. |
Claude-User |
Anthropic | User fetch | Stops fetches a user asked Claude to make. |
PerplexityBot |
Perplexity | AI search | Removes you from Perplexity citations. |
Perplexity-User |
Perplexity | User fetch | Stops user-requested page fetches. |
Google-Extended |
Training | Gemini training opt-out; Search is unaffected. | |
Applebot-Extended |
Apple | Training | Apple AI training opt-out; Siri/Spotlight unaffected. |
Meta-ExternalAgent |
Meta | Training | Opts out of Meta AI training crawls. |
CCBot |
Common Crawl | Training | Keeps you out of the open dataset many models use. |
Each token is independent. Blocking ClaudeBot does not block
Claude-SearchBot, and blocking GPTBot does not block
OAI-SearchBot. A blanket "block all AI" list usually removes you from AI search
by accident.
RFC 9309StandardAfter 28 years as an informal convention, the Robots Exclusion Protocol became an IETF standard in 2022. It fixes the rules parsers must follow: the most specific (longest) matching rule wins, * and $ wildcards are official, crawlers must parse at least 500 KiB, a 4xx response means "no rules, crawl everything", and a 5xx means "assume everything is disallowed".
Sitemap:DiscoveryThe one discovery directive every major crawler reads. It works for any bot, needs no account, and can be listed multiple times. It is how agents find your full URL inventory — make sure the URL returns a valid sitemap.xml, not a redirect or a 404.
Per-purpose user-agentsAI eraOpenAI, Anthropic, Perplexity, Google and Apple publish separate tokens for training, search and user-initiated fetching. This is the single biggest change to how robots.txt should be written — and the main thing most generator templates still miss.
Content-Signal / Content-UsageDraftLines that express how content may be used (search, AI input, AI training) rather than whether it may be crawled. Two spellings exist: Cloudflare's Content-Signal and the IETF AIPREF drafts' Content-Usage, with different labels. Google's parser lists only user-agent, allow, disallow and sitemap as supported fields, and the major AI crawlers do not document honoring either line. Treat them as a declaration of intent, not a control.
Web Bot AuthEmergingUser-agent strings can be spoofed. IETF drafts for cryptographically signed bot requests let a server or CDN verify that "ClaudeBot" really is ClaudeBot. It does not replace robots.txt — it makes the rules in it enforceable at the edge.
User-agent: * # Cloudflare Content Signals spelling (what current validators read) Content-Signal: search=yes, ai-input=yes, ai-train=no # IETF AIPREF draft spelling (different labels, still a draft) # Content-Usage: train-ai=n
If you add a usage line, keep your real decisions in User-agent /
Disallow groups. Those are what crawlers actually obey today.
Indexed by search, citable by AI answer engines, training left as an explicit choice.
# robots.txt — visibility-first starter (Lab451) # 1. Everyone: crawl the site, skip private paths User-agent: * Allow: / Disallow: /admin/ Disallow: /cart/ Disallow: /*?sessionid= # 2. AI search + user-requested fetches — keep allowed to stay citable # A named group REPLACES the * group, so repeat your private paths. User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: Claude-SearchBot User-agent: Claude-User User-agent: PerplexityBot User-agent: Perplexity-User Allow: / Disallow: /admin/ Disallow: /cart/ # 3. Model training — your call. Delete this group to allow training. User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: CCBot User-agent: Meta-ExternalAgent Disallow: / # 4. Discovery Sitemap: https://example.com/sitemap.xml # Agent-readable summary: https://example.com/llms.txt
Replace example.com, then run it through the Checker — or let the generator build it from a crawl of your site.
Disallow: / for "all AI bots" also removes OAI-SearchBot, Claude-SearchBot and PerplexityBot. Block training tokens only.*. A crawler follows only its most specific group. Private paths in User-agent: * don't apply to User-agent: GPTBot unless you repeat them.noindex on a crawlable page instead.Sitemap: line, or one that 404s. The cheapest discovery win available, and the one agents without a Search Console account depend on.robots.txt points the way; sitemap.xml, llms.txt and llms-full.txt are what agents read once they're in.
Paste a domain. The Checker reads the robots.txt you already publish and flags blocked AI search bots, broken sitemap links and group-precedence traps.
Crawl your site and get robots.txt, sitemap.xml, llms.txt and llms-full.txt built from real pages — every file verified for dead links and hidden text.
A deploy or CDN rule can silently block every AI crawler. Monitoring re-checks your files and emails you when something breaks.
Yes. Disallow the training tokens (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot) and leave the search tokens (OAI-SearchBot, Claude-SearchBot, PerplexityBot) allowed. Each is controlled independently.
The documented crawlers from the major AI companies say they do. Compliance is voluntary, though, and user-agent strings can be faked. For hard blocking, enforce at the CDN or firewall; robots.txt states your policy.
There is no official directive for it. A comment line pointing to /llms.txt is harmless and human-readable; what matters is that llms.txt lives at the root and isn't disallowed.
Google ignores it; Bing and some others still read it. It isn't part of RFC 9309. For real load problems, rate-limit at the server.
RFC 9309 asks crawlers not to use a cached copy for more than 24 hours, so most changes are picked up within a day.