News

One line for AI training: the shared robots.txt group Anthropic, Microsoft, Google and OpenAI back

November 12, 2026 7 minute read

Until now, opting out of AI training meant naming crawlers one at a time: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, and whoever launched last month. On October 30, four of the largest AI providers backed a single robots.txt group instead. It's a real step forward, and it comes with two catches that can quietly undo it.

What was announced

Anthropic submitted an Internet-Draft to the IETF, Machine-Readable Training Preference Signal, written by Kevin Kelley. The same day, Google, Microsoft and OpenAI published posts saying they would read it. The draft reserves one name for the User-agent line of an ordinary robots.txt group:

# Don't use this site to train generative AI models
User-agent: __usage_ai-training
Disallow: /

No new directive and no new file: the usual Allow and Disallow lines, the same longest-match rules from RFC 9309. Any provider that honours the draft reads that group as your preference about training, so you no longer need to know every training crawler by name.

Who said what, and when

Provider	Position	When
Microsoft (Bing)	Will honour it, along with new page-level training controls in a meta tag or X-Robots-Tag header	From Feb 1, 2027
Google	Will read an opt-out as a Google-Extended opt-out where you haven't set one (see the catch below)	Early 2027
Anthropic	Has started implementing support; the draft's author works there	No date given
OpenAI	Will implement it as a "voluntary effort"; existing GPTBot rules keep working	"Coming months"
Meta, Amazon, Apple	Not among the commitments	—

That's an announcement table, not a compliance report. Nobody has switched it on at scale yet, and the dates are the providers' own.

What the line does, and what it doesn't

The draft is short, and it's precise about its limits. Four of them matter for anyone editing a robots.txt:

It's about use, not crawling. The draft says outright that the token "is not a directive for crawling behavior". Crawlers can still fetch every page; the group tells them what they may do with it afterwards. Your AI search and answer agents (OAI-SearchBot, Claude-SearchBot, PerplexityBot) are untouched, so you stay citable.
User-agent: * doesn't count. A wildcard group "MUST NOT" be read as matching the token. If your file has no group naming __usage_ai-training, it states no training preference at all, whatever the * group says. A blanket Disallow: / written years ago says nothing here.
It looks forward only. The preference that applies to a page is the one in robots.txt when that page was fetched. Nothing reaches back to earlier crawls or models already trained, and the draft lets private licensing agreements override it.
It isn't a standard yet. It's an individual draft with no IETF endorsement, and it lapses on May 3, 2027 unless revised. Even the definition of "training" isn't settled: OpenAI's wording refers to "foundational" models, which is narrower than the draft's.

Spelling matters too. Providers must match the token exactly (case aside): two underscores, then usage_ai-training. User-agent: usage-ai-training reads as an unknown crawler and states nothing.

Catch one: your old per-crawler groups can cancel it

The draft says the more specific expression should win, and providers may treat their own crawler names as more specific than the shared token. So this file opts out of training for every participating provider except OpenAI:

User-agent: __usage_ai-training
Disallow: /

# Written years ago, never revisited
User-agent: GPTBot
Allow: /

Plenty of robots.txt files, including generated ones, give training crawlers an explicit Allow: /. Adding the shared group on top doesn't override that. Make the named training groups say the same thing, or remove them.

Catch two: Google ties training to Gemini citations

Google will treat a shared opt-out as a Google-Extended disallow when your file has no Google-Extended group. But Google-Extended governs more than training: it also decides whether Gemini Apps and Vertex AI may use your pages to ground answers. Opt out through the shared token and you can lose Gemini citations as a side effect.

Google offers no separate grounding token. If Gemini citations matter more to you than the training opt-out with Google, add an explicit Google-Extended group with Allow: /. Google follows that instead of the shared group. AI Overviews and AI Mode in Search are unaffected either way; they follow Googlebot.

Where it fits in a visibility-first robots.txt

In our robots.txt guide for SEO and GEO we split AI bots into three jobs: training, AI search, and fetching a page because a person asked. The shared group slots into the training job and leaves the other two alone. The training block from that guide becomes:

# Opt out of model training
User-agent: __usage_ai-training   # every participating provider (IETF draft)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

Keeping the named crawlers in the same group means the shared signal and the per-crawler rules can't disagree, and crawlers that don't read the draft yet still get a clear answer. Want some sections left open, say a press kit you'd like quoted accurately? Add Allow: /press/ to the group and it applies to both.

Should you add it?

If you already refuse training crawlers one by one, yes: the shared group says the same thing to providers you haven't listed, in one place. If you want maximum reach in AI answers, leaving it out is a valid choice; no group means no preference, which is how things worked before October 30. What matters is choosing on purpose, and checking the Google catch before you publish.

What's new in Lab451

We've added the shared group across the generator, the Checker and Monitoring:

Generator. When you generate robots.txt, a new AI training preference box offers three options: no preference (the default, writes nothing), opt out for the whole site, or opt out except sections you pick from a dropdown (/blog/, /press/, /docs/ and others, or a custom path). Lab451 writes the shared group and sets the named training crawlers to match, so catch one can't happen in a file we generate.
Checker. Reads the shared group in the robots.txt you publish and reports your current preference. It flags a misspelled token, per-crawler rules that contradict it, and the Google-Extended side effect.
Monitoring. Because the preference that counts is the one published when a page is fetched, a plugin or CDN setting that quietly rewrites it matters on the day it happens. Pro and Agency members now get an email the moment it changes, not at the end of the week.
Reference. A plain-language explainer of the token, kept current as providers ship, and a step-by-step how-to in the docs.

If a later revision of the draft renames the token, we'll update all of these. That's a one-line change on our side, by design.

See what your robots.txt says about training

Run your domain through the Checker to see whether you state a training preference today and whether anything contradicts it. Or generate a robots.txt with the new option, alongside your sitemap.xml, llms.txt and llms-full.txt.