If your robots.txt ends with User-agent: * and Disallow: /, you have not opted out of AI training under the signal Anthropic, OpenAI, Google and Microsoft said on October 30 they will honor. The Internet-Draft that defines it says wildcard groups “MUST NOT be interpreted as matching the policy token,” and a file without the new group means “no preference has been expressed.”
That is the first thing to know about __usage_ai-training. The second is that it covers training only, and the four companies draw the line between training and everything else in different places.
What each company actually committed to
The token comes from a draft written by Kevin Kelley of Anthropic: version 00, dated October 30, 2026, expiring May 3, 2027. It is an individual submission with no formal IETF standing, although its acknowledgments list contributors from Google, Microsoft, OpenAI, Meta, Cloudflare and several publishers. It borrows its definition from the IETF AIPREF working group’s vocabulary: training means “Using an asset to modify the learned parameters of a generative AI model.” The syntax is an ordinary robots.txt group with a reserved name:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
The commitments behind it are uneven:
- Anthropic wrote the draft. Kelley told the IETF mailing list that Anthropic “has already started implementing support,” that “It’ll take us some time to add support across our services,” and that the work “complements our existing crawler controls rather than replacing them.” He gave no date.
- OpenAI says it aims to implement the signal “in the coming months” and that controls like GPTBot keep working, per The Next Web’s summary of its announcement.
- Google: John Mueller wrote, “We aim to support this new control along with other AI providers in 2027, and will provide updates on our progress.” Its crawler documentation says early 2027.
- Microsoft gave the firmest date. Bing starts honoring the preference on February 1, 2027, and from then on it replaces the NoArchive and NoCache tags Bing has used for these purposes.
So the one company that may act on the token today is the one that wrote it, and its rollout is gradual. For at least the next quarter, the vendor-specific crawler rules are still doing the real work.
Training and grounding are split differently at Google and Microsoft
Training changes model weights. Much of the AI traffic publishers care about now is grounding: a model reading your page when it answers a question, and sometimes citing it (occasionally for claims the page never made, as the deep research citation problem showed). The draft does not mention grounding at all.
Microsoft built a separate grounding control, and it works only at the page level, not in robots.txt. It has three values: Allow (the default), Disallow, and DisallowExtended. DisallowExtended keeps Microsoft’s own products such as Copilot and Microsoft 365 but shuts out grounding services Microsoft sells to other companies, including Web IQ and Foundry API grounding. It is set with <meta name="Bingbot" content="ai-grounding:disallowextended"> or the equivalent X-Robots-Tag header. Krishna Madhavan’s post argues that “Publishers should not have to choose between being discoverable in Bing and deciding how generative models use their work,” and also warns that the two restrictive settings “will significantly reduce your content’s visibility in AI-generated experiences.” Neither setting, Bing adds, will “establish licensing or payment arrangements.”
Google ties the two together. Its crawler page says Google-Extended governs both training and grounding for Gemini Apps and Grounding with Google Search on Vertex AI, and that “Google-Extended does not impact a site’s inclusion in Google Search.” Mueller’s post says that when a site opts out through the shared token and does not mention Google-Extended, Google will treat that as a Google-Extended disallow. It also says: “Websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps.”
Read those together. Once Google supports the token, a site that adds only the shared group will also drop out of Gemini Apps grounding, and the clicks that come with it. Google’s documentation describes no Google-Extended setting that blocks training while keeping grounding. Neither page mentions AI Overviews or AI Mode, so neither control is documented as reaching them.
OpenAI and Anthropic make the split by crawler name instead. At OpenAI, GPTBot handles training, OAI-SearchBot handles search, and ChatGPT-User fetches pages on user request, where “robots.txt rules may not apply.” Anthropic’s three crawlers follow the same pattern: ClaudeBot, Claude-SearchBot and Claude-User. The shared token governs none of the search or user-initiated agents.
A robots.txt you can ship this week
Keep the rules that work now, add the shared group so it takes effect as each vendor turns it on, and make the Google-Extended choice deliberately:
# Shared training preference (honored as each vendor implements it)
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
# Vendor training crawlers that are honored today
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
# At Google this also removes Gemini Apps grounding. Allow it if you want that traffic.
User-agent: Google-Extended
Disallow: /
Three limits apply regardless of what you write. First, the token is not retroactive: it says nothing about content already in a dataset or a trained model. Second, it is a preference, not an access control. The draft calls usage preferences and traffic management “independent and orthogonal,” and private licensing deals between a site and an AI company override it. If you need enforcement, that is a job for your CDN or WAF, the layer where Cloudflare switched agent-category crawlers to blocked by default in September. Third, the dates matter. Sites relying on NoArchive or NoCache to keep content out of Bing grounding have until March 1, 2027 to publish new preferences, or to record a temporary one in Bing Webmaster Tools, which opens December 15.
Google presents the signal as fitting the EU AI Act’s general-purpose AI Code of Practice, which recognizes robots.txt as a main machine-readable way to state an opt-out. That makes it the second EU-facing AI control due to arrive this winter, after the machine-readable marking rules that apply from December 2. With both, a visible statement on the page is not enough. The preference has to be in a form a crawler can read, and under this draft a crawler reads only the group you name exactly.