🔍 Read the full analysis: How OpenAI, Google, Microsoft And Anthropic’s Opt-Out Works on ThorstenMeyerAI.com
Get audio and creator gear delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
OpenAI, Google, Microsoft and Anthropic have said they will support a shared robots.txt signal for publishers who do not want their content used to train generative AI models. The signal is a voluntary request, not a technical lock, and does not cover every use of web content in AI products. Rollout dates differ, while Meta has voiced support without committing to honor it.
OpenAI, Google, Microsoft and Anthropic said on October 30 that they will recognize a shared robots.txt signal asking them not to use a website’s content to train generative AI models. The common rule could reduce the need for publishers to maintain separate training opt-outs for each company, but it is a voluntary request, not a block, and companies have set different implementation timelines.
The proposed rule uses the user-agent name __usage_ai-training with Disallow: / in a site’s robots.txt file. It is described in an individual Internet-Draft titled “Machine-Readable Training Preference Signal,” written by Anthropic standards lead Kevin Kelley and published on October 30. The draft is associated with the Internet Engineering Task Force’s AI Preferences working group, but it is not an adopted internet standard.
The companies’ plans are not identical. Anthropic says it has started implementing support, while noting that support will take time to reach its services. OpenAI says it aims to implement the signal “in the coming months.” Google says it aims to support it in 2027; its crawler documentation gives an “early 2027” target. Microsoft will honor the training signal from February 1, 2027, as it crawls sites.
Microsoft is also introducing a separate page-level preference for AI Grounding—the use of web content to inform AI answers in Bing, Copilot and Microsoft 365. Microsoft says both preferences take effect February 1, 2027. Its existing NoArchive and NoCache tags will stop counting as preferences for training or grounding after March 1, 2027. Existing controls such as GPTBot, Google-Extended and ClaudeBot remain available during the rollout.
Infographic 01 · Who backs it, and when
One line in robots.txt, four companies, four timelines.
A shared signal against AI training. Only Anthropic says it has started; the others name dates or months. The standard itself is still a draft.
- 30 Oct 2026Draft published, four companies announce supportdoneIndividual Internet-Draft by Kevin Kelley (Anthropic)
- 30 Oct 2026Anthropic: “has already started implementing support”startedIts standards lead on the IETF list: “It’ll take us some time to add support across our services.”
- Coming monthsOpenAI aims to implement itplanExisting GPTBot and OAI-SearchBot rules keep applying
- 16 + 19 Nov 2026Working group meets at IETF 127, San FranciscoopenOn 21 Oct a chair said no consensus has been declared yet
- 1 Feb 2027Microsoft honors the new preferences as it crawlsplanTraining in robots.txt or per page; grounding per page
- 1 Mar 2027Bing stops reading NoArchive / NoCache as AI preferencesdeadlinePublish the new settings before this date
- 2027Google plans support; its docs say “early 2027”planA “no” maps to Google-Extended unless you set that token yourself
- 3 May 2027Draft version 00 expires unless updatedInternet-Drafts are valid for at most six months
Binding? No, voluntary
- OpenAI calls it a “voluntary effort”.
- The vocabulary draft does not “ensure that preferences are followed”.
- The draft “has no formal standing in the IETF standards process”.
- Private agreements may override it, the draft says.
Who else?
- Meta supports a common signal, but did not say whether it will honor it.
- Apple, Amazon, xAI, Perplexity: no statement in the sources reviewed for this article.
- The draft’s acknowledgments list contributors “for identification only”, not supporters.
Sources: IETF draft, 30 Oct 2026 · Anthropic on the IETF list · AIPREF minutes, 21 Oct 2026 · Google, 30 Oct 2026 · Google crawler docs · OpenAI, 30 Oct 2026 · Microsoft Bing, Oct 2026 · Meta (Andy Stone on X). Company plans are company statements.
Powered by Thorsten Meyer AI
One Rule, Several AI Uses
For publishers, a shared signal could make it simpler to state a preference about training across participating services. Until support is in place, however, the existing company-specific controls still matter: a site that relies only on the new line may not have its request recognized immediately by every provider. Publishers will need to check each company’s documentation and keep relevant older rules where appropriate.
The boundary between model training and AI answers also matters. The shared rule is described as a preference against training, not a general ban on crawling or using pages to answer user questions. Microsoft’s separate grounding control makes that distinction explicit for its products. The signal does not establish a payment arrangement, guarantee traffic, or determine whether a page appears in search results. Microsoft says neither preference guarantees “inclusion, traffic, or payment.”
For readers and site owners, this is a change to instructions that websites give automated crawlers—not a setting for an individual’s conversations with an AI assistant. A publisher’s choice also cannot reverse the use of material in models that have already been trained. The practical effect depends on how companies interpret and implement the request.
Products Worth Considering
Samsung 27 Odyssey OLED G5 (G50SF) QHD QD-OLED Gaming Monitor
Samsung 27-inch Odyssey OLED G6 QD-OLED QHD Gaming Monitor
LG 34GX90SA-W 34-inch Ultragear WQHD OLED Curved Gaming Monitor
As an affiliate, we earn on qualifying purchases.
From Separate Crawler Rules
Before this proposal, publishers generally needed different instructions for different services. OpenAI identifies GPTBot; Google uses Google-Extended for certain AI-related uses; Anthropic lists ClaudeBot. Microsoft’s existing NoArchive and NoCache page tags are being replaced for training and grounding preferences on its stated schedule.
The shared signal grew out of work in the IETF’s AI Preferences working group. The draft is an individual contribution, not a consensus standard. At a working-group meeting on October 21, one of the chairs said, “We haven’t declared consensus on anything yet.” That distinction matters: company announcements describe their own intentions, while the draft itself does not oblige other services to comply.
Meta spokesperson Andy Stone said the company “strongly support[s] publishers’ ability to express preferences” and called a common signal “an important step.” That statement did not include a commitment that Meta would honor this specific rule. The source report also said it found no statements from Apple, Amazon, xAI or Perplexity about adopting it.
“Google aims to support the new control “along with other AI providers in 2027.””
Adoption and Enforcement Remain Open
The signal’s reach depends on which services implement it, when they do so, and how consistently they follow it. Anthropic’s announcement says implementation has begun, but its crawler help page still described only per-crawler controls on October 31. Google and OpenAI have stated targets rather than completed rollouts. Meta has expressed support for publishers’ ability to state preferences but has not said it will honor this signal.
The request is not a technical lock: robots.txt communicates a site’s preference, but does not itself prevent a crawler from accessing pages. The draft also has no formal standing in the IETF standards process, and the working group has not declared consensus. It is not yet clear whether additional providers will adopt it, how they will document compliance, or whether a shared signal will receive formal standards status.
There is also a scope limit. The shared rule concerns training use; it does not, on the available announcements, cover live retrieval or grounding for AI-generated answers across all services. Microsoft has announced a separate control for its own grounding use. The companies have not announced that this signal will remove content from models already trained.
Rollout Dates and Publisher Steps
Publishers can add the shared user-agent group and Disallow: / to robots.txt if they want to express a preference against training, while retaining company-specific rules until providers confirm support. They should review each provider’s documentation rather than assume that the new line already replaces every existing control.
Microsoft’s stated date is February 1, 2027 for the training and AI Grounding preferences, followed by the end of NoArchive and NoCache as training or grounding preferences after March 1. Google targets early 2027, OpenAI says “the coming months,” and Anthropic says implementation is underway. Those dates and descriptions may change as the companies extend support.
The next milestones are actual deployments, updated crawler guidance and any further decisions by the IETF working group. Until those appear, the shared rule is best understood as a developing, voluntary way for publishers to express a preference—not a universal prohibition on AI companies using web content.
Key Questions
What does the shared robots.txt signal say?
It uses User-Agent: __usage_ai-training followed by Disallow: / to express a site’s preference that its content not be used for generative AI training.
Does the signal block AI companies from crawling a site?
No. It is a robots.txt request, not a technical barrier. The proposal concerns training preferences and does not itself prevent access to pages.
Does opting out stop AI answers from using a page?
Not necessarily. The shared signal is for training, while use of web content to ground AI answers is a separate issue. Microsoft has announced a separate AI Grounding preference for its products.
When will the companies support it?
Anthropic says implementation has started; OpenAI aims for the coming months; Google targets 2027, with its documentation saying early 2027; and Microsoft says February 1, 2027. The timelines are company plans, not evidence that every service already supports the rule.
Has Meta agreed to honor the signal?
Meta has voiced support for publishers expressing preferences, but the cited statement did not commit the company to honoring this specific signal.
Source: ThorstenMeyerAI.com
