OpenAI, Google, Microsoft and Anthropic back shared AI opt-out token
Four major AI companies say they will honor a shared 'policy token' in robots.txt that lets publishers apply AI training opt-outs across multiple crawlers.
The story in focus
The Verge reports OpenAI, Google, Microsoft and Anthropic will adopt a shared 'policy token' signal in robots.txt, letting publishers apply AI training preferences across multiple crawlers instead of configuring each company separately. Anthropic's Kevin Kelley proposed it within the IETF; Meta signaled support without committing to honor it.
- OpenAI, Google, Microsoft and Anthropic say they will restrict crawlers from sites carrying the shared token
- The policy token is proposed by Anthropic standards lead Kevin Kelley within the IETF and lives in robots.txt
- Today publishers must signal AI training preferences to each AI company separately
- Meta's Andy Stone voiced support but did not say whether Meta will honor the setting
- The move follows publisher pushback including the RSL licensing standard and Cloudflare blocking AI crawlers by default
What the token does
The Verge reports that OpenAI, Google, Microsoft and Anthropic say they will adopt a shared 'policy token' signal that publishers place in a website's robots.txt file. The four companies say they will restrict their AI crawlers from a site when they encounter the token.
Robots.txt has long told crawlers whether a site may be scraped for search indexing, archiving or AI training. Under current practice, publishers must express AI training preferences to each company individually; the proposed token is intended to convey the same preference to multiple providers at once.
Who proposed it and who has responded
Anthropic technical standards lead Kevin Kelley proposed the token as part of the Internet Engineering Task Force, per The Verge.
Meta communications head Andy Stone signaled support, calling a common signal to multiple providers an important step and saying the company looks forward to working on standards and implementation. The report says Stone did not say whether Meta will honor the setting. The supplied source ends mid-sentence at Microsoft's statement, so the full scope of Microsoft's commitment is not available.
Why publishers are pushing back
The announcement comes as publishers contend with AI crawlers scraping their sites to feed content into models. Some organizations have turned to the Really Simple Licensing standard, which lets publishers charge crawlers that visit their sites, while Cloudflare blocks AI crawlers by default.
Will the shared signal replace separate agreements with each AI company?
It is intended to. The Verge reports publishers currently signal AI training preferences to each company separately, and the token would apply those preferences across multiple crawlers, provided the companies honor it.
Sources & further reading
Help keep the record accurate. Send a correction.
