OpenAI, Google, Microsoft, and Anthropic Agree on New “Policy Token” to Let Publishers Opt Out of AI Scraping

Summary

Major AI developers – OpenAI, Google, Microsoft and Anthropic – have announced a joint effort to create a standard “policy token” that publishers can place in their robots.txt file to signal that their content should not be used for AI training. The proposal, drafted by Anthropic’s standards lead Kevin Kelley under the IETF, aims to replace the current practice of sending separate opt‑out requests to each company.

How the token works

• The token is a simple line added to robots.txt (e.g., `AI-Training: no`).
• All participating AI crawlers will read the file before fetching a page.
• If the token is present, the crawler must refrain from using the page for model training or grounding.
• The signal does not affect a site’s visibility in search results on Bing or Google.

Companies’ commitments

• Anthropic has already begun honoring the token.
• Google and Microsoft say they will support it starting next year.
• OpenAI plans to roll out support within the “coming months.”
• Meta expressed support publicly, but has not confirmed implementation.

Why it matters to publishers

• Single‑point control: No longer need to send separate emails or configure multiple dashboards.
• Legal clarity: Provides a documented, industry‑wide signal that can be referenced in copyright or licensing disputes.
• Potential revenue: The token could be combined with the Really Simple Licensing (RSL) model, allowing sites to charge for legitimate crawling.

Practical steps to adopt the token

1. Edit your site’s robots.txt file.
2. Add the line `AI-Training: no` (or the exact syntax published by the IETF draft).
3. Verify the file is publicly reachable (e.g., `https://example.com/robots.txt`)
4. Monitor crawler logs for compliance; most major AI providers now publish compliance reports.

What remains uncertain

• Meta’s participation: The company has only signaled support; no timeline is given.
• Enforcement: The standard relies on voluntary compliance; there is no technical block to ignore the token.
• Granular controls: Microsoft mentions future options to allow “grounding” while blocking training, but details are still under development.

How to verify the announcement

• Check the IETF draft repository for the official “policy token” specification.
• Look for blog posts or press releases on the corporate sites of OpenAI, Google, Microsoft, and Anthropic.
• Follow updates from the Search Engine Roundtable and reputable tech news outlets.

--- *The information above reflects statements made by the companies as of the latest public announcements and does not constitute a guarantee of future implementation.*