Will unified robots.txt signals standardise AI training opt-outs?

OpenAI, Google, Microsoft and Anthropic announced they will support a new standard allowing publishers to opt out of AI scraping, as reported by The Verge AI.reported This shared mechanism aims to simplify how websites communicate their data preferences to multiple AI companies simultaneously.

The claim

The proposed standard introduces a shared policy token within the robots.txt file, a system long used to signal web crawler access rules. Kevin Kelley, a technical lead for standards at Anthropic, proposed this signal as part of the Internet Engineering Task Force. According to Search Engine Roundtable, this approach allows publishers to apply AI training preferences across multiple crawlers rather than setting them individually for each company.reported

The response

Anthropic has already adopted the signal. Google and Microsoft plan to support it next year, while OpenAI says it will implement the change in the coming months. Microsoft notes it will offer more granular controls, giving publishers the ability to opt out of AI training or AI grounding separately. Microsoft stated that websites will still appear in Bing search results regardless of these preferences.

The bottom line for teams

The move comes as publishers grapple with the influx of AI crawlers scraping their sites. Some organisations have attempted to combat unauthorised scraping through other means. The Really Simple Licensing standard gives publishers the ability to charge crawlers, while Cloudflare blocks AI crawlers by default. The new shared token offers a unified alternative to these fragmented approaches.

What the paper shows

This development tests whether a single technical signal can replace individual company-specific opt-out mechanisms. The reported results indicate broad industry backing from four major firms, though implementation is staggered.reported The stated limitation is that support is not yet universal; Anthropic is current, while others are future-dated. The material does not confirm if this standard will become mandatory or if it will fully replace existing methods like RSL or Cloudflare blocks. It remains to be seen if this unified approach will effectively reduce the administrative burden on publishers or if fragmentation will persist through varying interpretations of granular controls.