AI News · Good news ·
Google, Microsoft, OpenAI and Anthropic back a one-line AI training opt-out
Google, Microsoft, Anthropic and OpenAI plan to support a new robots.txt standard that lets website operators block all crawlers collecting generative AI training data with a single entry. The companies have submitted a draft to the Internet Engineering Task Force, which maintains the Robots Exclusion Protocol.
Key points
- Four major AI developers plan to support a single robots.txt AI training opt-out.
- A draft has been submitted to the Internet Engineering Task Force.
- OpenAI and Microsoft plan support in coming months, Google sometime next year.
What happened: Google, Microsoft, Anthropic and OpenAI announced plans to support an upcoming website crawling standard that would let operators stop AI developers from using their content to train models, SiliconANGLE reported. The companies have submitted a draft to the Internet Engineering Task Force, the nonprofit that maintains the Robots Exclusion Protocol behind robots.txt files.
The details: Today, a site that wants to block training scrapers has to add a separate robots.txt entry for every AI provider, and for each individual scraper that a provider runs. The proposed standard would block all scrapers collecting training data for generative AI models with one entry. It would not apply to models that perform exclusively non-generative tasks. Microsoft will also offer a setting called AI Grounding, which lets operators stop Microsoft services from fetching their pages to answer user questions, including third-party AI tools built on Microsoft technology. OpenAI plans to offer similar controls.
Who it affects: Publishers, brands and any company with a public website that wants a say in how its content is used by generative AI, without tracking a growing list of individual AI crawlers.
What to watch: OpenAI and Microsoft intend to add support in the coming months, while Google will follow sometime next year. Meta has expressed support for the idea but has not committed to implementing it. The name of the standard and a firm release date have not been disclosed.
Our take
Content and marketing teams will get a simpler way to control whether their sites feed AI training, which may also shape what future models learn from.