AI giants promise publishers new way to block some scraping - Inquirer Technology

SAN FRANCISCO — OpenAI, Google, Microsoft and Anthropic pledged Friday to give media outlets and other websites a single way to prevent their content from being scraped to train artificial intelligence (AI) models, amid growing criticism and legal battles involving news publishers.

Publishers currently have to implement multiple blocking commands for every bot that crawls the web to collect data, but the proposed solution would allow them to enter a single, standardized instruction recognized and honored by any AI company that adopts the protocol.

The solution, however, will not take full effect for months.

Microsoft will implement it beginning Feb. 1, 2027, while Google plans to do so sometime in 2027. ChatGPT maker OpenAI said it will roll it out “in the coming months,” while Claude maker Anthropic said it has “already started” rolling it out.

The announcement comes as publishers face a surge in bots that scrape their articles to train AI models. Some, including The New York Times, have taken the matter to court.

In December 2023, the Times sued OpenAI and Microsoft for allegedly using millions of its articles without authorization or payment to train ChatGPT. The two tech companies have invoked an exception to copyright law known as fair use.

Meanwhile, internet infrastructure provider Cloudflare has blocked AI bots by default on new customer sites since July 2025, while a standard called Really Simple Licensing already allows publishers to charge bots that visit their pages.

The new setting will apply only to the future training of models. It will not otherwise prevent bots from crawling websites, and content scraping will be allowed by default until a publisher changes the setting, Microsoft said in a blog post.

Opting out of AI training could reduce the chances that a publisher’s content will surface in AI apps and experiences, Microsoft and Google said in separate blog posts.

“We believe creating a common signal that conveys such preferences to multiple providers at once is an important step. And we look forward to working with other AI companies on the standards and their implementation across the industry,” Meta spokesperson Andy Stone wrote on X, expressing support for the effort. /dm