Publishers have spent years maintaining a messy list of AI crawler names in their robots.txt files. Four of the biggest AI companies now say they’ll stop making them do that. Google, Anthropic, OpenAI and Microsoft have committed to honoring a single robots.txt directive that tells them not to use a site’s content for generative AI training.
The directive comes from a proposal under consideration at the IETF, the internet’s open standards body. It was written by Kevin Kelley, Anthropic’s technical lead for standards. Today, site owners have to identify each provider’s crawler and write separate rules for it. Miss one, or let a new bot appear, and your content can end up in a training set. The new approach replaces that with one common signal. Publishers add a user agent called __usage_ai-training, then use Disallow to block everything and Allow to carve out exceptions, such as a /press-kit/ folder.
The scope is narrow, and that matters. This is not a block on crawling in general, and it won’t remove a site from Google Search. Google says it will read the signal as a stop sign for Google-Extended, the user agent that gathers material for training Gemini models. Googlebot, which feeds search, is untouched. That split has been a sore point for publishers who want out of AI training but can’t afford to vanish from search results.
The bigger limitation is time. The directive can’t claw back anything already scraped. If a page was crawled last year, the content is already in the pipeline. It only governs future collection. So for many publishers this is a forward-looking fix, not a remedy for past grievances, and it does nothing for the lawsuits and licensing fights still playing out.
Rollout is also uneven. Anthropic already supports it. Microsoft says it will start on February 1, 2027, Google plans early 2027, and OpenAI only promised the coming months. Until all four are live, publishers will probably keep their existing per-crawler rules as a backup.
Does this matter? Yes, mostly for the plumbing. One standard cuts the admin work and gives smaller sites the same control as large ones. But robots.txt has always been a voluntary honor system, and it only works on companies that choose to comply. The notable gaps are Meta, xAI, Perplexity and the many smaller scrapers that have a history of ignoring it. Watch whether they sign on. If they don’t, this is a useful courtesy among the major players and not a real industry standard.
