New Standard for AI Crawling Controls

Google, OpenAI, Anthropic, and Microsoft have committed to adopting a unified robots.txt directive for AI training data. This standard, proposed by the IETF, allows website operators to use a single user-agent, __usage_ai-training, to manage access for AI model training crawlers across these platforms.

Anthropic has already implemented the directive. Google plans to support it in early 2027, while Microsoft will begin adoption on February 1, 2027.