Back to news
policyOpenAIAnthropicGoogleMicrosoft2026-10-30

OpenAI to respect a generic robots.txt AI training preference signal from the IETF AI Preferences working group

OpenAI announced on October 30 that it will respect a general signal in robots.txt that lets website operators express to multiple participating AI providers at once whether their content may be used for foundational generative AI training.

OpenAI announced on October 30 that it will respect a general signal in robots.txt that lets website operators express to multiple participating AI providers at once whether their content may be used for foundational generative AI training. The company said the signal builds on in-progress work by the IETF AI Preferences working group, will be implemented within the coming months, and does not replace existing per-crawler robots.txt controls. OpenAI said the change lowers the cost for publishers of maintaining separate rules for many crawlers and called on other AI providers to follow. The draft was written by Anthropic engineer Kevin Kelley and deliberately introduces no new directive names or syntax, leaving robots.txt path language, longest-match precedence and group structure untouched, so site owners express preferences by adding a policy token on the User-Agent line, for example writing User-Agent: __usage_ai-training together with ordinary Disallow and Allow to mark paths as disallowed or allowed for AI training, with no token meaning no preference expressed. Because deployed robots.txt parsers behave inconsistently, old parsers ignore the policy token group as an unrecognized user-agent, preserving backward compatibility and avoiding large-scale reconfiguration. Google said it plans to support the signal with other vendors in 2027, and Microsoft's Bing split AI training and AI grounding into two separate preferences, with training preferences refinable per page and grounding allowed by default under a separate DisallowExtended directive.

robots.txtAI 训练偏好IETF内容授权爬虫OpenAI