OpenAI, Google, Microsoft and Anthropic back a generic robots.txt signal expressing AI training preferences
OpenAI, Google, Microsoft and Anthropic announced they will adopt a new standard letting content publishers refuse AI crawling of their sites in one step rather than setting rules per company.
OpenAI, Google, Microsoft and Anthropic announced they will adopt a new standard letting content publishers refuse AI crawling of their sites in one step rather than setting rules per company. The standard was proposed to the Internet Engineering Task Force by Anthropic technical lead Kevin Kelley and will be integrated into the robots.txt file. robots.txt has long told crawlers which content may or may not be fetched, for search indexing, web archiving and AI model training alike, and sites previously had to declare AI training preferences separately to each AI company. OpenAI plans to implement within the coming months, Google and Microsoft are expected next year, and Meta spokesperson Andy Stone said the company supports a universal signal without committing to a timeline. The work stems from the IETF AI Preferences group's ongoing effort, whose signal is expressed by writing a policy token on the User-Agent line so old parsers ignore it as an unrecognized user-agent, preserving backward compatibility.