ITmedia Reports OpenAI, Google and Microsoft Will Honor One Shared robots.txt Signal That Lets Sites Refuse AI Training
AI学習への利用拒否をrobots.txtで一括指定へ OpenAIやGoogleなどが対応表明
Context
Website owners could refuse AI training by every major lab with one line instead of tracking each company's crawler, which changes how publishers control their content.
Until now, a site that wanted to keep its content out of AI training had to name each company's crawler separately in robots.txt, the file that tells crawlers what they may access. The new signal comes from an Internet Draft that Anthropic's standards lead Kevin Kelly submitted to the IETF, based on work in its AI Preferences (AIPREF) working group. ITmedia NEWS is a major Japanese technology news site.
Deep dive
ITmedia NEWS reported that OpenAI, Google and Microsoft each announced on Oct 30, US time, that they will support a common way for website owners to tell several AI companies at once whether their content may be used for AI training. According to the report, Anthropic is also implementing it.
The mechanism is proposed in an Internet Draft titled Machine Readable Training Preference Signal, submitted to the IETF by Anthropic standards lead Kevin Kelly. Site owners add a shared policy token as a User Agent line in robots.txt. Under it, Disallow signals that content should not be used for training and Allow signals that it may be. ITmedia says the signal is separate from crawl control, so it does not stop crawling, and sites that set nothing are treated as expressing no preference. The draft defines training as using content to change a generative model's parameters, lets companies give their own controls priority, and has no formal IETF standing yet because it was filed in a personal capacity.
The companies' timelines differ, according to ITmedia. OpenAI aims to support it within the next few months. Google aims for 2027 and will treat a site that refuses training through the shared signal, without explicitly setting Google Extended, as having refused Google Extended too. Microsoft says it will start reading the new settings on Feb 1, 2027, and will also let pages set training preferences in meta tags or HTTP headers. Kelly reported on the AIPREF mailing list that Anthropic has begun implementing the draft, though rolling it out across its services will take time.
What to expect next
Watch for the draft to move through the IETF AIPREF working group and for each company to publish implementation details on the dates ITmedia reports. Publishers will want to know whether other AI companies and smaller crawlers honor the signal, since it is a voluntary preference rather than a block. Another open question is how it interacts with lawsuits over AI training and with the EU AI Act code of practice, which Google says recognizes robots.txt as a machine readable opt out.