Google, Microsoft, And OpenAI Plan To Honor Shared AI Training Opt-Out
Google, Microsoft, and OpenAI will honor a shared robots.txt line for AI training preferences, allowing sites to state their training preferences to multiple providers at once. This move aims to simplify the process of opting out of AI training for websites.
Original reporting · searchenginejournal.com
Original: “Google, Microsoft & OpenAI Plan To Honor Shared AI Training Opt-Out via @sejournal, @MattGSouthern”
Read “Google, Microsoft & OpenAI Plan To Honor Shared AI Training Opt-Out via @sejournal,…” at Search Engine Journal →
How this was written. Awesome B2B wrote this article with a language model from Search Engine Journal’s reporting (model: @cf/meta/llama-3.3-70b-instruct-fp8-fast); it describes that reporting rather than adding to it. The original, linked above, is the full story.
Key points from Search Engine Journal’s report
- Google, Microsoft, and OpenAI will honor a shared robots.txt line for AI training preferences, making it easier for sites to manage their settings.
- The shared line is based on an Internet Draft submitted by Anthropic's Kevin Kelley to the AI Preferences working group at the IETF.
- Websites can use a specific token to specify their AI training preferences, giving them more control over how their content is used.
- Google and Microsoft will handle the shared line differently, with Google treating it as a Google-Extended block and Microsoft introducing a separate AI Grounding preference.
- OpenAI aims to honor the shared line in the next few months, although its current robots.txt policy will still be in effect.
Google, Microsoft, and OpenAI announced plans to recognize a shared line in robots.txt files that indicates a site's AI training preferences. This development is expected to make it easier for websites to manage their AI training settings.
The shared line is based on an Internet Draft submitted by Anthropic's Kevin Kelley to the AI Preferences working group at the IETF. Although this draft does not represent an official IETF standard, it has prompted the three companies to take action.
According to the draft, websites can use a specific token to specify their AI training preferences. This token can be used to allow or disallow AI training for certain parts of a website, giving site owners more control over how their content is used.
Google plans to treat the shared opt-out line as a Google-Extended block when a website does not specify Google-Extended. This means that sites opting out of AI training will be treated similarly to those that have explicitly disallowed Google-Extended.
Microsoft, on the other hand, will handle the shared line differently. The company will use the line to determine AI training preferences, but it will also introduce a separate AI Grounding preference that governs how content can be used to inform AI-generated answers.
OpenAI aims to honor the shared line in the next few months, although its current robots.txt policy will still be in effect. The company has not yet provided details on how it will handle conflicting rules or preferences.
The introduction of the shared line is expected to simplify the process of managing AI training preferences for websites. However, it may not completely replace existing controls, and sites may still need to use company-specific rules or settings to achieve their desired level of control. This article is based on reporting by Search Engine Journal.
Questions and answers
What is the purpose of the shared robots.txt line?
The shared line allows sites to state their AI training preferences to multiple providers at once, simplifying the process of opting out of AI training.
How will Google handle the shared opt-out line?
Google will treat the shared opt-out line as a Google-Extended block when a website does not specify Google-Extended.
What is the difference between AI training and AI grounding?
AI training refers to the use of content to modify a generative AI model's learned parameters, while AI grounding refers to the use of content to inform AI-generated answers.
