A single robots.txt line could soon let site owners tell multiple AI vendors whether their content may be used for model training. Google announced it will support a shared AI‑training control — a “policy token” defined by the IETF AIPREF working group — and says it aims to adopt the mechanism with other providers in 2027.

Why this matters now

Publishers and site operators currently juggle provider‑specific settings when they want to allow or block AI training on their content. The new shared token promises a simpler, machine‑readable signal that multiple AI developers can respect at once. For organizations that care about content licensing, brand safety, or regulatory compliance, that can significantly reduce operational friction.

How the new control works

Instead of adding separate instructions for each AI developer, a website can place a shared policy token inside robots.txt. When a publisher uses the token to opt out, Google says it will treat that as if the site had disallowed Google‑Extended unless Google‑Extended is explicitly permitted. Other AI providers can adopt the same interpretation, enabling a single robots.txt change to carry cross‑provider weight.

Google stresses this shared token does not replace vendor‑specific controls. Existing company controls such as Google‑Extended will keep working alongside the industry signal. That lets publishers choose a two‑tier approach: a broad industry preference via the shared token and granular, provider‑level exceptions where needed.

What publishers and marketers should do next

There is no immediate technical change to roll out from Google beyond the announcement. Still, treat this as actionable notice and undertake three steps now:

  • Audit your signals. Inventory current robots.txt rules and any provider‑specific directives you already publish (for example, Google‑Extended) so you know the baseline behaviour.
  • Set a policy stance. Decide whether your organisation prefers a broad industry opt‑out, selective provider opt‑outs, or a mixed approach — and align product, legal and publishing teams accordingly.
  • Track standardisation and adoption. The token is based on an IETF working group draft (AIPREF). Adoption depends on other providers and a finalized spec, so watch for IETF updates and vendor implementation guides before making large‑scale changes.

Limits and dependencies

This announcement defines Google’s intended behaviour, not an industry‑wide outcome. The shared token delivers practical cross‑provider effect only if other AI providers implement compatible interpretations. The IETF effort supplies the technical vocabulary and the working definition of “AI training” used by the mechanism, but multistakeholder discussions on the broader AIPREF vocabulary are ongoing.

What to watch

Key near‑term signals that will determine how useful the token becomes: a formal IETF AIPREF specification, explicit adoption statements or implementation guides from other major AI model providers, and published example syntax site operators can safely add to robots.txt. Those developments will tell publishers whether a single robots.txt token can genuinely manage AI‑training preferences at scale, and how to combine it with provider‑specific permissions for product features like grounding in Gemini Apps.

For now, treat Google’s timeline — aiming to support the mechanism with other AI providers in 2027 — as a planning horizon rather than an immediate change to content workflows. Take inventory, set a clear policy, and be ready to adapt once the IETF draft and vendor implementations provide definitive, publishable syntax.