Glossary · Crawlers and files

Google-Extended

Definition

Google-Extended is a robots.txt token that lets publishers control whether content Google crawls from their site may be used to train future Gemini models and to ground answers in Gemini Apps and Vertex AI. It does not affect Google Search.

What it means

Google-Extended is not a separate crawler. Google crawls with its existing user agents, and the token works only in a control capacity: disallowing it tells Google not to use the content for training Gemini models or for grounding in Gemini Apps and Grounding with Google Search on Vertex AI.

Google says Google-Extended does not affect a site's inclusion in Google Search and is not a ranking signal. AI Overviews and AI Mode are part of Search, so robots.txt rules for Googlebot and snippet controls, not Google-Extended, govern them.

Why it matters

Google-Extended is often mistaken for an 'AI Overviews opt-out'. Blocking it to leave AI Overviews changes nothing there, while it may keep the site out of grounded answers in Gemini Apps, where buyers also research vendors.

Example

A marketing team adds 'User-agent: Google-Extended' and 'Disallow: /' to stop its blog appearing in AI Overviews. The blog keeps appearing, because AI Overviews follow Googlebot rules and snippet controls. The team reverts the change after deciding that grounded Gemini answers are visibility it wants.

Common misunderstandings

  • Google-Extended does not control AI Overviews or AI Mode.
  • It is not a user agent you will find in server logs; Google crawls with its usual agents.
  • Blocking it does not affect Google Search rankings.