What it means
Operators document separate tokens for separate purposes. OpenAI uses GPTBot for training and OAI-SearchBot for ChatGPT search; Perplexity uses PerplexityBot for search; Anthropic separates ClaudeBot from Claude-SearchBot. Some tokens, such as Google-Extended and Applebot-Extended, are control tokens that do not crawl on their own.
User-triggered fetchers such as ChatGPT-User and Perplexity-User visit a page because a person asked, and their operators say robots.txt may not apply to them. CDN and firewall bot rules can also block a crawler before it ever reads robots.txt.
Why it matters
Blocking the wrong token can remove a site from AI search answers while leaving training untouched, or the reverse. Decide separately whether to allow training and whether to be citable, then check that robots.txt and the CDN say the same thing.
Example
A SaaS company's robots.txt disallows GPTBot and OAI-SearchBot in one group because both sounded like training bots. It stays out of OpenAI's training, as intended, but it also opted out of ChatGPT search answers. Splitting the group restores search access and keeps the training opt-out.
Common misunderstandings
- Blocking training crawlers does not remove a site from AI search answers; search crawlers are separate.
- robots.txt is a request, not access control, and user-triggered fetchers may not follow it.
- Being crawlable does not mean being recommended; it is the floor, not the goal.