RecoProof
Run check
Menu
Free tool · no account

AI Crawler Checker

See whether your robots.txt lets GPTBot, ClaudeBot, PerplexityBot, Google-Extended and other AI crawlers read your site, with the exact line that decides each one.

Check my site Check if AI recommends you

We read your live robots.txt and test your homepage, or the page path you enter, against 25 AI crawlers. Nothing is stored.

Crawler reference

The 25 AI crawlers this tool checks.

Each token and purpose below comes from the operator's own documentation, checked on 23 September 2026. The difference that matters most: search crawlers decide whether an assistant can cite you; training crawlers do not.

Glossary: AI crawler, GPTBot, OAI-SearchBot, Google-Extended.

AI search and answer crawlers

Index pages that AI assistants can show, cite and link to. Blocking these is what removes you from AI answers.

Model training crawlers

Collect data to train future models. Blocking these is a common, deliberate choice and does not by itself remove you from AI search answers.

Copy-ready robots.txt

Three common AI crawler policies.

Stay in AI search, opt out of training

A common deliberate setup: AI assistants can still fetch and cite your pages, while the main training crawlers are asked to stay out.

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: CCBot
Disallow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Open to every crawler

Allows everything and points crawlers at your sitemap. Equivalent to having no rules, but explicit.

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Keep private paths out, everything else open

Blocks app and account areas for all crawlers without touching your marketing pages. Check that pricing, product and comparison pages are not under a disallowed path.

User-agent: *
Disallow: /app/
Disallow: /account/
Allow: /

Sitemap: https://example.com/sitemap.xml

Several User-agent lines above one set of rules form a single group, so every crawler named there gets the same rules. Replace example.com with your domain.

Know the boundary

Being crawlable is the floor, not the goal.

A page AI crawlers cannot fetch will not be cited. But a page they can fetch is not automatically recommended. B2B buyers ask assistants for shortlists, and the answer depends on the evidence the model finds about you and your competitors: reviews, comparisons, documentation and third-party coverage.

Test the recommendation, not just the access

The free AI visibility check runs a real buyer question for your category and shows whether your brand is named, which competitor appears instead, and which sources are cited.

Run the free AI visibility check
FAQ

AI crawler and robots.txt questions.

Does blocking GPTBot remove my site from ChatGPT?

No. GPTBot collects training data. ChatGPT search uses OAI-SearchBot, and OpenAI says sites that block OAI-SearchBot are not shown in ChatGPT search answers, though they can still appear as navigational links. Block GPTBot and allow OAI-SearchBot if you want to opt out of training but stay citable.

Does Google-Extended control AI Overviews or AI Mode?

No. Google-Extended covers Gemini model training and grounding in Gemini Apps and Vertex AI. Google says robots.txt rules for Googlebot are the control for Search, including AI Overviews and AI Mode.

Do AI crawlers obey robots.txt?

Most operators in this check say their automatic crawlers follow robots.txt; ByteDance publishes no policy, and Mistral states none for MistralAI-Index. Several say their user-triggered fetchers, which visit a page because a person asked, may not. robots.txt is a voluntary protocol; enforcing it takes firewall or CDN rules.

Why does the checker say allowed when my CDN blocks AI bots?

This tool reads robots.txt only. CDN and firewall features, such as bot-blocking rules, can refuse AI crawlers before robots.txt is ever read. If robots.txt allows a crawler but your CDN blocks it, the crawler still cannot reach your pages.

How are the rules applied?

Following RFC 9309: every group that names the crawler is merged, the User-agent: * group applies only when none do, the longest matching path wins, and Allow wins a tie. A robots.txt that returns 4xx means no rules; a 5xx or an unreachable server means crawlers must treat the whole site as disallowed.

If AI crawlers can reach my site, will AI recommend it?

Not necessarily. Crawler access only makes your pages available. Whether an AI answer names you for a buyer question depends on what it finds about you and your competitors across the web. The free AI visibility check tests that directly.

Method: rules are parsed per RFC 9309, the first 500 KiB are read, and up to five redirects are followed. Related free tools: the llms.txt generator and validator and the AI visibility score calculator. See how RecoProof measures recommendations in the methodology.