AI Crawler robots.txt Checker
See whether GPTBot, ClaudeBot, PerplexityBot and Google-Extended can read your homepage before you spend a month on GEO content.
What this tool gives you
Check a robots.txt file
Fetch a live file or paste one. The checker scores GPTBot, ClaudeBot, PerplexityBot, Google-Extended and the rest of the citation-relevant agents.
Agents this tool scores
Unspecified usually means the wildcard group applies, or the crawler is allowed by default. Bytespider is scored as review-only because of bandwidth, not citation value.
GPTBot
OpenAI training crawler
Feeds model training. Allow public content; keep /api/ and private paths blocked via *.
ChatGPT-User
ChatGPT browsing / user fetch
Used when a person asks ChatGPT to open a live page.
OAI-SearchBot
ChatGPT search citations
Separate from GPTBot. Needed if you want pages considered for ChatGPT search snippets.
ClaudeBot
Anthropic training crawler
Training crawler. Pair with Claude-SearchBot if you care about Claude search citations.
Claude-SearchBot
Claude search / citation fetch
Search/citation agent, not the training crawler.
PerplexityBot
Perplexity answer citations
Primary Perplexity crawler for cited answers.
Google-Extended
Gemini / Google AI training opt-out
Does not control Google Search or AI Overviews. Blocking it only opts out of Gemini training.
Applebot-Extended
Apple Intelligence / Applebot AI opt-out
Does not control Applebot search crawling. Blocking it opts out of Apple Intelligence training.
CCBot
Common Crawl
Public web corpus used by many research and AI pipelines.
Amazonbot
Amazon / Alexa crawler
Used for Amazon surfaces including Alexa-related retrieval.
meta-externalagent
Meta AI crawler
Meta AI training/fetch agent. Distinct from Facebook's link preview bot.
Googlebot
Google Search
Classic search crawler. Blocking it hurts Google Search, not just AI features.
Bytespider
ByteDance / TikTok training
Often high bandwidth. Do not allow by default unless you explicitly want ByteDance training.
How to use the checker
robots.txt is an access gate. llms.txt and schema cannot recover citations if the crawler never gets in.
Check the live file
Fetch https://yourdomain.com/robots.txt or paste the current file from your host.
Unblock citation crawlers
Allow GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot on public pages. Keep /api/ closed.
Recheck after deploys
CDN rules, WAFs and security plugins can block the same bots even when robots.txt looks open.
What a healthy file usually includes
Allow public HTML. Disallow private apps. Name the AI crawlers you actually want. Do not treat Google-Extended as a Search kill switch.
| Agent | Job | Default for public sites |
|---|---|---|
| GPTBot / ChatGPT-User / OAI-SearchBot | OpenAI training, live browsing, ChatGPT search | Allow public pages |
| ClaudeBot / Claude-SearchBot | Anthropic training and Claude search | Allow public pages |
| PerplexityBot | Cited answers in Perplexity | Allow public pages |
| Google-Extended | Gemini training opt-out only | Allow unless you opt out of training |
| Googlebot | Google Search | Allow unless the site is private |
| Bytespider | ByteDance training | Review; often blocked for bandwidth |
Related GEO pages
Open the crawlers first, then score content, mentions and schema.
robots.txt for AI Crawlers
Copy-paste rules for GPTBot, ClaudeBot, PerplexityBot and Google-Extended.
GEO Visibility Checklist
Score citation-ready content, entity clarity and weekly mention tracking.
llms.txt Checker
Score whether /llms.txt is useful Markdown after crawlers can actually reach the site.
GEO vs SEO: What Changes
Map the overlap between classic SEO and generative engine optimization.
JSON-LD Checker
Score schema.org identity, page type and review hygiene.
XML Sitemap Checker
Score sitemap.xml loc hygiene: HTTPS URLs, unique paths and lastmod.
Tools Lab
More generators, calculators and GEO workflows.
AI crawler robots FAQs
Which AI crawlers should robots.txt allow?
If you want public pages cited, allow GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Amazonbot, and meta-externalagent. Keep /api/ and private paths blocked for User-agent: *.
Does blocking Google-Extended hide me from AI Overviews?
No. Google-Extended is an opt-out for Gemini training. It does not control Googlebot or Google AI Overviews. Blocking Googlebot is what hurts Search.
Should every site allow Bytespider?
No. Bytespider is a ByteDance crawler that can be expensive on bandwidth. This checker flags it as review-only so you can allow or block it on purpose.