How to Configure robots.txt for AI Crawlers (GPTBot, ClaudeBot, PerplexityBot)

Generative engine optimization has a technical prerequisite that is easy to miss: AI answer engines can only cite content their crawlers can read. GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and Applebot-Extended are the bots that feed ChatGPT, Claude, Perplexity, Google AI Overviews, and Apple Intelligence.

If robots.txt, an edge firewall, or a security tool blocks them, your brand simply does not exist in AI answers, no matter how good the content is.

This guide covers the exact rules to allow, the trade-offs of blocking, and how to verify the crawlers actually arrive.

The copy-paste robots.txt baseline

Most sites can safely allow all AI crawlers while still excluding private or low-value paths. This baseline keeps the important bots open:

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Applebot-Extended
Allow: /

User-agent: *
Allow: /

Deploy that, and every major AI answer engine can read your pages. The site already uses this pattern in production; this guide exists because most small business sites do not.

What each crawler feeds

| User agent | Feeds | Notes | |---|---|---| | GPTBot | ChatGPT and OpenAI products | The most important one for answer visibility | | ClaudeBot | Claude and Anthropic products | Added by Anthropic, respects robots.txt | | PerplexityBot | Perplexity answers | Also used for live answer retrieval | | Google-Extended | Gemini and AI Overviews | Google's AI-specific crawler | | Applebot-Extended | Apple Intelligence and Siri | Apple's AI-specific crawler | | * | Everything else | Keep generic access open too |

When blocking makes sense

There are legitimate reasons to exclude specific paths. The pattern is to block the path, not the crawler:

User-agent: GPTBot
Disallow: /admin/
Disallow: /checkout/
Disallow: /search?

Blocking only what must stay private keeps citation value on public content. Fully blocking AI crawlers (a Disallow: / for GPTBot) removes your pages from the pool ChatGPT can cite and should be reserved for cases where bandwidth, content theft, or policy is a real concern.

How to verify AI crawlers actually visit

1. Access logs. Search your host or edge logs for user agents containing GPTBot, ClaudeBot, and PerplexityBot. A healthy content site should show steady, recurring visits.
2. Traffic analysis. In analytics, look at referral or source reports for AI platforms such as chatgpt.com, perplexity.ai, and gemini.google.com. Users arriving from those platforms are a sign the citation loop works.
3. Monthly audit. Add this check to your GEO visibility checklist run: did GPTBot and ClaudeBot crawl at least once in the last 30 days? If not, inspect robots.txt, edge firewall rules, and CDN security policies.

Common ways AI crawlers get blocked without you noticing

• Security tools and WAFs that rate-limit or challenge unknown user agents.
• CDN bot management configured to drop unrecognized bots.
• Caching layers that serve empty responses to crawler user agents.
• A generic Disallow: / added for an unrelated bot that also catches AI crawlers.
• Unexpected redirects on the canonical URL that break the fetch.

After any deploy or security change, run the crawler check above. Blocking is silent: citations disappear before traffic does.

The bigger picture

Opening robots.txt is the cheapest and highest-leverage GEO action available, and it connects to the rest of the strategy: GEO vs SEO: what changes explains where this fits next to classic ranking, brand mention tracking turns the resulting citations into a measurement loop, and the GEO visibility checklist audits the whole system in one pass.