Free tool · AI crawler checker
Which AI crawlers can read your site?
Your robots.txt decides which AI bots get in — and one stray Disallow can quietly erase you from ChatGPT search, Perplexity and AI Overviews. This checker reads your file the way the bots do and gives you a plain verdict for all 16.
Free · no signup · reads only your public robots.txt
Why this matters
AI answers start with a crawl
Before ChatGPT can mention you or Perplexity can cite you, a crawler has to read your pages. robots.txt is where that either happens or silently doesn’t.
No crawl, no citation
ChatGPT search, Perplexity and AI Overviews can only cite pages their crawlers have read. If OAI-SearchBot or PerplexityBot is blocked, your site simply doesn’t exist to those answers — no matter how good the content is.
Blocks are usually accidents
Copy-pasted robots.txt blocklists and aggressive anti-scraper rules routinely catch AI search crawlers by mistake. One stray Disallow written for a scraper in 2019 can be silently costing you AI visibility today.
Training is a separate lever
GPTBot collects training data; OAI-SearchBot powers ChatGPT search. Blocking one does not block the other. Knowing which token controls which behavior is the whole game — and it’s what this check makes explicit.
The catalog
The 16 AI bots we test against
Three categories, three different stakes: crawlers that feed AI search answers, assistants fetching pages for a user, and bots collecting model training data.
AI search & answers
OAI-SearchBotOpenAI
Builds the search index behind ChatGPT search results and link citations.
Claude-SearchBotAnthropic
Indexes pages so Claude can ground its answers in current web results.
PerplexityBotPerplexity
Indexes pages so Perplexity can surface and cite them as sources in answers.
GooglebotGoogle
Google's core Search crawler — the same index feeds AI Overviews and AI Mode.
AmazonbotAmazon
Crawls the web to power Alexa's question answering.
DuckAssistBotDuckDuckGo
Gathers sources for DuckAssist's AI-generated answers.
AI assistants (user-triggered)
ChatGPT-UserOpenAI
Visits your pages on demand when a ChatGPT user asks about them — user-triggered visits may not honor robots.txt.
Claude-UserAnthropic
Visits your pages on demand when a Claude user asks about them — honors robots.txt, unlike most user-triggered fetchers.
Perplexity-UserPerplexity
Fetches pages a Perplexity user opens or cites — user-triggered visits may not honor robots.txt.
AI training
GPTBotOpenAI
Collects public web text used to train OpenAI's future models.
ClaudeBotAnthropic
Collects public web text used to train Anthropic's Claude models.
Google-ExtendedGoogle
Opt-out control for Gemini training and Gemini/Vertex grounding — blocking it does not remove you from Search or AI Overviews.
CCBotCommon Crawl
Builds the open Common Crawl corpus that many AI labs train models on.
BytespiderByteDance
Gathers training data for ByteDance's models, including Doubao.
meta-externalagentMeta
Collects training data for Meta's Llama models.
Applebot-ExtendedApple
Opt-out control for Apple's model training — normal Applebot crawling is separate.
How the check works
Read like a bot, reported like a person
The verdicts follow the same standard the crawlers themselves implement — RFC 9309 — so what you see here is what a compliant bot concludes.
- 01
We fetch your robots.txt
We request https://yourdomain/robots.txt over HTTPS with an 8-second timeout and a 256 KB size cap, following redirects the way a crawler would. We read only that one public file — nothing else on your site.
- 02
We parse it per RFC 9309
Rules are grouped under consecutive User-agent lines. Each bot obeys the most specific group that matches its token — an exact match beats a partial one, which beats the * wildcard. Within a group the longest matching path rule wins, Allow wins length ties, and * wildcards plus the $ end anchor are honored.
- 03
We evaluate your homepage path
Each verdict is that crawler’s access to the / path. A file that blocks only some subfolders still reads as allowed here — homepage access is the gate that decides whether a crawler can see your site at all.
- 04
We report the edge cases honestly
No robots.txt (a 404 or any 4xx) means unrestricted — the standard says a missing file allows everything. A server error (5xx) is the opposite: while errors persist, crawlers must assume your entire site is disallowed. We tell you which situation you’re in rather than guessing.
One honest caveat: robots.txt is advisory, not a firewall. The user-triggered fetchers (ChatGPT-User and Perplexity-User) document that they may ignore it — Claude-User is the exception that honors it — and Bytespider has been observed fetching robots.txt without enforcing its rules. For everything you truly need blocked, network-level bot rules on your CDN are the enforcement layer.
FAQ
Common questions
Should I block GPTBot?
Only if you’ve made a deliberate decision to keep your content out of OpenAI’s model training. Blocking GPTBot does not affect ChatGPT search — that’s OAI-SearchBot — or the fetches ChatGPT makes when a user asks about your site, which come from ChatGPT-User. Many publishers block training bots and keep search bots open; that combination preserves AI visibility while withholding training data.
Does blocking Google-Extended remove me from AI Overviews?
No. AI Overviews and AI Mode are built from Google’s normal Search index, which Googlebot crawls. Google-Extended is purely an opt-out control for Gemini model training. Blocking it keeps you in Search and AI Overviews; blocking Googlebot itself is what removes you from both.
What if I have no robots.txt at all?
Then every crawler is allowed — RFC 9309 treats a missing file (any 4xx response) as permission to crawl everything. That’s the default state of the web and perfectly fine for most sites. You only need a robots.txt when you want to restrict something.
How do I unblock an AI crawler?
Open your robots.txt and find the User-agent group that matches the crawler — either its own token or a User-agent: * group. Remove the Disallow rule that covers your pages, or add an explicit Allow for them, then redeploy the file. Most crawlers cache robots.txt for up to about 24 hours, so expect the change to take effect within a day — Amazonbot is the outlier, caching for up to 30 days.
The checker says a crawler is allowed — why doesn’t it visit?
robots.txt is a published request, not an enforcement layer. CDN and firewall bot rules (Cloudflare, Akamai and similar) act at the network level and can block or challenge crawlers regardless of what robots.txt says. This tool reads only your robots.txt, so treat its verdict as necessary but not sufficient — if an allowed crawler never shows up in your logs, check your CDN’s bot settings next.
Next steps: give AI assistants a curated map of your site with the llms.txt generator, or see whether ChatGPT already mentions you with the AI visibility checker.
Start tracking
Crawlable is step one
OptimizeCamp tracks whether AI answers actually mention you — every week, against your competitors — and finds the content changes that move it.