AI Crawler Checker
Is Your Site Blocking the AI Crawlers That Matter?
Paste a URL and see, per bot, whether GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and 20+ other named AI crawlers are allowed or blocked — a page you never see can never be cited.
What you get
Every dimension of GEO Readiness, including crawler access
AI Crawlability is one of four pillars behind the GEO Readiness Score (15% weight) — see the GEO Audit Tool for the full breakdown.
Training, search, and live-fetch bots are different
Not every AI bot does the same job
Blocking a training-only crawler (like Applebot-Extended) affects a future model, not today's answers. Blocking a search-indexing crawler (like OAI-SearchBot) can remove you from a future AI answer. Blocking a live-fetch crawler (like ChatGPT-User) can remove you from an answer being generated right now, for a real user, in this moment. The checker groups every bot it finds by which of these three it is, so a block means something specific instead of just "blocked."
What this doesn't do
Crawler access is a prerequisite for AI visibility, not a guarantee of it — an allowed bot still has to decide your content is worth citing. This reports what your robots.txt currently allows; it can't observe or influence how any AI system actually ranks or selects content once it can reach it.
FAQ
AI crawler checker — FAQ
Which AI crawlers does this check? +
Over twenty named user-agents across OpenAI (GPTBot, OAI-SearchBot, ChatGPT-User), Anthropic (ClaudeBot, Claude-Web, Claude-User), Google (Google-Extended, Google-Agent), Perplexity, Meta, Amazon, Apple, ByteDance, Common Crawl, Cohere, Mistral, DuckDuckGo, Diffbot, and xAI. It reads your robots.txt and reports which are allowed and which are blocked, per bot.
Why does it matter whether a bot trains, searches, or fetches live? +
Blocking a training-only crawler (like Applebot-Extended) affects a future model, not today's answers. Blocking a search or live-fetch crawler (like OAI-SearchBot or ChatGPT-User) can remove you from an AI answer being generated right now. The checker groups every bot by purpose — training, search-index, or live-fetch — so a blocked bot tells you which kind of visibility you're actually losing.
Why would a bot be blocked without me knowing? +
Most often a blanket "Disallow: /" left over from a staging environment, a security plugin's default bot-blocking rule, or a wildcard rule meant for scrapers that also catches AI crawlers by accident. None of these show up unless robots.txt is checked against these specific user-agent strings, one by one.
Is AI crawler access the same as AI visibility? +
No — it's a prerequisite, not a guarantee. A blocked crawler can never cite your page; an allowed one still has to decide your content is worth citing. AI Crawlability is one of four pillars in the GEO Readiness Score (15% weight) alongside evidence density, structure, and authority signals — see the GEO Audit Tool for the full picture.
Is the AI crawler checker free? +
Yes — paste a URL and get the full per-bot allow/block breakdown with no signup, as part of the same free check as the GEO Readiness Score.
Test your AI crawler access
Free, no signup — paste a URL and see the full per-bot breakdown in minutes.