AI Crawls My Site But Doesn't Cite It: A 3-Stage Fix

Stage one: is the page fetchable by the bots that feed AI answers

Fetchable means the specific bots that build AI answers can retrieve the page, not just that a generic crawler can. There are two different kinds worth separating: search crawlers that build the engine's index ahead of time (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and on-demand fetchers that load a page in real time while answering a user's question (ChatGPT-User, Claude-User, Perplexity-User). Neither is the same as a training crawler like GPTBot, which feeds model training rather than live answers and has no bearing on whether a page gets cited today. Checking fetchability means confirming robots.txt allows these specific user-agents, that no firewall or CDN rule (Cloudflare bot-fight mode is a common culprit) is silently dropping them, and that your server answers fast enough before the fetcher times out. One wrinkle matters here: some on-demand fetchers don't consistently honor robots.txt the way search crawlers do, so a clean robots.txt doesn't guarantee access for them the way it does for the indexing bots. This stage has enough moving parts that it deserves its own walkthrough — the can ChatGPT crawl your website article covers the exact checks and fixes in detail.

Robots.txt blocking is becoming rare in practice. A KinetixSEO crawl study of 45 public B2B SaaS homepages, run on October 6, 2026, found that 0% of them still carried a robots.txt rule blocking OAI-SearchBot, PerplexityBot, or Claude-SearchBot from the site root. A direct check confirms the same pattern on a single site: KinetixSEO ran its "ai crawler robots block" test against kinetixseo.com's own robots.txt on October 7, 2026, and the result allowed all three AI search crawlers outright, with no Disallow rule blocking any of them from the site root. In other words, for most sites in this category, robots.txt is no longer the bottleneck — which means that if your page is being crawled but not cited, the problem usually sits in stage two or three, not stage one.

Stage two: is the page chosen when it should be the answer

Chosen means that, for the specific questions where your page is a plausible answer, the AI actually picks it — and the only honest way to know is to track a fixed set of prompts over several weeks, not run one query and draw a conclusion. AI answers vary between asks, between engines, and over time as the underlying index updates, so a single test tells you almost nothing about whether you're reliably chosen. A real example of this stage failing: on October 5, 2026, KinetixSEO's citation tracking ran the prompt "Can ChatGPT crawl my website? How do I check whether it can?" against Grok. Grok answered and cited nine sources — openai.com, ppc.land, diamondbackadvertising.com, highsalience.com, kitbase.dev, and four others — and kinetixseo.com was not among them, despite having a directly relevant article on that exact question. That's a chosen-stage failure: the page exists, is almost certainly fetchable, and still didn't make the cut on that run. See the tracked check for the full source list.

The most common failure at this stage isn't a bad page — it's a crowded topic. A page can answer a question well and still rank fourth-best on a subject where the engine only cites its top two sources for that prompt. No amount of schema or rewriting moves a fourth-place answer into a two-slot citation if three other sources are simply stronger on that specific question. That's a competitive gap, not a technical one, and it's worth reading alongside GEO ranking, which explains why there's no position number to chase in AI search the way there is in traditional rankings — being "chosen" is binary per answer, not a scored position you can nudge up incrementally.

Stage three: is the page extractable once it's fetched

Extractable means that, once a bot has the page open, there's something in it worth quoting directly: a direct answer in the first lines under the relevant heading, named entities, a concrete figure with its source stated, a visible date, and a named author where relevant. An AI engine assembling an answer is scanning for a sentence it can lift cleanly; if the real content is buried under an introduction, a mission statement, or a generic opener, there's nothing to extract even though the crawl succeeded. The GEO best practices checklist and the GEO checklist both work through this in more depth, and how to write an answer block is specifically about structuring that first-lines answer so it's quotable on its own.

Checking extractability means reading the page the way a bot scans it, using five checks. These map directly onto what a citing engine needs: a sentence it can quote without editing, and a fact it can attribute without guessing.

• Does the first sentence under each heading actually answer that heading, or does it build up to the answer two sentences later?
• Are there named entities, dates, or figures a bot could lift as a standalone fact, rather than only vague descriptions?
• If a statistic appears, is its source named next to it, not just asserted?
• Is there a visible publish or update date, and an author name, near the top of the page?
• Could you delete everything except the first paragraph of each section and still have a coherent summary of the page?

The most common failure here is a page that's genuinely well-written but front-loads context instead of the answer — a long introduction, a brand story, or throat-clearing before the actual information. If a reader has to scroll past three paragraphs to find the direct answer, so does the bot deciding what to quote.

Check the stages in order: fetchable, then chosen, then extractable

The three-stage diagnostic order

1. 1

   Fetchable

   Confirm robots.txt, firewall/CDN rules, and server response time for AI crawlers and on-demand fetchers.

2. 2

   Chosen

   Track a fixed set of prompts over several weeks to see whether the page is actually picked as a source.

3. 3

   Extractable

   Check for a direct answer up top, named entities, a sourced figure, a visible date, and an author.

Check the stages in that order — fetchable, then chosen, then extractable — because each later stage is meaningless if an earlier one is broken, and checking out of order is exactly how effort gets wasted on the wrong fix. Confirm fetchable first using the checks above: robots.txt, firewall and CDN rules, and server response time, since nothing downstream matters if the bot can't reach the page at all. Once fetchable is confirmed, measure chosen over several weeks with a fixed set of prompts rather than a single query, because one test run — like the Grok check above — only tells you about that one moment. Only once a page is both reachable and reliably in the running should you spend time on extractability: rewriting the first lines, adding named entities and a sourced figure, or adding a visible date and author. A page that is already extractable gains nothing from more schema if it's never chosen, and a page that's chosen elsewhere on a topic may simply be outranked by stronger sources on this particular question — no on-page rewrite fixes being the fourth-best answer where an engine only cites two.

None of this guarantees a citation. No AI engine publishes the exact mechanism it uses to pick sources for a given answer, and nothing here — including fixing all three stages cleanly — assures inclusion in any specific response. What this diagnostic does is tell you which stage is plausibly costing you citations, so the work goes where it can actually move the needle instead of being spent on a stage that was never the problem.

How KinetixSEO's AI Visibility tab checks these three stages

KinetixSEO's AI Visibility tab shows these three stages for a tracked site, judged from its homepage and its robots.txt. It also lists the tracked keywords where the site is reachable but isn't being cited yet, which is the specific signal that points at a chosen- or extractable-stage problem rather than a fetchable one. It doesn't check firewall or CDN rules, and it doesn't rate every page on the site — the homepage-and-robots.txt view is a starting signal for where to look, not a full crawl audit. For teams deciding how much of this to run themselves versus hand off, GEO agency vs. DIY and the GEO tools buying guide cover what each option actually involves.

Frequently asked questions

Why doesn't ChatGPT cite my website even though it reads the page?

ChatGPT reading or fetching your page only confirms the fetchable stage passed — it says nothing about whether the page was chosen for a given question or whether it contained something extractable once opened. The gap usually sits in one of the other two stages: either the page wasn't selected as a top source for that specific prompt, or the answer on the page wasn't positioned clearly enough near the top for the model to quote. Checking which stage is failing, rather than assuming the whole pipeline is broken, is the point of treating fetchable, chosen, and extractable as separate checks.

What does it mean when AI bots crawl a site but don't cite it?

AI bots crawling a site without citing it means the bot successfully reached the page but either didn't select it for the question being asked or didn't find a clean, quotable answer once it got there. Crawling and citing are separate events: a bot can fetch a page, decide it isn't the strongest source for a given prompt, and move on without citing it, or it can fetch a page and find the real answer buried under introductory text. Distinguishing these requires tracking actual citation outcomes over a set of prompts, not just confirming the crawl happened.

How do I check if ChatGPT or Perplexity can actually access my site?

Checking access means confirming robots.txt for the specific user-agents involved — OAI-SearchBot, Claude-SearchBot, and PerplexityBot for indexing, ChatGPT-User, Claude-User, and Perplexity-User for on-demand fetches — and confirming no firewall or CDN rule is silently blocking them. Keep in mind that some on-demand fetchers don't reliably follow robots.txt rules the way the indexing crawlers do, so a permissive robots.txt doesn't guarantee access for every fetcher type. The can ChatGPT crawl your website article walks through the full set of checks, including server response time under a fetch timeout.

Does adding schema markup help a page get cited by AI search?

Schema markup can help a page become extractable by making entities, dates, and figures explicit, but it doesn't address the fetchable or chosen stages at all. If a page is already extractable and still isn't getting picked, more schema won't fix that — the problem is either reach (fetchable) or competition on that specific question (chosen), neither of which markup changes. Schema is worth doing as part of a clean, well-structured page, but it's not a substitute for diagnosing which stage is actually failing.

Is there a guaranteed way to get cited by AI engines like ChatGPT or Perplexity?

No engine guarantees a citation, and none publishes the exact mechanism it uses to select sources for a given answer. Passing all three stages — fetchable, chosen, extractable — improves the odds without guaranteeing inclusion in any specific response. The GEO guide and the GEO paper on the Princeton study go into what research has actually found about what correlates with citation, as distinct from what any single engine guarantees.