Crawling vs Indexing Explained: A Diagnostic Walkthrough

The diagnostic scenario: 200 OK, no noindex, but no ranking

A page returning 200 OK with no noindex tag and still not ranking is not automatically a crawling problem or an indexing problem — it could be either, or neither, and the only way to know is to check each stage separately rather than assume. The status code and the noindex tag only rule out the two most obvious failure modes. A 200 OK tells you the server responded successfully when someone (or something) requested the URL — it says nothing about whether Googlebot was the one requesting it, or how often. The absence of a noindex tag tells you the page isn't explicitly blocking indexing — it doesn't tell you Google chose to index it anyway. Between "server works" and "page indexed" sits a chain of independent decisions: Googlebot has to discover the URL, decide it's worth crawling given the page's priority and the site's crawl budget, successfully render it, then separately decide the content is worth adding to the index rather than treating it as a near-duplicate, thin, or low-value page.

How to check the crawling stage in Search Console

Diagnosing a crawled-but-unranked page in Search Console

1. 1

   Open URL Inspection and paste the exact URL

2. 2

   Check the Crawl section for a Last crawl timestamp

3. 3

   If no recent crawl: diagnose as a crawling-stage problem (orphaned page, robots.txt, crawl budget)

4. 4

   If crawled recently: check the Coverage/Indexing status next

5. 5

   Read the specific status: Crawled-not indexed, Discovered-not indexed, or Duplicate-different canonical

6. 6

   Match the status to its fix: content quality, crawl priority, or canonical correction

Open the URL Inspection tool in Google Search Console and paste the exact URL — this tells you whether Googlebot has crawled it at all, and when. The report shows a "Crawl" section with a "Last crawl" timestamp and the crawl status. If there's no crawl recorded, or the last crawl is months old on a page you've since updated, that's a crawling-stage problem: Googlebot either hasn't found the URL yet or isn't prioritizing it highly enough to revisit. Common causes at this stage include the page being orphaned (no internal links pointing to it), buried too deep in the site's structure, excluded by robots.txt, or simply competing for crawl budget against thousands of other URLs on a large site. The how Googlebot works guide covers how crawl budget gets allocated and what wastes it. If the crawl report instead shows a recent, successful crawl, the problem is downstream — move to the indexing check rather than troubleshooting the server or robots.txt further.

How to check the indexing stage in Search Console

If Googlebot crawled the page successfully, the next question is whether Google chose to index what it found — and the URL Inspection tool answers this directly with its "Coverage" result. A crawled-but-unindexed page typically shows one of a small set of specific statuses: "Crawled - currently not indexed," "Discovered - currently not indexed," or "Duplicate, Google chose different canonical than user." Each points to a different cause. "Crawled - currently not indexed" usually means Google rendered the page and judged the content too thin, too similar to existing pages, or not valuable enough to add right now — it's a quality decision, not a technical block. "Discovered - currently not indexed" means Google knows the URL exists but hasn't crawled it yet, which is actually a crawling-stage symptom despite showing up in the indexing report. "Duplicate, Google chose different canonical" means indexing happened, just not under the URL you expected, because Google's canonicalization logic picked a different page to represent the set.

A mismatched canonical is common enough to check specifically, and it's worth confirming the page's own rel=canonical tag points at itself rather than somewhere else before assuming Google's algorithm made the call independently. A KinetixSEO crawl study that fetched 36 public SEO and web-performance guide articles from software vendors and developer publications on October 6, 2026, found that 3% of them pointed rel=canonical at a URL other than the one they actually served — a small but real failure rate even among technical-content publishers who should know better. As a comparison point, a canonical check KinetixSEO ran on October 19, 2026 against its own article on AI/LLM crawling found it self-referencing with no conflict, which is the result you want to see before ruling out canonicalization as the cause of an indexing mismatch.

Why "Google can reach my page" doesn't mean "Google indexed my page"

Treating crawl access and index inclusion as the same event is the single most common cause of this confusion, because both stages can produce a "successful" status that looks fine at a glance. A 200 OK and a clean crawl report only confirm that Googlebot's fetch succeeded — they say nothing about the separate evaluation that happens afterward, where Google's systems weigh the rendered content against everything else it already knows about similar pages before deciding whether storing it serves searchers. That evaluation isn't a yes/no gate tied to technical cleanliness; a page can be perfectly crawlable, carry no blocking directives, and still sit in "Crawled - currently not indexed" indefinitely because the content doesn't clear the bar on its own merits. This is also why indexing problems get described as "soft" failures compared to crawling problems: there's no error message, no blocked status, no tag to remove — just an absence, which is exactly why it gets misdiagnosed as a crawling issue when the fix actually needs to happen to the content or the canonical setup, not the server.

Crawling vs indexing: the core difference in practice

Crawling vs indexing: what each stage checks

	Crawling	Indexing
Question it answers	Can Googlebot fetch this page?	Did Google decide to store this page's content?
Search Console signal	Last crawl timestamp	Coverage / indexing status
Typical failure cause	Orphaned page, robots.txt block, crawl budget	Thin content, duplicate content, canonical conflict
Typical fix	Internal links, sitemaps, crawl-budget cleanup	Improve content quality, fix canonical tag

Crawling is Googlebot fetching a URL; indexing is Google's separate decision to store that page's content so it can be returned in search results, and conflating the two is what produces the "it's live but it's not ranking" confusion in the first place. In practice this plays out as a sequence: discovery (Google finds the URL exists, often via a link or sitemap), crawling (Googlebot requests and renders it), and indexing (Google evaluates the rendered content and decides whether to keep it). A page can stop at any one of those three points, and each stop looks different in Search Console — no crawl record means it never got past discovery, a crawl record with "Discovered - currently not indexed" means it's queued but not yet fetched, and a crawl record with "Crawled - currently not indexed" means it was fetched and rejected at the content-evaluation step. Knowing which of these three you're looking at is what turns a vague "it's not ranking" complaint into a specific, fixable diagnosis — and it's why checking the actual Search Console status beats guessing from the page's HTTP response alone.

Difference between crawling and indexing in one sentence

The difference is that crawling is about access and indexing is about inclusion: crawling asks "can Googlebot fetch this page," while indexing asks "has Google decided this page's content is worth storing and serving." A page can pass the first test and fail the second — that's the entire scenario this article walks through. It's also possible, though less common, to fail crawling while technically being "indexable" in the sense of carrying no blocking tags: a page robots.txt disallows, or one that's simply never linked from anywhere Googlebot looks, never gets the chance to be evaluated for indexing at all, because it's never fetched in the first place. Both failure types produce the same symptom to a site owner — the page doesn't show up in search — but they require different fixes, which is why separating the two stages, rather than treating "not ranking" as one undifferentiated problem, is the practical value of understanding this distinction.

Why crawling and indexing confusion extends beyond Google to AI engines

The same gap between "reachable" and "included" shows up when AI answer engines decide what to cite, and it compounds a step further because AI crawlers and indexing-equivalent inclusion are even less observable than Google's. On 19 October 2026, KinetixSEO tracked the prompt "Can ChatGPT crawl my website? How do I check whether it can?" run against Gemini with web search, and the response cited 9 sources — xseek.io, openai.com, unveilrai.com, crawlercheck.com, thats.be, and four others — with kinetixseo.com absent from that list despite publishing directly relevant material. That's the same two-stage failure pattern playing out in a newer channel: being technically crawlable by an AI agent is no guarantee of being selected and cited in a generated answer, just as being crawlable by Googlebot is no guarantee of being indexed. Site owners auditing AI referral traffic run into this distinction constantly, and the diagnostic habit is the same one described above: check reachability and inclusion as two separate questions rather than assuming one implies the other.

Where to go next after a crawling or indexing diagnosis

Once you know which stage failed, the fix lives in a different place depending on the diagnosis: crawling problems get fixed with internal linking, sitemaps, and crawl-budget management, while indexing problems get fixed by improving content quality or cleaning up canonical signals. For a broader view of how discovery, crawling, and indexing fit into the full search pipeline before ranking even enters the picture, see how search engines work. For the mechanics of crawl budget specifically — what consumes it and how to audit waste on a large site — the Googlebot crawling guide goes deeper than the Search Console check described here. And if the root cause turns out to be structural rather than content-related — duplicate URL variants, inconsistent canonicalization, or rendering issues that hide content from Googlebot — the technical SEO basics primer covers the underlying fixes rather than just the diagnosis.

Frequently asked questions

What is the meaning of crawling and indexing?

Crawling means a search engine's bot fetches a page's content by requesting its URL, and indexing means the search engine separately decides to store that content so it can appear in search results. The two are sequential but independent: a page must be crawled before it can be indexed, but being crawled doesn't guarantee indexing follows. Google's systems evaluate rendered content for quality, duplication, and value before adding it to the index, which is a distinct step from the fetch itself.

Can a page be crawled but not indexed?

Yes, and Search Console's URL Inspection tool labels this exact state "Crawled - currently not indexed." It means Googlebot successfully fetched and rendered the page but decided, after evaluating the content, not to add it to the index — usually because the content is thin, too similar to another indexed page, or judged low-value relative to what's already indexed on the same topic. This is a content or canonicalization issue, not a technical access problem.

Can a page be indexed without being crawled recently?

Yes, pages can remain indexed for a long time between crawls, since Google doesn't recrawl every indexed URL on every cycle. The "Last crawl" date in URL Inspection can be weeks or months old even for a page that's currently indexed and ranking, because Google prioritizes recrawl frequency based on how often a page tends to change and how much weight it carries in the index already.

How do I know if Google indexed my page at all?

Check the "Coverage" section of the URL Inspection tool in Search Console, which reports whether the URL is indexed and, if not, why. A direct "site:" search in Google isn't fully reliable for this because Google has changed how that operator surfaces results over time, but URL Inspection pulls the actual indexing status Google holds for that specific URL, including the exact non-indexing reason when one applies.

Does fixing a crawling problem automatically fix an indexing problem?

No, because they're separate evaluations. Making a page easier to crawl — adding internal links, fixing a robots.txt block, submitting it in a sitemap — only gets it fetched more reliably; it does nothing to change Google's separate judgment about whether the content is worth indexing once fetched. If a page shows "Crawled - currently not indexed," the fix is improving the content or resolving a canonical conflict, not re-crawling it.