How Do Search Engines Work? Crawling, Indexing, Ranking

The three-stage search pipeline

1. 1

   Crawling

   Bots discover URLs via links and sitemaps

2. 2

   Indexing

   Content and metadata are parsed and stored, or excluded

3. 3

   Ranking

   The query is matched against the index and results are ordered

Crawling: how bots find your pages at all

Crawling is the process by which automated programs — Googlebot is the best-documented example — discover URLs by following links and reading sitemaps. A crawler starts from a set of known pages and its own history, then extracts every link it finds on each page, queuing the destinations for a future visit. This is why orphan pages, ones with no internal or external link pointing to them, are so hard to get discovered: there's no path for the bot to follow in. XML sitemaps supplement this by explicitly listing URLs you want discovered, which matters most for large sites or pages published faster than natural linking can catch up. Crawlers also respect rules in your robots.txt file and operate within a "crawl budget" — a practical limit on how many pages of your site they'll fetch in a given window, shaped by your site's speed and perceived value. For a deeper look at how that budget is allocated and wasted, see how Googlebot crawls your site.

Indexing: how content actually gets stored and understood

Indexing is where a crawled page gets parsed, interpreted, and either stored in the searchable index or set aside. After fetching a URL, the engine extracts the text, HTML structure, and metadata — title tags, headings, structured data, canonical tags — and builds a representation of what the page is about. Crucially, being crawled doesn't guarantee being indexed: a page can be fetched successfully and still be excluded if it's a near-duplicate of another page, carries a noindex directive, or the engine judges it too thin to be worth storing. This is the distinction covered in more depth in crawlability vs. indexability — they're two separate gates, not one step with two names. Canonical tags play a specific role here: when several URLs show near-identical content, they tell the indexer which version to treat as authoritative, consolidating signals onto one entry rather than splitting them across duplicates.

Ranking: matching a query against the index

Signal categories used in ranking

• Relevance to the query's meaning

• Content quality and depth

• Authority signals, including links

• User experience: speed, mobile usability, layout stability

• Freshness of the content

• Context: searcher location, language, device

Ranking only happens at the moment someone searches, when the engine takes the query, finds every indexed page it considers a plausible match, and orders them using a large number of signals evaluated together. Those signal categories, described qualitatively rather than as any precise formula, generally include:

• Relevance — how closely the page's content, structure, and metadata match the meaning of the query, not just its exact words.
• Content quality and depth — whether the page substantively answers the query or is thin, duplicated, or auto-generated.
• Authority signals — including links from other sites, which historically function as a vote of confidence.
• User experience factors — page speed, mobile usability, and stability of layout as it loads.
• Freshness — how recently the content was updated, weighted more heavily for queries where recency matters, like news.
• Context — the searcher's location, language, and device all shape which indexed pages are considered a plausible match.

No public source, including Google itself, has published the exact weighting of these factors, and this article won't pretend otherwise — treat any claim of a precise ranking formula as marketing, not mechanics.

The pitfall that trips up modern sites: rendering

A JavaScript-heavy page can be crawled successfully and still index poorly, because the crawler's first fetch and the browser's final rendered page are not always the same document. When a bot requests a URL, it gets the initial HTML response — and if your key content, links, or metadata are injected afterward by client-side JavaScript, that content may be entirely absent from what the crawler receives on first pass. Google can render JavaScript, but it does so in a separate, delayed step, queued behind the initial crawl and dependent on available rendering resources; other engines and AI answer bots vary widely in whether they render JavaScript at all. The practical risk is that a page looks complete in a browser but reads as blank or near-empty in the crawler's first-pass HTML, which is a rendering failure, not a crawling or a ranking one, and it needs a different fix.

The fix is to make sure the content that matters is present in the initial HTML response, not dependent on a script running first. Server-side rendering, static generation, or hybrid approaches that pre-render the first paint all address this directly. A practical check: fetch a page's raw HTML with a tool that doesn't execute JavaScript — curl, or "view source" rather than "inspect element" — and confirm your primary content and internal links appear there without a script running first. If they don't, you have a rendering gap between what a crawler sees on first pass and what a visitor sees. This single issue explains a large share of "why isn't my page indexed" cases on modern JavaScript frameworks, and it sits squarely inside the technical checks covered in technical SEO basics.

Quick rendering check

• Fetch the page's raw HTML without executing JavaScript

• Confirm primary content appears in that raw response

• Confirm internal links appear in that raw response

• If either is missing, close the rendering gap with server-side or static rendering

Why "hundreds of signals" doesn't mean "no strategy"

Not knowing the exact algorithm doesn't mean ranking is unknowable — it means the useful work is in the mechanics you can verify directly, not in guessing weights. You can confirm whether a page is crawlable by checking your robots.txt and server logs, confirm whether it's indexed with a site search or the indexing report in Search Console, and confirm whether your key content survives to the initial HTML with a raw-fetch check. None of that requires knowing how Google weighs backlinks against page speed. It does mean being skeptical of anyone who claims to have reverse-engineered the ranking formula, or who frames a single tactic as a guaranteed win — those claims fail the same test this article applies to itself: a specific, checkable mechanism beats an unbacked superlative every time.

It's also worth separating this three-stage model from adjacent but distinct SEO concepts, since the terms get used loosely. Comparing paid and organic visibility is a different question, covered in SEO vs. SEM; manipulating rank through disallowed shortcuts is black hat SEO or its milder cousin, gray hat SEO, while techniques like keyword stuffing, cloaking, and link schemes are specific tactics engines actively penalize rather than signal categories they reward. And increasingly, the same crawling-indexing-ranking logic feeds a fourth layer: AI systems that synthesize answers directly, sometimes called an answer engine, which draw on the same underlying index but present a synthesized answer instead of a ranked list of links.

Frequently asked questions

How do search engines work, in one sentence?

Search engines work by running a three-stage pipeline — crawling to discover pages, indexing to store and interpret them, and ranking to order them per search query — where each stage gates the next, so a failure anywhere upstream prevents a page from ever reaching a searcher. Understanding which of the three stages a problem sits in is the fastest way to diagnose why a page isn't performing, rather than treating "SEO" as one undifferentiated process.

Can a page be crawled but not indexed?

Yes, and it's a common and specific failure mode, not an edge case. A crawler can fetch a page successfully and the engine can still decline to index it — because it's a near-duplicate of another URL, carries a noindex tag, or is judged too thin in content to be worth storing. This is exactly why crawling and indexing are treated as two separate gates rather than one combined step; a URL showing up in your server logs as crawled tells you nothing about whether it made it into the index.

Why does JavaScript rendering cause indexing problems?

JavaScript rendering causes indexing problems when a page's key content or links are injected by client-side scripts rather than present in the initial HTML response the crawler first receives. Google renders JavaScript, but as a separate, delayed step queued behind the initial crawl, and other bots vary in whether they render it at all — so content that only appears after a script runs can be missed entirely on first pass, even though the page displays fine to a human visitor in a browser.

Does Google publish its exact ranking algorithm?

No — the precise ranking algorithm, including exact signal weightings, is not public, and any source claiming to know it exactly is not describing verified mechanics. What is documented, in general terms, are categories of signals — relevance, content quality, authority indicators, user experience, freshness, and context — evaluated together per query, without a published formula for how heavily each one counts.

How is ranking different from indexing?

Ranking happens per search query, while indexing happens once when a page is stored. Indexing is the engine deciding whether a page belongs in its searchable database at all; ranking is the separate, later process of taking a specific query, finding every indexed page that's a plausible match, and ordering them by relevance and the other signal categories. A page can sit indexed for months without ranking well for a given query, because ranking is recalculated fresh each time someone searches.