What Is a Web Crawler? Definition, Examples, How It Works

What is a web spider?

A web spider is the same thing as a web crawler. The name comes from the way the software moves across the web, hopping from page to page along links like a spider along strands. "Bot" is the broader label for any automated program, and a crawler is one kind of bot. You will see all three terms used for the software that fetches pages for a search engine.

How does a web crawler work?

A web crawler works by fetching a page, pulling out the links on it, and adding any new links to a queue of pages to fetch next. The cycle repeats until the crawler runs out of budget or new URLs. In practice it runs through these steps:

1. Start from a list of known "seed" URLs, such as earlier crawls or submitted sitemaps.
2. Check the site's robots.txt file for rules about which paths this crawler may fetch.
3. Download the page's HTML and, for some crawlers, render its scripts.
4. Extract every link and add the new ones to the queue.
5. Send the downloaded content to the index.
6. Return later to re-fetch pages that may have changed.

Robots.txt is the control point for step 2. It is a plain text file at the root of a site, and each crawler looks for rules addressed to its own name.

What is a web crawler example?

Googlebot is the standard example: it is the crawler Google runs for classic search. GPTBot and PerplexityBot are different crawlers, run by AI companies, each with its own robots.txt rules. Allowing one does not automatically allow another, and blocking one does not automatically block another. The difference between Googlebot and Google's separate AI-related token is covered in What Is Googlebot?.

The AI side is easy to check. In a KinetixSEO crawl study of 45 public B2B SaaS homepages, measured on 6 October 2026, 0% had a Disallow rule in robots.txt blocking OAI-SearchBot, PerplexityBot or Claude-SearchBot from the site root. Those are AI search crawlers. On 19 October 2026, KinetixSEO also ran its "ai crawler robots block" check against its own robots.txt, and the result was that all three AI search crawlers are allowed.

What is a web crawler tool?

A web crawler tool is software you run yourself to crawl a site and report what it finds, rather than a crawler run by a search engine. SEO auditing tools use this approach. They follow your internal links the way a search crawler would, then flag broken links, redirect chains, blocked paths and pages that cannot be reached from anywhere else on the site. The result shows you the site roughly as a crawler sees it, which is useful for spotting access problems before they cost you visibility. A tool crawl is a simulation, though. It does not tell you whether Googlebot or an AI crawler has actually visited.

Where to go deeper on crawler access

To control which crawlers can reach your pages, read Does Robots.txt Block AI Crawlers?, which covers how each rule is matched to a crawler name. To test whether one specific AI crawler can get in, use Can ChatGPT Crawl My Website?, which walks through checking access and fixing it.