How Googlebot Crawls Your Site: Budget, Waste & Checks

Crawl budget is really two things combined: crawl capacity (how much load your server can handle without degrading) and crawl demand (how much Google actually wants to fetch, based on perceived value and freshness). Both move independently. You can raise your server's capacity and still get crawled less if Google doesn't think your pages change often enough to justify revisiting them. You can also have high-demand content and still get under-crawled if your infrastructure keeps signaling that fetching it is risky or slow. Google's own crawling documentation describes this same split between capacity and demand, and confirms that Googlebot deliberately throttles itself in response to server strain rather than pushing through it.

What actually wastes crawl budget

Two of the most common budget-drainers are slow server response times and long redirect chains, and both are entirely within your control.

Slow servers throttle Googlebot on purpose. Googlebot adjusts its fetch rate based on how your server responds. If response times climb or you start returning 5xx errors, Googlebot backs off to avoid making things worse — which means fewer pages get crawled per visit, not more patience. A site that consistently responds slowly trains Googlebot to request less from it over time, and that reduced rate doesn't bounce back the moment your server improves; it recovers gradually as Googlebot re-establishes trust in your capacity.

Redirect chains burn budget on nothing. Every hop in a redirect chain (A → B → C → final URL) is a separate request Googlebot has to make before it ever reaches content worth indexing. A single 301 is cheap. Three or four chained redirects — often the leftover of years of site migrations, URL restructuring, or plugin-driven redirect rules — multiply the number of fetches needed to resolve one URL, and that cost is paid on every crawl, not just once. The fix is mechanical: audit your redirect map periodically and point old URLs directly at their final destination instead of letting chains accumulate.

Other common drains worth checking: faceted navigation and URL parameters that generate near-infinite low-value URL combinations, soft 404s that look like valid pages but return no real content, and orphaned or duplicate pages that Googlebot keeps revisiting out of habit even though nothing new is there. None of these are exotic problems — they're the same fundamentals covered in technical SEO basics, just viewed through the lens of crawl efficiency rather than indexing.

How to check what Googlebot is actually doing

Stop guessing and look at the Crawl Stats report in Search Console — it's the one place Google tells you directly what its crawler did on your site, rather than you inferring it from server logs or ranking changes. Google's Search Console Help documentation describes the report as covering six areas: total crawl requests, total download size, average response time, a breakdown by response code, a breakdown by file type and Googlebot type, and host status.

The report, found under Settings > Crawl Stats for a verified property, is worth reading as a diagnostic sequence rather than six unrelated charts. Total crawl requests over the last 90 days shows whether crawl activity is trending up, flat, or dropping. Total download size flags whether Googlebot is spending disproportionate bandwidth on large files rather than your actual pages. Average response time is the clearest early-warning signal for the server-slowness problem described above — a rising trend line here should be treated as a technical SEO incident, not background noise. The breakdown by response code (200s vs. redirects vs. 4xx vs. 5xx) surfaces redirect-chain and broken-link problems at a glance. The breakdown by file type and by Googlebot type (desktop vs. smartphone vs. image vs. video) shows which crawler variant is doing the work. And the host status section flags whether Google had trouble fetching your robots.txt file or resolving your DNS — problems that can suppress crawling site-wide before content is even considered.

If your average response time is climbing at the same time your total crawl requests are falling, that's not a coincidence — that's Googlebot doing exactly what Google's documentation says it's designed to do: retreating from a site that looks like it's struggling to keep up. Conversely, a spike in 4xx or 5xx codes after a migration or a CMS change is usually the first hard evidence you get that a redirect map didn't hold up, days before rankings would show any effect.

Googlebot vs. Google-Extended: two different crawlers, two different jobs

Googlebot and Google-Extended are not the same crawler and they do not serve the same purpose, even though the two get conflated constantly in casual SEO discussion.

Googlebot is the crawler behind classic Google Search — it's what fetches and renders your pages so they can be indexed and shown in search results. Blocking Googlebot in robots.txt means disappearing from Google Search entirely. Google-Extended, by contrast, is a separate token that controls whether your content can be used to train Google's generative AI models and to power AI features like AI Overviews — it has nothing to do with whether you rank in traditional search. You can allow Googlebot and block Google-Extended, or vice versa (though blocking Googlebot is rarely advisable); they're independent toggles governing independent uses of your content.

This distinction matters because the two directives live in the same robots.txt file but control entirely different outcomes, and a copy-pasted rule meant for one can accidentally disable the other. If you want the full walkthrough of what each token controls and how the underlying definitions differ, the Googlebot glossary entry lays out the distinction in detail, and if you're specifically deciding whether and how to block AI crawlers without accidentally blocking search, the robots.txt AI crawler guide walks through the syntax and the tradeoffs.

Putting it together: a practical crawl health checklist

None of this requires guesswork — it requires checking the right report on a regular cadence and fixing the same handful of mechanical issues whenever they reappear.

Start by treating Crawl Stats as a recurring check, not a one-time audit: a site that changes frequently (new products, new articles, restructured categories) needs this reviewed more often than a static brochure site, because crawl demand and crawl capacity both drift as a site grows. Pair that with a periodic redirect audit, server response monitoring, and a clear separation in your robots.txt between Googlebot rules and Google-Extended rules so neither gets touched by mistake when you're only trying to adjust the other.

Frequently asked questions

How often does Googlebot crawl a website?

There's no fixed schedule — Googlebot crawls based on perceived demand, adjusting frequency for how often a given site's content actually changes and how much authority the site has accumulated. A news site publishing hourly gets revisited far more often than a static site that hasn't changed in months, and Google adjusts the rate dynamically rather than following a calendar.

Can I increase my crawl budget?

You can increase crawl demand by publishing genuinely new or updated content and building the kind of authority signals that make Google want to revisit more often, and you can protect crawl capacity by improving server response times and eliminating waste like redirect chains and duplicate URLs. Crawl budget isn't something you purchase or request directly — it's an outcome of how efficiently and how valuably your site behaves.

Does blocking Google-Extended affect my search rankings?

No — Google-Extended only governs whether your content is used for AI model training and related generative features, not whether Googlebot can crawl and index your pages for classic search results. Blocking it has no direct effect on search rankings, though it may affect whether your content is eligible to be surfaced in some AI-generated features.

What's the fastest way to tell if crawl budget is being wasted?

Open the Crawl Stats report in Search Console and compare your response-code breakdown against your average response time trend. A high share of redirect or error codes, combined with a climbing response time, is the clearest sign that Googlebot is spending its budget resolving problems instead of reaching indexable content.

Is a low crawl rate always a bad sign?

Not necessarily — a small, stable site that rarely changes doesn't need frequent crawling, and a low but steady crawl rate simply reflects low demand, not a penalty. The warning sign isn't a low number on its own; it's a low or falling number on a site that's actively publishing new content Google isn't picking up.