Does Robots.txt Block AI Crawlers? A Technical Guide

The confusion around this topic usually comes from one of two mistakes: assuming a generic Disallow: / under User-agent: * only affects Googlebot and Bingbot (it doesn't — it blocks everything that checks robots.txt and finds no more specific rule for itself), or assuming that blocking "AI" is one setting rather than a list of separately named crawlers that each need their own line. In a KinetixSEO research sweep that scanned the homepage robots.txt of 332 public sites between October 20 and October 20, 2026, 11% blocked at least one AI crawler — meaning the large majority leave every AI crawler free to fetch their content by default, often without having made that choice deliberately.

How robots.txt actually works with AI crawlers

Robots.txt is a voluntary protocol: it tells well-behaved crawlers what they're permitted to fetch, and compliant bots check it before requesting pages. It does not technically prevent access — a crawler that ignores robots.txt can still fetch your pages unless you block it at the server or firewall level. The major AI crawlers from OpenAI, Anthropic, Google, and Perplexity all state that they honor robots.txt, which is why the file is a meaningful control point even though it's not enforcement in the security sense.

The matching logic works like this: a crawler looks for a User-agent block that matches its own token exactly. If none exists, it falls back to the User-agent: * block. Rules are scoped per user-agent group, not globally, so a Disallow written under User-agent: Googlebot has zero effect on GPTBot, and vice versa. The same mechanics apply whether you're writing rules for classic search crawling — covered in how Googlebot crawls your site — or for the newer wave of AI bots, which just means more named blocks to track.

The user-agent tokens the major AI crawlers use

To control access, you need the exact token each system announces itself with. The primary ones in active use:

• GPTBot — OpenAI's crawler that gathers content to train and improve its models, detailed further in what GPTBot is.
• ChatGPT-User — OpenAI's separate agent used when a live ChatGPT user asks the assistant to browse or fetch a specific page in real time.
• OAI-SearchBot — OpenAI's crawler that powers search-style features and citations inside ChatGPT.
• ClaudeBot — Anthropic's general crawler for Claude, used for training data and retrieval.
• PerplexityBot — Perplexity's crawler, used to fetch and index content that Perplexity's answer engine can cite.
• Google-Extended — a control token Google offers specifically to govern whether content can be used for Gemini and other AI features, separate from Googlebot's normal search-indexing crawl, as explained in Googlebot vs. Google-Extended.
• CCBot and Applebot-Extended — the Common Crawl bot (whose archives feed many third-party model-training sets) and Apple's token for controlling AI-training use of your content separately from standard Applebot indexing.

These are distinct identities with distinct purposes — a crawler that fetches pages for real-time user requests (like ChatGPT-User) behaves differently and carries different tradeoffs than one that bulk-collects pages for model training (like GPTBot or ClaudeBot). Treating "AI crawlers" as a single monolith means you can't make that distinction in your rules, even though the underlying protocol supports it perfectly well. You can check which of these tokens your own site currently allows with a AI Crawler Checker.

Why Disallow: / blocks everything, not just search engines

A bare Disallow: / under User-agent: * blocks every crawler that doesn't have its own more specific rule elsewhere in the file — that includes GPTBot, ClaudeBot, PerplexityBot, and any other bot checking robots.txt, not just Googlebot and Bingbot. This is the single most common unintended outcome site owners run into: a rule added years ago to stop scrapers, or copied from a boilerplate template, ends up silently blocking every AI crawler that respects the protocol, with no one having made that decision on purpose.

The fix isn't to remove the wildcard block — it's to make your intent explicit. If you want classic search engines indexed but AI crawlers blocked, or the reverse, you need separate User-agent blocks for each token you care about, because a wildcard rule and a named-bot rule don't automatically talk to each other the way you might expect. Cloudflare Radar found that across the 200,000 most-visited domains, 78% publish a robots.txt file at all — but only 4% publish the newer Content Signals extension that states usage preferences more explicitly, which suggests most site owners are still relying on basic Allow/Disallow logic rather than finer controls.

Writing scoped rules: allow search, control AI selectively

The reliable pattern is to give every crawler you have an opinion about its own named block, and let the wildcard group handle only the bots you haven't thought about individually. This is list-shaped work, not prose, so it helps to think of it as a sequence:

Writing scoped robots.txt rules for AI crawlers

1. 1

   Allow classic search crawlers explicitly

   Named blocks for Googlebot and Bingbot with Allow: /

2. 2

   Add a named block per AI crawler

   Disallow or Allow each token individually, e.g. GPTBot, ClaudeBot, PerplexityBot

3. 3

   Set the wildcard block deliberately

   User-agent: * covers any bot you haven't named

4. 4

   Re-check after CMS or template changes

   Some platforms silently regenerate robots.txt

1. Start with an explicit Allow: / block for each classic search crawler you rely on (Googlebot, Bingbot), so their access isn't left to inference.
2. Add a named block for every AI crawler you have an opinion about, with Disallow: / for the ones you don't want and Allow: / for the ones you do.
3. Leave the User-agent: * block permissive (Allow: /) only if you're comfortable with any bot you haven't named explicitly getting default access.
4. Re-check the file after any CMS, theme, or template change, since some platforms regenerate robots.txt automatically.

To allow classic search crawlers while blocking specific AI bots from training on your content:

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /

To allow AI crawlers that can generate citations back to your site while blocking ones you don't want training on your content, apply the same structure but split the AI bots by purpose rather than blocking them all — for example, allowing OAI-SearchBot and PerplexityBot (which can drive attributed citations) while disallowing GPTBot and ClaudeBot (which primarily feed training data with no attribution mechanism). Because each token gets matched independently, you can make that distinction cleanly instead of choosing one blanket stance for "AI" as a category. A AI Crawler Robots.txt Generator can produce this structure for you token by token, and a robots.txt / sitemap.xml Validator checks that the resulting file parses the way you intended before you publish it.

One detail that trips people up: Disallow: / inside a bot's own named block still fully blocks that bot, even if a separate User-agent: * block says Allow: /. Specificity wins — a matching named block is used instead of the wildcard block, not in addition to it.

The real tradeoff: visibility in AI answers vs. protecting your content

Blocking AI crawlers is a legitimate choice, not a mistake, and it comes with a real cost on the other side. When you disallow GPTBot, ClaudeBot, or similar training-oriented crawlers, you keep your content from being absorbed into model training data and reused in ways you can't track or control — a genuine concern for publishers, businesses with proprietary detail, or anyone uneasy about their work resurfacing in an AI answer with no link back. Readers weighing this tradeoff in more detail, including the legal and practical mechanics of removal requests, can see how to opt out of AI training data.

The tradeoff is that you also remove yourself from consideration entirely for answer engines that cite sources — if PerplexityBot or OAI-SearchBot can't fetch your pages, your content can't be quoted, linked, or credited in the answers those systems generate, even when a user's question is one your page directly answers. There is no setting that gets you cited without being crawled; the two are the same mechanism. That gap is visible in a tracked prompt KinetixSEO ran on 17 October 2026: asking Gemini web search "what tools track brand mentions and citations across AI assistants like ChatGPT and Perplexity, name specific products" returned 14 cited sources — viali.ai, mersel.ai, noqode.fr, tryprofound.com, ziptie.ai and nine more — with kinetixseo.com absent from the list. Being crawlable is necessary for citation, but it isn't sufficient on its own; a crawlable page still has to be the one the model chooses to surface.

This is why the decision benefits from being made per-bot rather than as one global toggle. A page that depends on ranking in classic search but never wants AI reuse might block every AI token and keep Googlebot/Bingbot fully allowed. A brand actively trying to earn AI citations might allow the search-and-retrieval bots (OAI-SearchBot, PerplexityBot) while still blocking the pure-training crawlers (GPTBot, ClaudeBot) that offer no attribution back. Neither approach is universally correct — it depends on whether your traffic and revenue model benefits more from being found through AI answers or from keeping content exclusively behind your own indexed pages. If you're already crawled but not showing up in answers, that's a separate problem covered in AI crawls my site but doesn't cite it.

How to verify your robots.txt is doing what you intend

Check the live file at yourdomain.com/robots.txt and read it the way a crawler would: find the block matching each bot's exact token, and if none exists, apply the wildcard block to it. Don't assume a rule you wrote for one bot silently extends to another — GPTBot, ChatGPT-User, and OAI-SearchBot are three separate identities from the same company, and a rule addressing one doesn't touch the others unless you add it explicitly. As a worked example, KinetixSEO ran its own "ai training crawler robots block" check against its own robots.txt on 23 October 2026, testing whether the file blocks GPTBot, ClaudeBot, Google-Extended, CCBot, or Applebot-Extended from the site root — the result showed all five allowed, confirming the file matched the site's intent to stay crawlable by AI training bots.

A AI Crawler Access Checker can run the same kind of per-bot check against any domain, which is worth doing after any CMS, theme, or template change — some platforms regenerate robots.txt automatically and can quietly reintroduce a blanket Disallow: / under User-agent: * that reverses a deliberate AI-access decision you made earlier.

Frequently asked questions

Does a Disallow: / for User-agent: * block ChatGPT and other AI bots?

Yes, if no more specific block exists for that bot's own token. A wildcard Disallow: / applies to every crawler that doesn't have its own named User-agent section, which includes GPTBot, ChatGPT-User, ClaudeBot, and PerplexityBot by default.

Is blocking AI crawlers in robots.txt actually enforced?

Robots.txt is voluntary, and enforcement depends on the crawler choosing to comply. The major documented AI crawlers state that they respect robots.txt directives, so for those specific bots a correctly written rule is effective in practice, even though the protocol itself has no technical mechanism to force compliance.

What's the difference between GPTBot and ChatGPT-User?

GPTBot is OpenAI's crawler for gathering training and general data, while ChatGPT-User is a separate agent triggered when a live ChatGPT user asks the assistant to fetch or browse a specific page in real time. Because they use different tokens, you can allow one and block the other in the same robots.txt file.

If I block AI crawlers, can my content still show up in AI-generated answers?

No — if a crawler can't fetch your pages, that answer engine has no content of yours to cite or quote. Blocking a bot like PerplexityBot or OAI-SearchBot removes any possibility of attribution from that system, since crawling and citation share the same access mechanism.

Does Google-Extended affect my regular Google Search rankings?

No, Google-Extended is a separate control from Googlebot and only governs whether your content can be used for Google's AI features like Gemini. Blocking Google-Extended does not remove you from standard Google Search indexing, which Googlebot handles independently.

How common is it for sites to block AI crawlers in robots.txt?

It's uncommon — in KinetixSEO's research sweep of 332 public sites scanned on October 20, 2026, only 11% blocked at least one AI crawler in their robots.txt, leaving the vast majority of sites fully open to AI crawling by default, whether or not that was a deliberate choice.