Does Robots.txt Block AI Crawlers? A Technical Guide

Does robots.txt block AI crawlers?

Yes — robots.txt can block AI crawlers, but only if you address them by their specific user-agent token. Robots.txt is a set of instructions matched against a crawler's declared user-agent string. If a rule doesn't name a bot (or use a wildcard that covers it), that bot has no obligation to obey it. Most major AI companies do publish documented user-agent tokens and do respect robots.txt directives aimed at those tokens, which means the file works — but only as precisely as you write it.

The confusion around this topic usually comes from one of two mistakes: assuming a generic Disallow: / under User-agent: * only affects Googlebot and Bingbot (it doesn't — it blocks everything that checks robots.txt and finds no more specific rule for itself), or assuming that blocking "AI" is one setting rather than a list of separately named crawlers that each need their own line.

How robots.txt actually works with crawlers

Robots.txt is a voluntary protocol: it tells well-behaved crawlers what they're permitted to fetch, and compliant bots check it before requesting pages. It does not technically prevent access — a crawler that ignores robots.txt can still fetch your pages unless you block it at the server or firewall level. The major AI crawlers from OpenAI, Anthropic, Google, and Perplexity all state that they honor robots.txt, which is why the file is a meaningful control point even though it's not enforcement in the security sense.

The matching logic works like this: a crawler looks for a User-agent block that matches its own token exactly. If none exists, it falls back to the User-agent: * block. Rules are scoped per user-agent group, not globally, so a Disallow written under User-agent: Googlebot has zero effect on GPTBot, and vice versa.

The user-agent tokens the major AI crawlers use

To control access, you need the exact token each system announces itself with. The primary ones in active use:

• GPTBot — OpenAI's crawler that gathers content to train and improve its models.
• ChatGPT-User — OpenAI's separate agent used when a live ChatGPT user asks the assistant to browse or fetch a specific page in real time.
• OAI-SearchBot — OpenAI's crawler that powers search-style features and citations inside ChatGPT.
• ClaudeBot — Anthropic's general crawler for Claude, used for training data and retrieval.
• PerplexityBot — Perplexity's crawler, used to fetch and index content that Perplexity's answer engine can cite.
• Google-Extended — a control token Google offers specifically to govern whether content can be used for Gemini and other AI features, separate from Googlebot's normal search-indexing crawl.

These are distinct identities with distinct purposes — a crawler that fetches pages for real-time user requests (like ChatGPT-User) behaves differently and carries different tradeoffs than one that bulk-collects pages for model training (like GPTBot or ClaudeBot). Treating "AI crawlers" as a single monolith means you can't make that distinction in your rules, even though the underlying protocol supports it perfectly well.

Why Disallow: / blocks everything, not just search engines

A bare Disallow: / under User-agent: * blocks every crawler that doesn't have its own more specific rule elsewhere in the file — that includes GPTBot, ClaudeBot, PerplexityBot, and any other bot checking robots.txt, not just Googlebot and Bingbot. This is the single most common unintended outcome site owners run into: a rule added years ago to stop scrapers, or copied from a boilerplate template, ends up silently blocking every AI crawler that respects the protocol, with no one having made that decision on purpose.

The fix isn't to remove the wildcard block — it's to make your intent explicit. If you want classic search engines indexed but AI crawlers blocked, or the reverse, you need separate User-agent blocks for each token you care about, because a wildcard rule and a named-bot rule don't automatically talk to each other the way you might expect.

Writing scoped rules: allow search, control AI selectively

The reliable pattern is to give every crawler you have an opinion about its own named block, and let the wildcard group handle only the bots you haven't thought about individually.

To allow classic search crawlers while blocking specific AI bots from training on your content:

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /

To allow AI crawlers that can generate citations back to your site while blocking ones you don't want training on your content, apply the same structure but split the AI bots by purpose rather than blocking them all — for example, allowing OAI-SearchBot and PerplexityBot (which can drive attributed traffic) while disallowing GPTBot and ClaudeBot (which primarily feed training data with no attribution mechanism). Because each token gets matched independently, you can make that distinction cleanly instead of choosing one blanket stance for "AI" as a category.

One detail that trips people up: Disallow: / inside a bot's own named block still fully blocks that bot, even if a separate User-agent: * block says Allow: /. Specificity wins — a matching named block is used instead of the wildcard block, not in addition to it.

The real tradeoff: visibility in AI answers vs. protecting your content

Blocking AI crawlers is a legitimate choice, not a mistake, and it comes with a real cost on the other side. When you disallow GPTBot, ClaudeBot, or similar training-oriented crawlers, you keep your content from being absorbed into model training data and reused in ways you can't track or control — a genuine concern for publishers, businesses with proprietary detail, or anyone uneasy about their work resurfacing in an AI answer with no link back.

The tradeoff is that you also remove yourself from consideration entirely for answer engines that cite sources — if PerplexityBot or OAI-SearchBot can't fetch your pages, your content can't be quoted, linked, or credited in the answers those systems generate, even when a user's question is one your page directly answers. There is no setting that gets you cited without being crawled; the two are the same mechanism.

This is why the decision benefits from being made per-bot rather than as one global toggle. A page that depends on ranking in classic search but never wants AI reuse might block every AI token and keep Googlebot/Bingbot fully allowed. A brand actively trying to earn AI citations might allow the search-and-retrieval bots (OAI-SearchBot, PerplexityBot) while still blocking the pure-training crawlers (GPTBot, ClaudeBot) that offer no attribution back. Neither approach is universally correct — it depends on whether your traffic and revenue model benefits more from being found through AI answers or from keeping content exclusively behind your own indexed pages.

How to verify your robots.txt is doing what you intend

Check the live file at yourdomain.com/robots.txt and read it the way a crawler would: find the block matching each bot's exact token, and if none exists, apply the wildcard block to it. Don't assume a rule you wrote for one bot silently extends to another — GPTBot, ChatGPT-User, and OAI-SearchBot are three separate identities from the same company, and a rule addressing one doesn't touch the others unless you add it explicitly.

It's also worth testing after any template, CMS, or theme change, since some platforms regenerate robots.txt automatically and can quietly reintroduce a blanket Disallow: / under User-agent: * that reverses a deliberate AI-access decision you made earlier.

Frequently asked questions

Does a Disallow: / for User-agent: * block ChatGPT and other AI bots?

Yes, if no more specific block exists for that bot's own token. A wildcard Disallow: / applies to every crawler that doesn't have its own named User-agent section, which includes GPTBot, ChatGPT-User, ClaudeBot, and PerplexityBot by default.

Is blocking AI crawlers in robots.txt actually enforced?

Robots.txt is voluntary, and enforcement depends on the crawler choosing to comply. The major documented AI crawlers state that they respect robots.txt directives, so for those specific bots a correctly written rule is effective in practice, even though the protocol itself has no technical mechanism to force compliance.

What's the difference between GPTBot and ChatGPT-User?

GPTBot is OpenAI's crawler for gathering training and general data, while ChatGPT-User is a separate agent triggered when a live ChatGPT user asks the assistant to fetch or browse a specific page in real time. Because they use different tokens, you can allow one and block the other in the same robots.txt file.

If I block AI crawlers, can my content still show up in AI-generated answers?

No — if a crawler can't fetch your pages, that answer engine has no content of yours to cite or quote. Blocking a bot like PerplexityBot or OAI-SearchBot removes any possibility of attribution from that system, since crawling and citation share the same access mechanism.

Does Google-Extended affect my regular Google Search rankings?

No, Google-Extended is a separate control from Googlebot and only governs whether your content can be used for Google's AI features like Gemini. Blocking Google-Extended does not remove you from standard Google Search indexing, which Googlebot handles independently.