How to Opt Out of AI Training Data

Training crawlers vs. live-retrieval crawlers

Each major AI company typically operates separate bots for these two jobs. One bot crawls the web periodically to gather text for training new model versions. A separate bot fetches pages on demand when a user's query triggers a live search or citation. Blocking the training bot in robots.txt stops your content from entering future training sets, but it does nothing to the retrieval bot — that one keeps fetching pages so the assistant can answer questions and cite sources.

A concrete example

Disallowing OpenAI's training crawler in robots.txt opts your site out of training-data collection, but it does not block the separate crawler OpenAI uses for ChatGPT's live web-search retrieval. If your goal is to stop training use only, blocking the training bot is correct. If your goal is to also keep your pages out of live AI search results, you need to block the retrieval bot too — a distinct decision with different tradeoffs, since blocking retrieval can remove you from AI-generated answers entirely.

For the exact directives to add and which bot names correspond to which purpose, see this technical guide to blocking AI crawlers with robots.txt. Some sites also use llms.txt alongside robots.txt to more explicitly signal how AI systems may use their content.