# Content-Signal (contentsignals.org / Cloudflare, 2025) — a robots.txt-level # declaration of what may be done with this content, independent of which bot fetches # it. Deliberately permissive on all three axes, matching /.well-known/rsl.json # (permits ai-training) and /ai.txt (Training: allow); PLAN_137 §4.7. # # NOTE: this must stay an UNCOMMENTED directive line. Our own detector # (FillerGeoReadinessAgent::probeContentSignal, /^\s*Content-Signal\s*:/mi) and # Cloudflare's published format both read it bare — a leading # makes it invisible. Content-Signal: search=yes, ai-input=yes, ai-train=yes # Named AI-bot groups. A crawler that matches its own User-agent group reads ONLY that # group and ignores `User-agent: *` entirely, so every named group below carries an # identical copy of the wildcard group's Disallow list. Without it these bots would be # free to crawl the auth-gated app and per-customer result pages (/seo-check/, /results, # /dashboard ...). Add or remove a path in EVERY group in the same edit — # CitationBotPolicyFilesTest pins every named group to the wildcard list. User-agent: GPTBot Allow: / Disallow: /seo-check/ Disallow: /admin Disallow: /dashboard Disallow: /profile Disallow: /tokens Disallow: /results Disallow: /billing Disallow: /auth User-agent: OAI-SearchBot Allow: / Disallow: /seo-check/ Disallow: /admin Disallow: /dashboard Disallow: /profile Disallow: /tokens Disallow: /results Disallow: /billing Disallow: /auth User-agent: ChatGPT-User Allow: / Disallow: /seo-check/ Disallow: /admin Disallow: /dashboard Disallow: /profile Disallow: /tokens Disallow: /results Disallow: /billing Disallow: /auth User-agent: ClaudeBot Allow: / Disallow: /seo-check/ Disallow: /admin Disallow: /dashboard Disallow: /profile Disallow: /tokens Disallow: /results Disallow: /billing Disallow: /auth User-agent: Claude-Web Allow: / Disallow: /seo-check/ Disallow: /admin Disallow: /dashboard Disallow: /profile Disallow: /tokens Disallow: /results Disallow: /billing Disallow: /auth User-agent: Claude-SearchBot Allow: / Disallow: /seo-check/ Disallow: /admin Disallow: /dashboard Disallow: /profile Disallow: /tokens Disallow: /results Disallow: /billing Disallow: /auth User-agent: Claude-User Allow: / Disallow: /seo-check/ Disallow: /admin Disallow: /dashboard Disallow: /profile Disallow: /tokens Disallow: /results Disallow: /billing Disallow: /auth User-agent: Anthropic-AI Allow: / Disallow: /seo-check/ Disallow: /admin Disallow: /dashboard Disallow: /profile Disallow: /tokens Disallow: /results Disallow: /billing Disallow: /auth User-agent: Google-Extended Allow: / Disallow: /seo-check/ Disallow: /admin Disallow: /dashboard Disallow: /profile Disallow: /tokens Disallow: /results Disallow: /billing Disallow: /auth User-agent: Google-Agent Allow: / Disallow: /seo-check/ Disallow: /admin Disallow: /dashboard Disallow: /profile Disallow: /tokens Disallow: /results Disallow: /billing Disallow: /auth User-agent: Google-CloudVertexBot Allow: / Disallow: /seo-check/ Disallow: /admin Disallow: /dashboard Disallow: /profile Disallow: /tokens Disallow: /results Disallow: /billing Disallow: /auth User-agent: Google-NotebookLM Allow: / Disallow: /seo-check/ Disallow: /admin Disallow: /dashboard Disallow: /profile Disallow: /tokens Disallow: /results Disallow: /billing Disallow: /auth User-agent: PerplexityBot Allow: / Disallow: /seo-check/ Disallow: /admin Disallow: /dashboard Disallow: /profile Disallow: /tokens Disallow: /results Disallow: /billing Disallow: /auth User-agent: Perplexity-User Allow: / Disallow: /seo-check/ Disallow: /admin Disallow: /dashboard Disallow: /profile Disallow: /tokens Disallow: /results Disallow: /billing Disallow: /auth User-agent: Meta-ExternalAgent Allow: / Disallow: /seo-check/ Disallow: /admin Disallow: /dashboard Disallow: /profile Disallow: /tokens Disallow: /results Disallow: /billing Disallow: /auth User-agent: Meta-ExternalFetcher Allow: / Disallow: /seo-check/ Disallow: /admin Disallow: /dashboard Disallow: /profile Disallow: /tokens Disallow: /results Disallow: /billing Disallow: /auth User-agent: FacebookBot Allow: / Disallow: /seo-check/ Disallow: /admin Disallow: /dashboard Disallow: /profile Disallow: /tokens Disallow: /results Disallow: /billing Disallow: /auth User-agent: Amazonbot Allow: / Disallow: /seo-check/ Disallow: /admin Disallow: /dashboard Disallow: /profile Disallow: /tokens Disallow: /results Disallow: /billing Disallow: /auth User-agent: Applebot Allow: / Disallow: /seo-check/ Disallow: /admin Disallow: /dashboard Disallow: /profile Disallow: /tokens Disallow: /results Disallow: /billing Disallow: /auth User-agent: Applebot-Extended Allow: / Disallow: /seo-check/ Disallow: /admin Disallow: /dashboard Disallow: /profile Disallow: /tokens Disallow: /results Disallow: /billing Disallow: /auth User-agent: Bytespider Allow: / Disallow: /seo-check/ Disallow: /admin Disallow: /dashboard Disallow: /profile Disallow: /tokens Disallow: /results Disallow: /billing Disallow: /auth User-agent: CCBot Allow: / Disallow: /seo-check/ Disallow: /admin Disallow: /dashboard Disallow: /profile Disallow: /tokens Disallow: /results Disallow: /billing Disallow: /auth User-agent: cohere-ai Allow: / Disallow: /seo-check/ Disallow: /admin Disallow: /dashboard Disallow: /profile Disallow: /tokens Disallow: /results Disallow: /billing Disallow: /auth User-agent: MistralAI-User Allow: / Disallow: /seo-check/ Disallow: /admin Disallow: /dashboard Disallow: /profile Disallow: /tokens Disallow: /results Disallow: /billing Disallow: /auth User-agent: DuckAssistBot Allow: / Disallow: /seo-check/ Disallow: /admin Disallow: /dashboard Disallow: /profile Disallow: /tokens Disallow: /results Disallow: /billing Disallow: /auth User-agent: Diffbot Allow: / Disallow: /seo-check/ Disallow: /admin Disallow: /dashboard Disallow: /profile Disallow: /tokens Disallow: /results Disallow: /billing Disallow: /auth User-agent: xAI-Bot Allow: / Disallow: /seo-check/ Disallow: /admin Disallow: /dashboard Disallow: /profile Disallow: /tokens Disallow: /results Disallow: /billing Disallow: /auth User-agent: bingbot Allow: / Disallow: /seo-check/ Disallow: /admin Disallow: /dashboard Disallow: /profile Disallow: /tokens Disallow: /results Disallow: /billing Disallow: /auth User-agent: Meta-WebIndexer Allow: / Disallow: /seo-check/ Disallow: /admin Disallow: /dashboard Disallow: /profile Disallow: /tokens Disallow: /results Disallow: /billing Disallow: /auth User-agent: GoogleAgent-Mariner Allow: / Disallow: /seo-check/ Disallow: /admin Disallow: /dashboard Disallow: /profile Disallow: /tokens Disallow: /results Disallow: /billing Disallow: /auth User-agent: GoogleAgent-URLContext Allow: / Disallow: /seo-check/ Disallow: /admin Disallow: /dashboard Disallow: /profile Disallow: /tokens Disallow: /results Disallow: /billing Disallow: /auth User-agent: Gemini-Deep-Research Allow: / Disallow: /seo-check/ Disallow: /admin Disallow: /dashboard Disallow: /profile Disallow: /tokens Disallow: /results Disallow: /billing Disallow: /auth User-agent: YouBot Allow: / Disallow: /seo-check/ Disallow: /admin Disallow: /dashboard Disallow: /profile Disallow: /tokens Disallow: /results Disallow: /billing Disallow: /auth User-agent: kagi-fetcher Allow: / Disallow: /seo-check/ Disallow: /admin Disallow: /dashboard Disallow: /profile Disallow: /tokens Disallow: /results Disallow: /billing Disallow: /auth User-agent: DeepSeekBot Allow: / Disallow: /seo-check/ Disallow: /admin Disallow: /dashboard Disallow: /profile Disallow: /tokens Disallow: /results Disallow: /billing Disallow: /auth User-agent: Kimi-User Allow: / Disallow: /seo-check/ Disallow: /admin Disallow: /dashboard Disallow: /profile Disallow: /tokens Disallow: /results Disallow: /billing Disallow: /auth User-agent: TongyiBot Allow: / Disallow: /seo-check/ Disallow: /admin Disallow: /dashboard Disallow: /profile Disallow: /tokens Disallow: /results Disallow: /billing Disallow: /auth User-agent: YiyanBot Allow: / Disallow: /seo-check/ Disallow: /admin Disallow: /dashboard Disallow: /profile Disallow: /tokens Disallow: /results Disallow: /billing Disallow: /auth User-agent: PetalBot Allow: / Disallow: /seo-check/ Disallow: /admin Disallow: /dashboard Disallow: /profile Disallow: /tokens Disallow: /results Disallow: /billing Disallow: /auth User-agent: PanguBot Allow: / Disallow: /seo-check/ Disallow: /admin Disallow: /dashboard Disallow: /profile Disallow: /tokens Disallow: /results Disallow: /billing Disallow: /auth User-agent: YandexAdditional Allow: / Disallow: /seo-check/ Disallow: /admin Disallow: /dashboard Disallow: /profile Disallow: /tokens Disallow: /results Disallow: /billing Disallow: /auth User-agent: AI2Bot Allow: / Disallow: /seo-check/ Disallow: /admin Disallow: /dashboard Disallow: /profile Disallow: /tokens Disallow: /results Disallow: /billing Disallow: /auth User-agent: Ai2Bot-Dolma Allow: / Disallow: /seo-check/ Disallow: /admin Disallow: /dashboard Disallow: /profile Disallow: /tokens Disallow: /results Disallow: /billing Disallow: /auth User-agent: omgilibot Allow: / Disallow: /seo-check/ Disallow: /admin Disallow: /dashboard Disallow: /profile Disallow: /tokens Disallow: /results Disallow: /billing Disallow: /auth User-agent: Webzio-Extended Allow: / Disallow: /seo-check/ Disallow: /admin Disallow: /dashboard Disallow: /profile Disallow: /tokens Disallow: /results Disallow: /billing Disallow: /auth User-agent: Timpibot Allow: / Disallow: /seo-check/ Disallow: /admin Disallow: /dashboard Disallow: /profile Disallow: /tokens Disallow: /results Disallow: /billing Disallow: /auth User-agent: TikTokSpider Allow: / Disallow: /seo-check/ Disallow: /admin Disallow: /dashboard Disallow: /profile Disallow: /tokens Disallow: /results Disallow: /billing Disallow: /auth User-agent: cohere-training-data-crawler Allow: / Disallow: /seo-check/ Disallow: /admin Disallow: /dashboard Disallow: /profile Disallow: /tokens Disallow: /results Disallow: /billing Disallow: /auth User-agent: ImagesiftBot Allow: / Disallow: /seo-check/ Disallow: /admin Disallow: /dashboard Disallow: /profile Disallow: /tokens Disallow: /results Disallow: /billing Disallow: /auth User-agent: Brightbot Allow: / Disallow: /seo-check/ Disallow: /admin Disallow: /dashboard Disallow: /profile Disallow: /tokens Disallow: /results Disallow: /billing Disallow: /auth User-agent: Diffbot-User Allow: / Disallow: /seo-check/ Disallow: /admin Disallow: /dashboard Disallow: /profile Disallow: /tokens Disallow: /results Disallow: /billing Disallow: /auth User-agent: ExaSearchBot Allow: / Disallow: /seo-check/ Disallow: /admin Disallow: /dashboard Disallow: /profile Disallow: /tokens Disallow: /results Disallow: /billing Disallow: /auth User-agent: TavilyBot Allow: / Disallow: /seo-check/ Disallow: /admin Disallow: /dashboard Disallow: /profile Disallow: /tokens Disallow: /results Disallow: /billing Disallow: /auth # Googlebot, explicitly (PLAN_157 Phase 0). Googlebot is the ONLY crawler behind AI # Overviews and AI Mode — Google-Extended gates Gemini-app grounding and training, not # AIO — so it belongs in the citation-gate allowlist alongside OAI-SearchBot, # PerplexityBot, Claude-SearchBot and bingbot, all of which already have a named block # above. From 15 Sep 2026 Cloudflare's default posture treats "mixed-use" crawlers # (Googlebot and bingbot among them) as blockable, which makes an explicit, auditable # Allow worth having rather than relying on the wildcard group by implication. # # This group also gates Brave Search. Brave advertises no user agent of its own and only # crawls pages Googlebot is allowed to crawl (search.brave.com/help/brave-search-crawler), # and Brave is the search index behind Claude's web search — so tightening the Disallow # lines below, or blocking Googlebot, also removes those pages from Brave and from Claude # citations. robots.txt cannot deindex a page from Brave; only noindex does. # # The Disallow lines below are NOT optional padding. A named group means Googlebot stops # reading the `User-agent: *` group entirely, so without them Google would start crawling # the auth-gated app and per-customer result pages the wildcard group exists to keep out. # They are a deliberate copy of that group's list and must stay identical to it — # CitationBotPolicyFilesTest pins both sets to the same reviewed prefixes. # # `Allow: /` and `Disallow: /admin` do not conflict: both Google and Bing resolve a # path against the LONGEST matching rule, so `/admin` (the more specific rule) wins for # `/admin/...` while `/` allows everything else. User-agent: Googlebot Allow: / Disallow: /seo-check/ Disallow: /admin Disallow: /dashboard Disallow: /profile Disallow: /tokens Disallow: /results Disallow: /billing Disallow: /auth # Auth-gated app areas and per-customer result pages: nothing here is useful to a # crawler and every fetch burns crawl budget. A bot matching its own User-agent group # never reads this one, which is why every named group above repeats this exact list. # # Deliberately NOT disallowed: # /login, /register, /forgot-password, /reset-password — these carry # instead (PLAN_133 §5 P2). # Disallowing them too would stop a crawler fetching the page at all, so it # could never see the noindex and could still index the URL from inbound # links. Pick one signal; noindex is the one that actually deindexes. # /feed, /rss.xml — excluded from the sitemap (a feed is not an HTML page) but # deliberately crawlable: the feed is a discovery surface, and head # autodiscovery points at it. # /changelog/{version} — release notes are public product history and a legitimate # long-tail surface ("KinetixSEO 1.3.0"), so they are crawlable AND sitemap-listed. # They used to be disallowed here; that was a mixed signal against submitting # /changelog itself. # # /auth is the OAuth *initiation* endpoint: every fetch 302s to the provider with our # client_id and mints a throwaway `state` value. Nothing to index, and a crawler hitting # it generates server-side session churn for no reason. User-agent: * Disallow: /seo-check/ Disallow: /admin Disallow: /dashboard Disallow: /profile Disallow: /tokens Disallow: /results Disallow: /billing Disallow: /auth Sitemap: https://kinetixseo.com/sitemap.xml