KinetixSEO crawl study
AI training crawler blocking among public B2B SaaS homepages
7% of the 45 public B2B SaaS homepages block at least one AI training crawler in robots.txt.
Measured · n = 45 · cohort: public B2B SaaS homepages
The numbers
3 of 45 (6.7%)
- Block at least one AI training crawler
- 3
- Allow all five AI training crawlers
- 42
- Sites observed (sample size)
- 45
- Listed but unreachable on the day (excluded)
- 0 of 45
The headline rounds to a whole percent (7%); the exact share is 6.7% of the observed sample.
Methodology
How this was measured
Fetched robots.txt for each of the 45 public B2B SaaS homepages and counted the sites carrying a Disallow rule that blocks GPTBot, ClaudeBot, Google-Extended, CCBot, or Applebot-Extended from the site root.
One observation: Whether the site's robots.txt blocks GPTBot, ClaudeBot, Google-Extended, CCBot or Applebot-Extended from the site root.
How the cohort was chosen: Hand-curated 2026-09-04 from widely cited public "most used B2B SaaS products" listings. Every URL is the vendor's own public homepage; no customer data and no private endpoint is involved.
Every request was a single polite fetch per site with a short timeout and no retries. Only aggregate counts are published — no per-site result.
Limitations
What this figure does not tell you
- The cohort is a named list, not a random sample of the web. The share describes these sites, not every site.
- It is one measurement on the date shown. Any site can change its setup the next day.
- Only robots.txt was read. A site can also block AI crawlers at its CDN or firewall, which this check cannot see. Blocking a training crawler keeps content out of model training but does not stop the separate search crawlers that feed AI answers.
How to cite
Use this data
KinetixSEO (2026). AI training crawler blocking among public B2B SaaS homepages [Data set]. KinetixSEO crawl study, measured 23 October 2026, n = 45. https://kinetixseo.com/learn/studies/saas-homepages-ai-training-block
Published under a CC BY 4.0 license: free to reuse with attribution. Download the aggregate results as CSV (every published run of this study).
Every published run
Run history
| Measured | Block at least one AI training crawler | Share | Sample size | Unreachable |
|---|---|---|---|---|
| 3 | 6.7% | n = 45 | 0 of 45 |
Where does your own site land?
Run the same kinds of checks on your own page with a free SEO and GEO audit. No signup.