KinetixSEO crawl study
AI search crawler blocking among public B2B SaaS homepages
0% of the 45 public B2B SaaS homepages still block at least one AI search crawler in robots.txt.
Measured · n = 45 · cohort: public B2B SaaS homepages
Across the web: At least 14% of the websites we scanned block at least one AI crawler in robots.txt (n = 6,574, November 2026). That figure comes from our research sweep, which scans the homepages of a random draw of public websites with its own method, so it is a comparison, not this group measured twice. See How Many Websites Block AI Crawlers?.
The numbers
0 of 45 (0.0%)
- Block at least one AI search crawler
- 0
- Allow all three AI search crawlers
- 45
- Sites observed (sample size)
- 45
- Listed but unreachable on the day (excluded)
- 0 of 45
The headline rounds to a whole percent (0%); the exact share is 0.0% of the observed sample.
Methodology
How this was measured
Fetched robots.txt for each of the 45 public B2B SaaS homepages and counted the sites carrying a Disallow rule that blocks OAI-SearchBot, PerplexityBot, or Claude-SearchBot from the site root.
One observation: Whether the site's robots.txt blocks OAI-SearchBot, PerplexityBot or Claude-SearchBot from the site root.
How the cohort was chosen:Hand-curated 2026-09-04 from widely cited public "most used B2B SaaS products" listings. Every URL is the vendor's own public homepage; no customer data and no private endpoint is involved.
Check your own site: the free AI Crawler Checker reads your robots.txt and shows, per bot, which AI crawlers it blocks.
Every request was a single polite fetch per site with a short timeout and no retries. Only aggregate counts are published — no per-site result.
Limitations
What this figure does not tell you
- The cohort is a named list, not a random sample of the web. The share describes these sites, not every site.
- It is one measurement on the date shown. Any site can change its setup the next day.
- Only robots.txt was read. A site can also block AI crawlers at its CDN or firewall, which this check cannot see, and only the three citation-feeding crawlers were tested, against the site root.
How to cite
Use this data
KinetixSEO (2026). AI search crawler blocking among public B2B SaaS homepages [Data set]. KinetixSEO crawl study, measured 1 November 2026, n = 45. https://kinetixseo.com/learn/studies/saas-homepages-ai-crawler-block
Published under a CC BY 4.0 license: free to reuse with attribution. Download the aggregate results as CSV (every published run of this study).
Every published run
Run history
| Measured | Block at least one AI search crawler | Share | Sample size | Unreachable |
|---|---|---|---|---|
| 0 | 0.0% | n = 45 | 0 of 45 | |
| 0 | 0.0% | n = 45 | 0 of 45 | |
| 0 | 0.0% | n = 45 | 0 of 45 |
More crawl studies
- llms.txt adoption among public SEO and content-marketing tool vendor sites
- Article and FAQPage structured data among public SEO and web-performance guide articles from software vendors and developer publications
- Canonical tags pointing elsewhere among public SEO and web-performance guide articles from software vendors and developer publications
- Answer-first opening paragraphs among public SEO and web-performance guide articles from software vendors and developer publications
- Sites that turn away a declared crawler among public B2B SaaS homepages
- AI training crawler blocking among public B2B SaaS homepages
- Sites that turn away a declared crawler among large online stores in the Netherlands from ECDB's 2023 top 100
- AI training crawler blocking among large online stores in the Netherlands from ECDB's 2023 top 100
- AI search crawler blocking among large online stores in the Netherlands from ECDB's 2023 top 100
- llms.txt adoption among the top 50 sites of the Tranco ranking (list 64VVX)
- Article and FAQPage structured data among public B2B SaaS homepages
Where does your own site land?
Run the same kinds of checks on your own page with a free SEO and GEO audit. No signup.