KinetixSEO crawl study
Sites that turn away a declared crawler among public B2B SaaS homepages
2% of the 45 public B2B SaaS homepages turn away a declared research crawler before serving the page.
Measured · n = 45 · cohort: public B2B SaaS homepages
The numbers
1 of 45 (2.2%)
- Turn the crawler away
- 1
- Serve the page
- 44
- Sites observed (sample size)
- 45
- Listed but unreachable on the day (excluded)
- 0 of 45
The headline rounds to a whole percent (2%); the exact share is 2.2% of the observed sample.
Methodology
How this was measured
Read robots.txt and then requested each of the 45 public B2B SaaS homepages as a declared research crawler (the Kinetix-Research user agent, from our server's datacenter network, without running JavaScript), and counted the sites that turned it away: HTTP 401, 403, 406 or 429, or a bot-challenge page served in place of the page. Sites whose robots.txt disallows Kinetix-Research, or answered 429, a server error or nothing, were not asked, and sites that were down, timed out or answered any other error were left out of the sample.
One observation: Whether the site answered a request from our declared research crawler with HTTP 401, 403, 406 or 429, or with a bot-challenge page in place of the page.
How the cohort was chosen:Hand-curated 2026-09-04 from widely cited public "most used B2B SaaS products" listings. Every URL is the vendor's own public homepage; no customer data and no private endpoint is involved.
Check your own site: the free Bot View Checker shows whether your CDN or firewall challenges a bot it does not recognise, and what a crawler can read on the page.
Every request was a single polite fetch per site with a short timeout and no retries. Only aggregate counts are published — no per-site result.
Limitations
What this figure does not tell you
- The cohort is a named list, not a random sample of the web. The share describes these sites, not every site.
- It is one measurement on the date shown. Any site can change its setup the next day.
- This measures how the sites treat one declared research crawler from one datacenter network, not any named AI crawler: CDNs often let verified crawlers such as Googlebot or GPTBot through rules that stop unknown bots, and some sites block those crawlers too. Bot rules change often, so a site can answer differently on another day. A site whose robots.txt disallows us, or answered 429, a server error or nothing, was not asked at all, so the share is a lower bound.
How to cite
Use this data
KinetixSEO (2026). Sites that turn away a declared crawler among public B2B SaaS homepages [Data set]. KinetixSEO crawl study, measured 1 November 2026, n = 45. https://kinetixseo.com/learn/studies/saas-homepages-crawler-refusal
Published under a CC BY 4.0 license: free to reuse with attribution. Download the aggregate results as CSV (every published run of this study).
Every published run
Run history
| Measured | Turn the crawler away | Share | Sample size | Unreachable |
|---|---|---|---|---|
| 1 | 2.2% | n = 45 | 0 of 45 |
Where does your own site land?
Run the same kinds of checks on your own page with a free SEO and GEO audit. No signup.