KinetixSEO research sweep · November 2026
How Many Websites Block AI Crawlers?
At least 14% of the websites we scanned block at least one AI crawler in robots.txt.
The other 86% showed no block on the AI crawlers we check. A robots.txt we could not read counts here too, so the blocked share is a minimum.
Based on 6,574 homepages of public websites (Tranco ranks 1,000 to 100,000, drawn at random), scanned 5 to 12 November 2026. See the full edition.
14% of 6,574 sites
What it means
Why this number matters
AI crawlers come in two kinds: training crawlers such as GPTBot, ClaudeBot and Google-Extended, which collect text for model training, and search crawlers such as OAI-SearchBot and PerplexityBot, which fetch the pages AI answers cite.
Blocking an AI search crawler tells that engine not to fetch the site, which makes a citation from a fresh fetch unlikely. Blocking a training crawler keeps text out of model training but does not stop the search crawlers.
Check your own site: the free AI Crawler Checker reads your robots.txt and shows, per bot, which AI crawlers it blocks.
Named groups
How specific groups of sites compare
Our crawl studies measure the same topic on small, named lists of sites, with their own check and method, so their shares are not directly comparable to the figure above.
- AI training crawler blocking among public B2B SaaS homepages
7% of the 45 public B2B SaaS homepages block at least one AI training crawler in robots.txt. (n = 45)
- AI search crawler blocking among public B2B SaaS homepages
0% of the 45 public B2B SaaS homepages still block at least one AI search crawler in robots.txt. (n = 45)
- AI training crawler blocking among large online stores in the Netherlands from ECDB's 2023 top 100
10% of the 70 large online stores in the Netherlands from ECDB's 2023 top 100 block at least one AI training crawler in robots.txt. (n = 70)
- AI search crawler blocking among large online stores in the Netherlands from ECDB's 2023 top 100
3% of the 70 large online stores in the Netherlands from ECDB's 2023 top 100 still block at least one AI search crawler in robots.txt. (n = 70)
Methodology
How this was measured
Scanned the homepage of each site in KinetixSEO's research sweep, a seeded random draw of public sites, between Nov 5, 2026 and Nov 12, 2026 with one unchanged version of the scanner, and counted the 6574 distinct sites on which the "AI crawlers blocked in robots.txt" check could run. A site counts once however often it was scanned, and counts as affected if any of its scans showed the issue. A site whose robots.txt could not be read for another reason (a timeout, a 4xx response such as 403, or an HTML page served in its place) leaves no trace in the scan and is counted as not blocking, so the share is a lower bound.
The draw, its seed and the scanner version are public on the edition page, so anyone can redraw the same sites. Only aggregate counts are published — no per-site result.
Every edition
Edition history
Each edition is a new random draw of sites, so a change between editions is partly sampling noise.
| Edition | Share | Sample size |
|---|---|---|
| November 2026 | 14% | n = 6,574 |
| October 2026 | 13% | n = 3,024 |
| October 2026 | 11% | n = 332 |
How to cite
Use this data
KinetixSEO (2026). How Many Websites Block AI Crawlers [Data set]. KinetixSEO research sweep, edition of November 2026, n = 6574. https://kinetixseo.com/learn/statistics/how-many-websites-block-ai-crawlers
Published under a CC BY 4.0 license: free to reuse with attribution. Download the edition's aggregate results as CSV or JSON .
More website statistics
- How Many Websites Have an llms.txt File? — 18% of the websites we scanned publish an llms.txt file.
- How Many Websites Use JSON-LD Structured Data? — 46% of the websites we scanned carry JSON-LD structured data on their homepage.
- How Many Websites Lack a Meta Description? — 22% of the websites we scanned have no meta description on their homepage.
- How Many Websites Have No XML Sitemap? — On 32% of the websites we scanned, no XML sitemap could be found.
- How Many Websites Miss Image Alt Text? — On 21% of the websites we scanned that show images on their homepage, at least one image has no alt text.
- How Many Sites Skip Privacy, Contact or About? — 68% of the homepages we scanned do not link all three of a privacy policy, a contact page and an about page.
Where does your own site land?
Run the same scan on your own homepage with a free SEO and GEO check. No signup.