KinetixSEO crawl study
AI search crawler blocking among large online stores in the Netherlands from ECDB's 2023 top 100
3% of the 70 large online stores in the Netherlands from ECDB's 2023 top 100 still block at least one AI search crawler in robots.txt.
Measured · n = 70 · cohort: large online stores in the Netherlands from ECDB's 2023 top 100
Across the web: At least 14% of the websites we scanned block at least one AI crawler in robots.txt (n = 6,574, November 2026). That figure comes from our research sweep, which scans the homepages of a random draw of public websites with its own method, so it is a comparison, not this group measured twice. See How Many Websites Block AI Crawlers?.
The numbers
2 of 70 (2.9%)
- Block at least one AI search crawler
- 2
- Allow all three AI search crawlers
- 68
- Sites observed (sample size)
- 70
- Listed but unreachable on the day (excluded)
- 27 of 97
- 25 refused our crawler outright (HTTP 401, 403, 406 or 429, or a bot-challenge page)
The headline rounds to a whole percent (3%); the exact share is 2.9% of the observed sample.
Methodology
How this was measured
Fetched robots.txt for each of the 70 large online stores in the Netherlands from ECDB's 2023 top 100 and counted the sites carrying a Disallow rule that blocks OAI-SearchBot, PerplexityBot, or Claude-SearchBot from the site root.
One observation: Whether the site's robots.txt blocks OAI-SearchBot, PerplexityBot or Claude-SearchBot from the site root.
How the cohort was chosen:ECDB's ranking of the 100 largest e-commerce stores in the Netherlands by FY2023 net online sales (shared by ECDB analyst Mikael Brakker), taken whole on 2026-10-23: every store on it in rank order, at its Dutch storefront. Left out only Allegro, Pakuten and Avans, which have no Dutch storefront. Rank 63 is ICI Paris XL; the graphic shows the Praxis logo there, and Praxis is already rank 23.
Check your own site: the free AI Crawler Checker reads your robots.txt and shows, per bot, which AI crawlers it blocks.
Every request was a single polite fetch per site with a short timeout and no retries. Only aggregate counts are published — no per-site result.
Limitations
What this figure does not tell you
- The cohort is a named list, not a random sample of the web. The share describes these sites, not every site.
- It is one measurement on the date shown. Any site can change its setup the next day.
- 27 of the 97 listed URLs could not be reached or read on the day and were left out of the sample rather than counted either way.
- 25 of those refused our declared research crawler outright (HTTP 401, 403, 406 or 429, or a bot-challenge page), usually a CDN or firewall rule against bots. Those sites may block AI crawlers the same way, which robots.txt cannot show.
- Only robots.txt was read. A site can also block AI crawlers at its CDN or firewall, which this check cannot see, and only the three citation-feeding crawlers were tested, against the site root.
How to cite
Use this data
KinetixSEO (2026). AI search crawler blocking among large online stores in the Netherlands from ECDB's 2023 top 100 [Data set]. KinetixSEO crawl study, measured 1 November 2026, n = 70. https://kinetixseo.com/learn/studies/ecommerce-nl-be-ai-crawler-block
Published under a CC BY 4.0 license: free to reuse with attribution. Download the aggregate results as CSV (every published run of this study).
Every published run
Run history
| Measured | Block at least one AI search crawler | Share | Sample size | Unreachable |
|---|---|---|---|---|
| 2 | 2.9% | n = 70 | 27 of 97 |
More crawl studies
- llms.txt adoption among public SEO and content-marketing tool vendor sites
- Article and FAQPage structured data among public SEO and web-performance guide articles from software vendors and developer publications
- Canonical tags pointing elsewhere among public SEO and web-performance guide articles from software vendors and developer publications
- Answer-first opening paragraphs among public SEO and web-performance guide articles from software vendors and developer publications
- Sites that turn away a declared crawler among public B2B SaaS homepages
- AI training crawler blocking among public B2B SaaS homepages
- AI search crawler blocking among public B2B SaaS homepages
- Sites that turn away a declared crawler among large online stores in the Netherlands from ECDB's 2023 top 100
- AI training crawler blocking among large online stores in the Netherlands from ECDB's 2023 top 100
- llms.txt adoption among the top 50 sites of the Tranco ranking (list 64VVX)
- Article and FAQPage structured data among public B2B SaaS homepages
Where does your own site land?
Run the same kinds of checks on your own page with a free SEO and GEO audit. No signup.