KinetixSEO crawl study
AI search crawler blocking among large online stores in the Netherlands from ECDB's 2023 top 100
3% of the 70 large online stores in the Netherlands from ECDB's 2023 top 100 still block at least one AI search crawler in robots.txt.
Measured · n = 70 · cohort: large online stores in the Netherlands from ECDB's 2023 top 100
The numbers
2 of 70 (2.9%)
- Block at least one AI search crawler
- 2
- Allow all three AI search crawlers
- 68
- Sites observed (sample size)
- 70
- Listed but unreachable on the day (excluded)
- 27 of 97
- 25 refused our crawler outright (HTTP 401, 403, 406 or 429, or a bot-challenge page)
The headline rounds to a whole percent (3%); the exact share is 2.9% of the observed sample.
Methodology
How this was measured
Fetched robots.txt for each of the 70 large online stores in the Netherlands from ECDB's 2023 top 100 and counted the sites carrying a Disallow rule that blocks OAI-SearchBot, PerplexityBot, or Claude-SearchBot from the site root.
One observation: Whether the site's robots.txt blocks OAI-SearchBot, PerplexityBot or Claude-SearchBot from the site root.
How the cohort was chosen:ECDB's ranking of the 100 largest e-commerce stores in the Netherlands by FY2023 net online sales (shared by ECDB analyst Mikael Brakker), taken whole on 2026-10-23: every store on it in rank order, at its Dutch storefront. Left out only Allegro, Pakuten and Avans, which have no Dutch storefront. Rank 63 is ICI Paris XL; the graphic shows the Praxis logo there, and Praxis is already rank 23.
Check your own site: the free AI Crawler Checker reads your robots.txt and shows, per bot, which AI crawlers it blocks.
Every request was a single polite fetch per site with a short timeout and no retries. Only aggregate counts are published — no per-site result.
Limitations
What this figure does not tell you
- The cohort is a named list, not a random sample of the web. The share describes these sites, not every site.
- It is one measurement on the date shown. Any site can change its setup the next day.
- 27 of the 97 listed URLs could not be reached or read on the day and were left out of the sample rather than counted either way.
- 25 of those refused our declared research crawler outright (HTTP 401, 403, 406 or 429, or a bot-challenge page), usually a CDN or firewall rule against bots. Those sites may block AI crawlers the same way, which robots.txt cannot show.
- Only robots.txt was read. A site can also block AI crawlers at its CDN or firewall, which this check cannot see, and only the three citation-feeding crawlers were tested, against the site root.
How to cite
Use this data
KinetixSEO (2026). AI search crawler blocking among large online stores in the Netherlands from ECDB's 2023 top 100 [Data set]. KinetixSEO crawl study, measured 1 November 2026, n = 70. https://kinetixseo.com/learn/studies/ecommerce-nl-be-ai-crawler-block
Published under a CC BY 4.0 license: free to reuse with attribution. Download the aggregate results as CSV (every published run of this study).
Every published run
Run history
| Measured | Block at least one AI search crawler | Share | Sample size | Unreachable |
|---|---|---|---|---|
| 2 | 2.9% | n = 70 | 27 of 97 |
Where does your own site land?
Run the same kinds of checks on your own page with a free SEO and GEO audit. No signup.