KinetixSEO crawl study
Sites that turn away a declared crawler among large online stores in the Netherlands from ECDB's 2023 top 100
35% of the 91 large online stores in the Netherlands from ECDB's 2023 top 100 turn away a declared research crawler before serving the page.
Measured · n = 91 · cohort: large online stores in the Netherlands from ECDB's 2023 top 100
The numbers
32 of 91 (35.2%)
- Turn the crawler away
- 32
- Serve the page
- 59
- Sites observed (sample size)
- 91
- Listed but unreachable on the day (excluded)
- 6 of 97
The headline rounds to a whole percent (35%); the exact share is 35.2% of the observed sample.
Methodology
How this was measured
Read robots.txt and then requested each of the 91 large online stores in the Netherlands from ECDB's 2023 top 100 as a declared research crawler (the Kinetix-Research user agent, from our server's datacenter network, without running JavaScript), and counted the sites that turned it away: HTTP 401, 403, 406 or 429, or a bot-challenge page served in place of the page. Sites whose robots.txt disallows Kinetix-Research, or answered 429, a server error or nothing, were not asked, and sites that were down, timed out or answered any other error were left out of the sample.
One observation: Whether the site answered a request from our declared research crawler with HTTP 401, 403, 406 or 429, or with a bot-challenge page in place of the page.
How the cohort was chosen:ECDB's ranking of the 100 largest e-commerce stores in the Netherlands by FY2023 net online sales (shared by ECDB analyst Mikael Brakker), taken whole on 2026-10-23: every store on it in rank order, at its Dutch storefront. Left out only Allegro, Pakuten and Avans, which have no Dutch storefront. Rank 63 is ICI Paris XL; the graphic shows the Praxis logo there, and Praxis is already rank 23.
Check your own site: the free Bot View Checker shows whether your CDN or firewall challenges a bot it does not recognise, and what a crawler can read on the page.
Every request was a single polite fetch per site with a short timeout and no retries. Only aggregate counts are published — no per-site result.
Limitations
What this figure does not tell you
- The cohort is a named list, not a random sample of the web. The share describes these sites, not every site.
- It is one measurement on the date shown. Any site can change its setup the next day.
- 6 of the 97 listed URLs could not be reached or read on the day and were left out of the sample rather than counted either way.
- This measures how the sites treat one declared research crawler from one datacenter network, not any named AI crawler: CDNs often let verified crawlers such as Googlebot or GPTBot through rules that stop unknown bots, and some sites block those crawlers too. Bot rules change often, so a site can answer differently on another day. A site whose robots.txt disallows us, or answered 429, a server error or nothing, was not asked at all, so the share is a lower bound.
How to cite
Use this data
KinetixSEO (2026). Sites that turn away a declared crawler among large online stores in the Netherlands from ECDB's 2023 top 100 [Data set]. KinetixSEO crawl study, measured 1 November 2026, n = 91. https://kinetixseo.com/learn/studies/ecommerce-nl-be-crawler-refusal
Published under a CC BY 4.0 license: free to reuse with attribution. Download the aggregate results as CSV (every published run of this study).
Every published run
Run history
| Measured | Turn the crawler away | Share | Sample size | Unreachable |
|---|---|---|---|---|
| 32 | 35.2% | n = 91 | 6 of 97 |
Where does your own site land?
Run the same kinds of checks on your own page with a free SEO and GEO audit. No signup.