KinetixSEO research method
How our research sweep of public websites works
Some of the figures in our articles come from the KinetixSEO research sweep: our own scanner run over the homepages of a seeded random draw of public websites. This page shows how those figures are computed, what they currently say, and what they cannot tell you.
Data last updated (1 week ago) · 7 published figures
What these figures mean for your site
The sample
Which sites were measured, and how to check it
The sites were drawn before anything was scanned, from Tranco list JZPGY (a research ranking of popular domains): 10,000 domains out of the 99,001 ranked 1,000 to 100,000. The seed was picked at random by the server, so nobody chose which sites landed in it.
- Seed
- 5e2133590e30eb3d1cc64dafc0c7e2f0
- Method
- Every domain with min_rank <= rank <= max_rank gets the key sha256(seed + ":" + domain) as lowercase hex; the band is sorted by that key ascending and the first sample_size domains are the sample, in that order.
- Scanned
- 5 November 2026 to 12 November 2026, one homepage per site, one attempt each
- Full sample
- Download the draw (JSON): list ID, seed, method and every domain in order. SHA-256 3cf12837c34ae8432a92310a8d977a2e17db6d5afb7d51b46126166d2fdbd68a
Exclusion rules, fixed before the draw
- robots_blocked: robots.txt disallows the Kinetix-Research token (or *) at "/", or answers 429 or a server error (RFC 9309: complete disallow). Recorded before scanning; never fetched.
- non_resolving: the homepage fetch fails, or robots.txt gets no answer (DNS, connection, TLS or timeout). Recorded as a failed scan; not retried until it succeeds.
- non_html: the homepage answers with something other than an HTML document.
- off_domain_redirect: the homepage redirects to a different registrable domain.
An excluded domain is counted below and never replaced, so the denominator stays the full draw.
What happened to every domain
6,767 of 10,000 homepages could be scanned
- Scanned
- 6767
- Could not be reached or read
- 2814
- Asked us not to crawl (robots.txt)
- 359
14.5% of the 10000 sites refused our crawler: 1092 answered with HTTP 403, 406 or 429 (or a storefront wall) and 359 disallow it in robots.txt. Such an answer can be a bot filter, a firewall, a rate limit or a regional block; a site that refuses our crawler may refuse some search and AI crawlers the same way.
Unreachable covers no DNS answer, broken TLS, refused connections, timeouts and bot walls; a site that answered with no HTML page is counted there too. Tranco ranks domains by traffic and lookup signals, not by whether they have a website, so the draw also holds service domains with no homepage at all: DNS hosts, CDNs, tracking and API hosts. Most of the domains with no DNS answer are of that kind. The figures describe sites that have a homepage. 60 scans failed on our side and say nothing about the site; they are left out of every figure. Each domain got one attempt; a failure is recorded as a failure, not retried.
Every scan in this edition ran on one scanner version (runner image sha256:c5553e2c63dd0f2b2eea588ca48349a4b00e21d214f8b8b5671df1ed9928928c).
Methodology
How the data is collected
Our scanner runs over the homepage of each site in a seeded random draw of public websites. Each figure counts distinct sites, not scans: one site scanned several times is still one site.
Only sites where the check could actually run are counted. A homepage that did not load, or one where that part of the scan was unavailable, drops out of that figure's sample instead of counting as passing. Every figure states its own sample and the first and last scan dates it rests on, and a figure below 30 sites is not published at all.
Scans people run on their own sites with our free checker never feed these figures. Those sites chose themselves, so they cannot describe the web at large.
Current figures
What the data says now
Share of homepages in our seeded random research sweep of public sites that publish no llms.txt file.
82% of the 6226 sites in the KinetixSEO research sweep, scanned Nov 5, 2026 to Nov 12, 2026, publish no llms.txt file.
n = 6226 · measured (1 week ago)
Method: Scanned the homepage of each site in KinetixSEO's research sweep, a seeded random draw of public sites, between Nov 5, 2026 and Nov 12, 2026 with one unchanged version of the scanner, and counted the 6226 distinct sites on which the "Missing llms.txt file" check could run. A site counts once however often it was scanned, and counts as affected if any of its scans showed the issue. From scanner version 7 on, a site whose /llms.txt request got no usable answer (a timeout, a server error or a firewall refusal) is left out of the sample; in earlier sweeps it counted as having none, so their share is an upper bound. A site that answered with an HTML page in place of the file is counted as having none.
Share of homepages in our seeded random research sweep of public sites whose robots.txt blocks at least one AI crawler.
14% of the 6574 sites in the KinetixSEO research sweep, scanned Nov 5, 2026 to Nov 12, 2026, block at least one AI crawler in robots.txt.
n = 6574 · measured (1 week ago)
Method: Scanned the homepage of each site in KinetixSEO's research sweep, a seeded random draw of public sites, between Nov 5, 2026 and Nov 12, 2026 with one unchanged version of the scanner, and counted the 6574 distinct sites on which the "AI crawlers blocked in robots.txt" check could run. A site counts once however often it was scanned, and counts as affected if any of its scans showed the issue. A site whose robots.txt could not be read for another reason (a timeout, a 4xx response such as 403, or an HTML page served in its place) leaves no trace in the scan and is counted as not blocking, so the share is a lower bound.
Share of homepages in our seeded random research sweep of public sites that carry no JSON-LD structured data at all.
54% of the 6586 sites in the KinetixSEO research sweep, scanned Nov 5, 2026 to Nov 12, 2026, carry no JSON-LD structured data on their homepage.
n = 6586 · measured (1 week ago)
Method: Scanned the homepage of each site in KinetixSEO's research sweep, a seeded random draw of public sites, between Nov 5, 2026 and Nov 12, 2026 with one unchanged version of the scanner, and counted the 6586 distinct sites on which the "No structured data found" check could run. A site counts once however often it was scanned, and counts as affected if any of its scans showed the issue.
Share of homepages in our seeded random research sweep of public sites that have no meta description.
22% of the 6587 sites in the KinetixSEO research sweep, scanned Nov 5, 2026 to Nov 12, 2026, have no meta description on their homepage.
n = 6587 · measured (1 week ago)
Method: Scanned the homepage of each site in KinetixSEO's research sweep, a seeded random draw of public sites, between Nov 5, 2026 and Nov 12, 2026 with one unchanged version of the scanner, and counted the 6587 distinct sites on which the "Missing meta description" check could run. A site counts once however often it was scanned, and counts as affected if any of its scans showed the issue.
Share of homepages with images, in our seeded random research sweep of public sites, where at least one image has no alt text.
On 21% of the 5203 sites in the KinetixSEO research sweep with images on their homepage, scanned Nov 5, 2026 to Nov 12, 2026, at least one image has no alt text.
n = 5203 · measured (1 week ago)
Method: Scanned the homepage of each site in KinetixSEO's research sweep, a seeded random draw of public sites, between Nov 5, 2026 and Nov 12, 2026 with one unchanged version of the scanner, and counted the 5203 distinct sites on which the "Missing image alt text" check could run. A site counts once however often it was scanned, and counts as affected if any of its scans showed the issue.
Share of sites in our seeded random research sweep of public sites where no XML sitemap could be found.
On 32% of the 6496 sites in the KinetixSEO research sweep, scanned Nov 5, 2026 to Nov 12, 2026, no XML sitemap could be found.
n = 6496 · measured (1 week ago)
Method: Scanned the homepage of each site in KinetixSEO's research sweep, a seeded random draw of public sites, between Nov 5, 2026 and Nov 12, 2026 with one unchanged version of the scanner, and counted the 6496 distinct sites on which the "Missing XML sitemap" check could run. A site counts once however often it was scanned, and counts as affected if any of its scans showed the issue. The scanner looks for a sitemap declared in robots.txt and at the usual paths (/sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml). A sitemap kept at an unusual path and declared nowhere counts as missing, as it does for a crawler that looks in the same places.
Share of homepages in our seeded random research sweep of public sites that link no privacy policy, contact page or about page, or only some of them.
68% of the 6587 homepages in the KinetixSEO research sweep, scanned Nov 5, 2026 to Nov 12, 2026, do not link all three of a privacy policy, a contact page and an about page.
n = 6587 · measured (1 week ago)
Method: Scanned the homepage of each site in KinetixSEO's research sweep, a seeded random draw of public sites, between Nov 5, 2026 and Nov 12, 2026 with one unchanged version of the scanner, and counted the 6587 distinct sites on which the "Missing trust pages" check could run. A site counts once however often it was scanned, and counts as affected if any of its scans showed the issue. Only links on the homepage itself are read. A site that links its privacy policy, contact page or about page only from inner pages counts as missing it.
All data in this edition
Every finding, every score, 6,767 sites
Each rate is the share of the 6,767 sites in our research corpus for this sweep where the finding fired, with its 95% range. Only findings that fired on at least one site are listed. A check that could not run on a site counts as not firing there, so a rate can sit slightly below the same headline figure above, which leaves those sites out. Aggregates only: no site is named. A group of fewer than 30 sites reads "suppressed"; inside a larger group a count can be small.
See every finding rate on the edition page.
Download the data
- CSV (70 KB) · SHA-256 a059fa9c1474212c2611a2775a35587762f5080bf253bf7683206402e252f1ba
- JSON (148 KB) · SHA-256 12b26d14ee76fde8c8680bbde02e2dba107fb8630082fccc20bb823f974d8095
Frozen 12 November 2026 with the sample, the scanner version and the caveats known then; the files never change, so the SHA-256 holds for a copy you saved. Free to reuse with attribution (license).
Editions
Every sweep, kept as it was published
- 12 November 2026 · 6767 of 10000 homepages scanned · Tranco list JZPGY (shown on this page)
- 25 October 2026 · 3096 of 5000 homepages scanned · Tranco list 262W9
- 20 October 2026 · 347 of 500 homepages scanned · Tranco list GQG5K
Quote a figure from its edition page: it never changes, so a reader who follows your link finds the number you quoted.
Limitations
What these figures do not tell you
- Only the homepage of each site is scanned. Inner pages, which often differ, are not measured.
- Each figure counts sites where the check could actually run. Sites that failed to load, or where that part of the scan was unavailable, are left out of that figure's sample rather than counted as passing.
- A site counts as affected if any of its scans in the period showed the issue, so a site that fixed the issue between two scans still counts once as affected.
- Figures are only computed within one unchanged version of the scanner. When our checks change, the figure restarts from the newest version rather than mixing the two.
- Scans that people ran on their own sites, and scans we ran for outreach or competitor research, are never used. Those sites chose themselves or were chosen by us, so they say nothing about the web at large.
How to cite
Use this data
KinetixSEO (2026). How our research sweep of public websites works [Method and data]. KinetixSEO research sweep, retrieved 12 November 2026. https://kinetixseo.com/learn/methods/research-sweep
Published under a CC BY 4.0 license: free to reuse with attribution. Quote a figure together with its sample size and measurement date, as shown above.
Where does your own site land?
Run the same kinds of checks on your own page with a free SEO and GEO audit. No signup.