KinetixSEO research method
Research sweep edition of 25 October 2026
Some of the figures in our articles come from the KinetixSEO research sweep: our own scanner run over the homepages of a seeded random draw of public websites. This page shows how those figures are computed, what they currently say, and what they cannot tell you.
Data last updated (10 hours ago) · 7 published figures
This is a fixed edition: the figures, the sample and the outcomes on this page never change, so a figure quoted from it stays checkable. The method page always shows the newest edition.
The sample
Which sites were measured, and how to check it
The sites were drawn before anything was scanned, from Tranco list 262W9 (a research ranking of popular domains): 5,000 domains out of the 99,001 ranked 1,000 to 100,000. The seed was picked at random by the server, so nobody chose which sites landed in it.
- Seed
- 178bf6353fecc897e000d7a3205f563c
- Method
- Every domain with min_rank <= rank <= max_rank gets the key sha256(seed + ":" + domain) as lowercase hex; the band is sorted by that key ascending and the first sample_size domains are the sample, in that order.
- Scanned
- 21 October 2026 to 25 October 2026, one homepage per site, one attempt each
- Full sample
- Download the draw (JSON): list ID, seed, method and every domain in order. SHA-256 9b1f714352235dac4ce2ee4fecc9b30ff760c1d7016156f4dc2a10b172bfe8f4
Exclusion rules, fixed before the draw
- robots_blocked: robots.txt disallows the Kinetix-Research token (or *) at "/", or answers 429 or a server error (RFC 9309: complete disallow). Recorded before scanning; never fetched.
- non_resolving: the homepage fetch fails, or robots.txt gets no answer (DNS, connection, TLS or timeout). Recorded as a failed scan; not retried until it succeeds.
- non_html: the homepage answers with something other than an HTML document.
- off_domain_redirect: the homepage redirects to a different registrable domain.
An excluded domain is counted below and never replaced, so the denominator stays the full draw.
What happened to every domain
3,096 of 5,000 homepages could be scanned
- Scanned
- 3096
- Could not be reached or read
- 1734
- Asked us not to crawl (robots.txt)
- 149
12.6% of the 5000 sites refused our crawler: 479 answered with HTTP 403, 406 or 429 (or a storefront wall) and 149 disallow it in robots.txt. Such an answer can be a bot filter, a firewall, a rate limit or a regional block; a site that refuses our crawler may refuse some search and AI crawlers the same way.
Unreachable covers no DNS answer, broken TLS, refused connections, timeouts and bot walls; a site that answered with no HTML page is counted there too. Tranco ranks domains by traffic and lookup signals, not by whether they have a website, so the draw also holds service domains with no homepage at all: DNS hosts, CDNs, tracking and API hosts. Most of the domains with no DNS answer are of that kind. The figures describe sites that have a homepage. 21 scans failed on our side and say nothing about the site; they are left out of every figure. Each domain got one attempt; a failure is recorded as a failure, not retried.
Every scan in this edition ran on one scanner version (runner image sha256:7c0aa…).
Methodology
How the data is collected
Our scanner runs over the homepage of each site in a seeded random draw of public websites. Each figure counts distinct sites, not scans: one site scanned several times is still one site.
Only sites where the check could actually run are counted. A homepage that did not load, or one where that part of the scan was unavailable, drops out of that figure's sample instead of counting as passing. Every figure states its own sample and the first and last scan dates it rests on, and a figure below 30 sites is not published at all.
Scans people run on their own sites with our free checker never feed these figures. Those sites chose themselves, so they cannot describe the web at large.
Figures in this edition
What this sweep measured
Share of homepages in our seeded random research sweep of public sites that publish no llms.txt file.
83% of the 3028 sites in the KinetixSEO research sweep, scanned Oct 21, 2026 to Oct 25, 2026, publish no llms.txt file.
n = 3028 · measured (10 hours ago)
Method: Scanned the homepage of each site in KinetixSEO's research sweep, a seeded random draw of public sites, between Oct 21, 2026 and Oct 25, 2026 with one unchanged version of the scanner, and counted the 3028 distinct sites on which the "Missing llms.txt file" check could run. A site counts once however often it was scanned, and counts as affected if any of its scans showed the issue. A site whose /llms.txt request failed (an error response such as 403 or 5xx, a timeout, or an HTML page served in its place) is counted as having no llms.txt, so the share is an upper bound.
Share of homepages in our seeded random research sweep of public sites whose robots.txt blocks at least one AI crawler.
13% of the 3024 sites in the KinetixSEO research sweep, scanned Oct 21, 2026 to Oct 25, 2026, block at least one AI crawler in robots.txt.
n = 3024 · measured (10 hours ago)
Method: Scanned the homepage of each site in KinetixSEO's research sweep, a seeded random draw of public sites, between Oct 21, 2026 and Oct 25, 2026 with one unchanged version of the scanner, and counted the 3024 distinct sites on which the "AI crawlers blocked in robots.txt" check could run. A site counts once however often it was scanned, and counts as affected if any of its scans showed the issue. A site whose robots.txt could not be read for another reason (a timeout, a 4xx response such as 403, or an HTML page served in its place) leaves no trace in the scan and is counted as not blocking, so the share is a lower bound.
Share of homepages in our seeded random research sweep of public sites that carry no JSON-LD structured data at all.
52% of the 3028 sites in the KinetixSEO research sweep, scanned Oct 21, 2026 to Oct 25, 2026, carry no JSON-LD structured data on their homepage.
n = 3028 · measured (10 hours ago)
Method: Scanned the homepage of each site in KinetixSEO's research sweep, a seeded random draw of public sites, between Oct 21, 2026 and Oct 25, 2026 with one unchanged version of the scanner, and counted the 3028 distinct sites on which the "No structured data found" check could run. A site counts once however often it was scanned, and counts as affected if any of its scans showed the issue.
Share of homepages in our seeded random research sweep of public sites that have no meta description.
20% of the 3028 sites in the KinetixSEO research sweep, scanned Oct 21, 2026 to Oct 25, 2026, have no meta description on their homepage.
n = 3028 · measured (10 hours ago)
Method: Scanned the homepage of each site in KinetixSEO's research sweep, a seeded random draw of public sites, between Oct 21, 2026 and Oct 25, 2026 with one unchanged version of the scanner, and counted the 3028 distinct sites on which the "Missing meta description" check could run. A site counts once however often it was scanned, and counts as affected if any of its scans showed the issue.
Share of homepages with images, in our seeded random research sweep of public sites, where at least one image has no alt text.
On 23% of the 2027 sites in the KinetixSEO research sweep with images on their homepage, scanned Oct 21, 2026 to Oct 25, 2026, at least one image has no alt text.
n = 2027 · measured (10 hours ago)
Method: Scanned the homepage of each site in KinetixSEO's research sweep, a seeded random draw of public sites, between Oct 21, 2026 and Oct 25, 2026 with one unchanged version of the scanner, and counted the 2027 distinct sites on which the "Missing image alt text" check could run. A site counts once however often it was scanned, and counts as affected if any of its scans showed the issue.
Share of sites in our seeded random research sweep of public sites where no XML sitemap could be found.
On 30% of the 2984 sites in the KinetixSEO research sweep, scanned Oct 21, 2026 to Oct 25, 2026, no XML sitemap could be found.
n = 2984 · measured (10 hours ago)
Method: Scanned the homepage of each site in KinetixSEO's research sweep, a seeded random draw of public sites, between Oct 21, 2026 and Oct 25, 2026 with one unchanged version of the scanner, and counted the 2984 distinct sites on which the "Missing XML sitemap" check could run. A site counts once however often it was scanned, and counts as affected if any of its scans showed the issue. The scanner looks for a sitemap declared in robots.txt and at the usual paths (/sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml). A sitemap kept at an unusual path and declared nowhere counts as missing, as it does for a crawler that looks in the same places.
Share of homepages in our seeded random research sweep of public sites that link no privacy policy, contact page or about page, or only some of them.
67% of the 3028 homepages in the KinetixSEO research sweep, scanned Oct 21, 2026 to Oct 25, 2026, do not link all three of a privacy policy, a contact page and an about page.
n = 3028 · measured (10 hours ago)
Method: Scanned the homepage of each site in KinetixSEO's research sweep, a seeded random draw of public sites, between Oct 21, 2026 and Oct 25, 2026 with one unchanged version of the scanner, and counted the 3028 distinct sites on which the "Missing trust pages" check could run. A site counts once however often it was scanned, and counts as affected if any of its scans showed the issue. Only links on the homepage itself are read. A site that links its privacy policy, contact page or about page only from inner pages counts as missing it.
Editions
Every sweep, kept as it was published
- 25 October 2026 · 3096 of 5000 homepages scanned · Tranco list 262W9 (shown on this page)
- 20 October 2026 · 347 of 500 homepages scanned · Tranco list GQG5K
Quote a figure from its edition page: it never changes, so a reader who follows your link finds the number you quoted.
Limitations
What these figures do not tell you
- Only the homepage of each site is scanned. Inner pages, which often differ, are not measured.
- Each figure counts sites where the check could actually run. Sites that failed to load, or where that part of the scan was unavailable, are left out of that figure's sample rather than counted as passing.
- A site counts as affected if any of its scans in the period showed the issue, so a site that fixed the issue between two scans still counts once as affected.
- Figures are only computed within one unchanged version of the scanner. When our checks change, the figure restarts from the newest version rather than mixing the two.
- Scans that people ran on their own sites, and scans we ran for outreach or competitor research, are never used. Those sites chose themselves or were chosen by us, so they say nothing about the web at large.
How to cite
Use this data
KinetixSEO (2026). Research sweep edition of 25 October 2026 [Method and data]. KinetixSEO research sweep, retrieved 25 October 2026. https://kinetixseo.com/learn/methods/research-sweep/2026-10-21-tranco-262W9-178bf635
Published under a CC BY 4.0 license: free to reuse with attribution. Quote a figure together with its sample size and measurement date, as shown above.
Where does your own site land?
Run the same kinds of checks on your own page with a free SEO and GEO audit. No signup.