KinetixSEO research method
Research sweep edition of 20 October 2026
Some of the figures in our articles come from the KinetixSEO research sweep: our own scanner run over the homepages of a seeded random draw of public websites. This page shows how those figures are computed, what they currently say, and what they cannot tell you.
Data last updated (1 day ago) · 7 published figures
This is a fixed edition: the figures, the sample and the outcomes on this page never change, so a figure quoted from it stays checkable. The method page always shows the newest edition.
The sample
Which sites were measured, and how to check it
The sites were drawn before anything was scanned, from Tranco list GQG5K (a research ranking of popular domains): 500 domains out of the 99,001 ranked 1,000 to 100,000. The seed was picked at random by the server, so nobody chose which sites landed in it.
- Seed
- dd6642610f1c873abbdc41b5df576367
- Method
- Every domain with min_rank <= rank <= max_rank gets the key sha256(seed + ":" + domain) as lowercase hex; the band is sorted by that key ascending and the first sample_size domains are the sample, in that order.
- Scanned
- 20 October 2026 to 20 October 2026, one homepage per site, one attempt each
- Full sample
- Download the draw (JSON): list ID, seed, method and every domain in order. SHA-256 79911d147b32b03beb41383c09303b301935cb62df745bb1f6e29087d76ed513
Exclusion rules, fixed before the draw
- robots_blocked: robots.txt disallows the Kinetix-Research token (or *) at "/". Recorded before scanning; never fetched.
- non_resolving: the homepage fetch fails (DNS, connection, TLS or timeout). Recorded as a failed scan; not retried until it succeeds.
- non_html: the homepage answers with something other than an HTML document.
- off_domain_redirect: the homepage redirects to a different registrable domain.
An excluded domain is counted below and never replaced, so the denominator stays the full draw.
What happened to every domain
347 of 500 homepages could be scanned
- Scanned
- 347
- Could not be reached or read
- 142
- Asked us not to crawl (robots.txt)
- 9
12.2% of the 500 sites refused our crawler: 52 answered with HTTP 403, 406 or 429 (or a storefront wall) and 9 disallow it in robots.txt. Such an answer can be a bot filter, a firewall, a rate limit or a regional block; a site that refuses our crawler may refuse some search and AI crawlers the same way.
Unreachable covers no DNS answer, broken TLS, refused connections, timeouts and bot walls; a site that answered with no HTML page is counted there too. 2 scans failed on our side and say nothing about the site; they are left out of every figure. Each domain got one attempt; a failure is recorded as a failure, not retried.
Every scan in this edition ran on one scanner version (runner image sha256:c98e9…).
Methodology
How the data is collected
Our scanner runs over the homepage of each site in a seeded random draw of public websites. Each figure counts distinct sites, not scans: one site scanned several times is still one site.
Only sites where the check could actually run are counted. A homepage that did not load, or one where that part of the scan was unavailable, drops out of that figure's sample instead of counting as passing. Every figure states its own sample and the first and last scan dates it rests on, and a figure below 30 sites is not published at all.
Scans people run on their own sites with our free checker never feed these figures. Those sites chose themselves, so they cannot describe the web at large.
Figures in this edition
What this sweep measured
Share of homepages in our seeded random research sweep of public sites that publish no llms.txt file.
85% of the 332 sites in the KinetixSEO research sweep, scanned Oct 20, 2026 to Oct 20, 2026, publish no llms.txt file.
n = 332 · measured (1 day ago)
Method: Scanned the homepage of each site in KinetixSEO's research sweep, a seeded random draw of public sites, between Oct 20, 2026 and Oct 20, 2026 with one unchanged version of the scanner, and counted the 332 distinct sites on which the "Missing llms.txt file" check could run. A site counts once however often it was scanned, and counts as affected if any of its scans showed the issue. A site whose /llms.txt request failed (an error response such as 403 or 5xx, a timeout, or an HTML page served in its place) is counted as having no llms.txt, so the share is an upper bound.
Share of homepages in our seeded random research sweep of public sites whose robots.txt blocks at least one AI crawler.
11% of the 332 sites in the KinetixSEO research sweep, scanned Oct 20, 2026 to Oct 20, 2026, block at least one AI crawler in robots.txt.
n = 332 · measured (1 day ago)
Method: Scanned the homepage of each site in KinetixSEO's research sweep, a seeded random draw of public sites, between Oct 20, 2026 and Oct 20, 2026 with one unchanged version of the scanner, and counted the 332 distinct sites on which the "AI crawlers blocked in robots.txt" check could run. A site counts once however often it was scanned, and counts as affected if any of its scans showed the issue. A site whose robots.txt could not be read for another reason (a timeout, a 4xx response such as 403, or an HTML page served in its place) leaves no trace in the scan and is counted as not blocking, so the share is a lower bound.
Share of homepages in our seeded random research sweep of public sites that carry no JSON-LD structured data at all.
53% of the 332 sites in the KinetixSEO research sweep, scanned Oct 20, 2026 to Oct 20, 2026, carry no JSON-LD structured data on their homepage.
n = 332 · measured (1 day ago)
Method: Scanned the homepage of each site in KinetixSEO's research sweep, a seeded random draw of public sites, between Oct 20, 2026 and Oct 20, 2026 with one unchanged version of the scanner, and counted the 332 distinct sites on which the "No structured data found" check could run. A site counts once however often it was scanned, and counts as affected if any of its scans showed the issue.
Share of homepages in our seeded random research sweep of public sites that have no meta description.
22% of the 332 sites in the KinetixSEO research sweep, scanned Oct 20, 2026 to Oct 20, 2026, have no meta description on their homepage.
n = 332 · measured (1 day ago)
Method: Scanned the homepage of each site in KinetixSEO's research sweep, a seeded random draw of public sites, between Oct 20, 2026 and Oct 20, 2026 with one unchanged version of the scanner, and counted the 332 distinct sites on which the "Missing meta description" check could run. A site counts once however often it was scanned, and counts as affected if any of its scans showed the issue.
Share of homepages with images, in our seeded random research sweep of public sites, where at least one image has no alt text.
On 26% of the 217 sites in the KinetixSEO research sweep with images on their homepage, scanned Oct 20, 2026 to Oct 20, 2026, at least one image has no alt text.
n = 217 · measured (1 day ago)
Method: Scanned the homepage of each site in KinetixSEO's research sweep, a seeded random draw of public sites, between Oct 20, 2026 and Oct 20, 2026 with one unchanged version of the scanner, and counted the 217 distinct sites on which the "Missing image alt text" check could run. A site counts once however often it was scanned, and counts as affected if any of its scans showed the issue.
Share of sites in our seeded random research sweep of public sites where no XML sitemap could be found.
On 33% of the 332 sites in the KinetixSEO research sweep, scanned Oct 20, 2026 to Oct 20, 2026, no XML sitemap could be found.
n = 332 · measured (1 day ago)
Method: Scanned the homepage of each site in KinetixSEO's research sweep, a seeded random draw of public sites, between Oct 20, 2026 and Oct 20, 2026 with one unchanged version of the scanner, and counted the 332 distinct sites on which the "Missing XML sitemap" check could run. A site counts once however often it was scanned, and counts as affected if any of its scans showed the issue. The scanner looks for a sitemap declared in robots.txt and at the usual paths (/sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml). A sitemap kept at an unusual path and declared nowhere counts as missing, as it does for a crawler that looks in the same places.
Share of homepages in our seeded random research sweep of public sites that link no privacy policy, contact page or about page, or only some of them.
68% of the 332 homepages in the KinetixSEO research sweep, scanned Oct 20, 2026 to Oct 20, 2026, do not link all three of a privacy policy, a contact page and an about page.
n = 332 · measured (1 day ago)
Method: Scanned the homepage of each site in KinetixSEO's research sweep, a seeded random draw of public sites, between Oct 20, 2026 and Oct 20, 2026 with one unchanged version of the scanner, and counted the 332 distinct sites on which the "Missing trust pages" check could run. A site counts once however often it was scanned, and counts as affected if any of its scans showed the issue. Only links on the homepage itself are read. A site that links its privacy policy, contact page or about page only from inner pages counts as missing it.
Editions
Every sweep, kept as it was published
- 20 October 2026 · 347 of 500 homepages scanned · Tranco list GQG5K (shown on this page)
Quote a figure from its edition page: it never changes, so a reader who follows your link finds the number you quoted.
Limitations
What these figures do not tell you
- Only the homepage of each site is scanned. Inner pages, which often differ, are not measured.
- Each figure counts sites where the check could actually run. Sites that failed to load, or where that part of the scan was unavailable, are left out of that figure's sample rather than counted as passing.
- A site counts as affected if any of its scans in the period showed the issue, so a site that fixed the issue between two scans still counts once as affected.
- Figures are only computed within one unchanged version of the scanner. When our checks change, the figure restarts from the newest version rather than mixing the two.
- Scans that people ran on their own sites, and scans we ran for outreach or competitor research, are never used. Those sites chose themselves or were chosen by us, so they say nothing about the web at large.
How to cite
Use this data
KinetixSEO (2026). Research sweep edition of 20 October 2026 [Method and data]. KinetixSEO research sweep, retrieved 20 October 2026. https://kinetixseo.com/learn/methods/research-sweep/2026-10-20-tranco-GQG5K-dd664261
Published under a CC BY 4.0 license: free to reuse with attribution. Quote a figure together with its sample size and measurement date, as shown above.
Where does your own site land?
Run the same kinds of checks on your own page with a free SEO and GEO audit. No signup.