Original research
Crawl Studies: What Real Websites Actually Do
Each study runs one of KinetixSEO's own checks across a named, hand-curated list of public websites on a stated date, and publishes the count, the sample size and the method. Only studies with at least 30 observed sites, every listed URL attempted, and a method sentence that matches the check actually run are shown here.
Original research (November 2026): 10,000 sites drawn at random from the Tranco top 100,000, 6,767 homepages scanned. SEO Statistics 2026: What 6,767 Real Homepages Show →
Research sweep
Across thousands of websites
Our research sweep scans the homepages of a random draw of public websites, not a named group. Each figure below has its own page with the sample, the method and every earlier edition.
How Many Websites Have an llms.txt File?
18% of the websites we scanned publish an llms.txt file.
n = 6,226 · November 2026
How Many Websites Block AI Crawlers?
At least 14% of the websites we scanned block at least one AI crawler in robots.txt.
n = 6,574 · November 2026
How Many Websites Use JSON-LD Structured Data?
46% of the websites we scanned carry JSON-LD structured data on their homepage.
n = 6,586 · November 2026
How Many Websites Lack a Meta Description?
22% of the websites we scanned have no meta description on their homepage.
n = 6,587 · November 2026
How Many Websites Have No XML Sitemap?
On 32% of the websites we scanned, no XML sitemap could be found.
n = 6,496 · November 2026
How Many Websites Miss Image Alt Text?
On 21% of the websites we scanned that show images on their homepage, at least one image has no alt text.
n = 5,203 · November 2026
How Many Sites Skip Privacy, Contact or About?
68% of the homepages we scanned do not link all three of a privacy policy, a contact page and an about page.
n = 6,587 · November 2026
Published studies
The findings
llms.txt adoption among public SEO and content-marketing tool vendor sites
Only 37% of the 41 public SEO and content-marketing tool vendor sites publish an llms.txt file at all.
· n = 41 · 15 of 41 (36.6%)
Method, limits and data →Article and FAQPage structured data among public SEO and web-performance guide articles from software vendors and developer publications
58% of the 33 public SEO and web-performance guide articles from software vendors and developer publications publish Article or FAQPage structured data on the page itself.
· n = 33 · 19 of 33 (57.6%)
Method, limits and data →Canonical tags pointing elsewhere among public SEO and web-performance guide articles from software vendors and developer publications
3% of the 33 public SEO and web-performance guide articles from software vendors and developer publications point rel=canonical at a URL other than the one they serve.
· n = 33 · 1 of 33 (3.0%)
Method, limits and data →Answer-first opening paragraphs among public SEO and web-performance guide articles from software vendors and developer publications
Only 28% of the 32 public SEO and web-performance guide articles from software vendors and developer publications answer their own headline question in the first paragraph.
· n = 32 · 9 of 32 (28.1%)
Method, limits and data →Sites that turn away a declared crawler among public B2B SaaS homepages
2% of the 45 public B2B SaaS homepages turn away a declared research crawler before serving the page.
· n = 45 · 1 of 45 (2.2%)
Method, limits and data →AI training crawler blocking among public B2B SaaS homepages
7% of the 45 public B2B SaaS homepages block at least one AI training crawler in robots.txt.
· n = 45 · 3 of 45 (6.7%)
Method, limits and data →AI search crawler blocking among public B2B SaaS homepages
0% of the 45 public B2B SaaS homepages still block at least one AI search crawler in robots.txt.
· n = 45 · 0 of 45 (0.0%)
Method, limits and data →Sites that turn away a declared crawler among large online stores in the Netherlands from ECDB's 2023 top 100
35% of the 91 large online stores in the Netherlands from ECDB's 2023 top 100 turn away a declared research crawler before serving the page.
· n = 91 · 32 of 91 (35.2%)
Method, limits and data →AI training crawler blocking among large online stores in the Netherlands from ECDB's 2023 top 100
10% of the 70 large online stores in the Netherlands from ECDB's 2023 top 100 block at least one AI training crawler in robots.txt.
· n = 70 · 7 of 70 (10.0%)
Method, limits and data →AI search crawler blocking among large online stores in the Netherlands from ECDB's 2023 top 100
3% of the 70 large online stores in the Netherlands from ECDB's 2023 top 100 still block at least one AI search crawler in robots.txt.
· n = 70 · 2 of 70 (2.9%)
Method, limits and data →llms.txt adoption among the top 50 sites of the Tranco ranking (list 64VVX)
Only 16% of the 32 the top 50 sites of the Tranco ranking (list 64VVX) publish an llms.txt file at all.
· n = 32 · 5 of 32 (15.6%)
Method, limits and data →Article and FAQPage structured data among public B2B SaaS homepages
5% of the 44 public B2B SaaS homepages publish Article or FAQPage structured data on the page itself.
· n = 44 · 2 of 44 (4.5%)
Method, limits and data →
Reuse
Free to cite
Every study is published under a CC BY 4.0 license. Each study page has a suggested citation line and a CSV of its aggregate results. No per-site result is published. For our AI citation benchmark, see the AI search citation statistics.