Our crawlers
KinetixSEO checks web pages for SEO and AI-citation problems. To do that it fetches pages under two names, Kinetix-Auditor and Kinetix-Research. If one of them showed up in your logs, this page says what it was doing and how to stop it.
We never send another crawler's User-Agent. We do not pretend to be Googlebot, GPTBot or any other bot.
The names we send
| Name | What it fetches | Full User-Agent |
|---|---|---|
| Kinetix-Auditor | Almost everything: a scan of a page, a crawl of a site, a check of the links on a page, our free tools, the weekly check of a site its owner monitors with us, and some lookups of our own: the sites in our crawl studies, a site's contact page when we reach out, a paper we cite. | KinetixSEO/1.0 |
| Kinetix-Auditor in Chrome | The same checks when they need a real browser: running a page's JavaScript, measuring speed, taking a screenshot. It runs Chrome, so it sends a Chrome User-Agent with our name added at the end, as Google's Lighthouse does. | Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36 Kinetix-Auditor/1.0 |
| Kinetix-Research | Our research sweeps, which scan many sites for the studies we publish, the weekly check that the sources our own articles cite still work, and lookups of a brand on Wikipedia and Wikidata. | Kinetix-Research/1.0 (+https://kinetixseo.com/bot) |
Two names so you can block one without the other. Blocking our research does not break the audits you or your agency run on your own site.
How to block us in robots.txt
Each name has its own robots.txt token. To keep our research sweeps out of your site:
User-agent: Kinetix-Research
Disallow: /To keep our site crawls out:
User-agent: Kinetix-Auditor
Disallow: / Rules you wrote for our old name, Kinetix-AuditorBot, still apply when no group names Kinetix-Auditor. With no group for either name, we follow your User-agent: * group.
What follows robots.txt, and what does not
- A research sweep follows it before it scans a site. The weekly check of the sources our articles cite does not: it makes one request per cited page, and one retry if that fails.
- A crawl of a whole site follows it, including
Crawl-delayup to 2 seconds. The page you start the crawl from is always fetched. The owner of a site can turn off theDisallowrules for crawls of their own site, but only after proving ownership by connecting Google Search Console. - Everything else does not follow it: a scan of one page, the links on it, our free tools, the weekly check of a monitored site and the lookups of our own named above. A scan still reads your robots.txt, to audit it. A person, or a site's owner, asked for those pages, which is also how Google's user-triggered fetchers work.
Where our requests come from
Our requests leave from one IP address: 13.140.134.50. A request that carries our name but comes from another address is not ours.
How fast we go
- A site crawl fetches pages one at a time and stops at 200 pages.
- Without a
Crawl-delayit waits about 0.15 seconds between requests. With one it waits as long as it asks, up to 2 seconds. - When your server answers 403, 406 or 429, the crawl slows down. Each new refusal doubles the wait, up to 60 seconds. It retries such a page once with the
Acceptheaders an ordinary browser sends. The retry still carries our own name. - A scan of one page reads your sitemaps to count your pages: up to 10 sitemap files at once.
- The weekly check of a monitored site fetches at most 21 pages, each one once, plus robots.txt.
What we cannot see about AI crawlers
Our tools tell you whether your robots.txt lets AI crawlers in, and whether your CDN or firewall challenges our own crawler. Blocks that only target AI crawler names can't be tested without impersonating those crawlers, so we don't. OpenAI, Anthropic and Perplexity publish their IP ranges; check your firewall rules against them. Your server logs show what the real crawlers were served.
Requests to our own site
Some checks only fetch kinetixseo.com itself, so you will not see them on your site: KinetixSEO-DeploySmoke/1.0 after each release and KinetixSEO-ContentUptime/1.0 every few minutes.
Something wrong?
If one of our crawlers causes you trouble, mail hello@kinetixseo.com with the URL and the time, or use the contact form. You will hear back from the person who runs KinetixSEO.