AI traffic
AI crawlers and verification
Which AI crawlers we recognise, why they come (indexing, live fetches, training) and how we check they are who they claim.
An AI crawler is a program run by an AI company that requests your pages. We recognise it by the name it introduces itself with (the user agent) and verify it by the address it comes from.
Why they come: the three purposes
| Purpose | What it does | Examples | If you block it |
|---|---|---|---|
| AI indexing | Indexes pages for the assistant's search, like a search engine | OAI-SearchBot, Claude-SearchBot, PerplexityBot, Applebot | The assistant no longer finds your pages when it searches |
| Live fetches | Opens a page right now, for someone asking the assistant | ChatGPT-User, Claude-User, Perplexity-User | The assistant cannot quote the page in that answer |
| AI training | Collects text to train models | GPTBot, ClaudeBot, Google-Extended, CCBot | You leave the training data; search is not affected |
One operator usually runs a crawler per purpose. You can block training (GPTBot) and leave search (OAI-SearchBot) open.
Verification
Anyone can send a request with the user agent "GPTBot". So we check the IP of every hit:
- Published IP ranges — the operator (OpenAI, Anthropic, Perplexity, Google, Apple, DuckDuckGo, Mistral, Common Crawl) publishes its addresses. The IP must be in the list.
- Reverse DNS — the address's name must end in the operator's domain and lead back to the same address (Amazon, some Google crawlers).
- Unverifiable — the operator publishes no method (Meta, ByteDance, Cohere, DeepSeek…).
Every hit gets one of these statuses:
| Status | Meaning | Counted |
|---|---|---|
| Verified | the IP belongs to the operator | yes |
| Unverifiable | the operator publishes no way to verify it | only with "Include unverifiable" |
| Unchecked | the integration sent no IP, or the operator's list could not be read at the time | only with "Include unverifiable" |
| Spoofed | the IP does not belong to the operator: someone is pretending to be that crawler | never |
Spoofed hits stay in AI crawlers → Request log, marked, so you can see who tries.
If the site sits behind Cloudflare or a load balancer and the integration does not see the real IP, every hit comes out unchecked or spoofed. Each install guide says how to set it.
Crawlers we do not count
- Googlebot and Bingbot. They also feed AI Overviews or Copilot, but they are classic search crawlers; counted here, they would drown every other number.
- Crawlers removed from the list. When a crawler leaves the list its old hits stay stored but no longer count.
- Bots that are not AI (SEO tools, uptime monitors, scripts). They are not in the daily total either.
The list is updated weekly. Integrations download it daily; even with an old copy, we match every hit again when it arrives.
Crawler list
| Crawler | Operator | Purpose | Verification |
|---|---|---|---|
| GPTBot | OpenAI | AI training | published IPs |
| OAI-SearchBot | OpenAI | AI indexing | published IPs |
| ChatGPT-User | OpenAI | Live fetches | published IPs |
| ClaudeBot | Anthropic | AI training | published IPs |
| Claude-SearchBot | Anthropic | AI indexing | published IPs |
| Claude-User | Anthropic | Live fetches | published IPs |
| Claude-Web | Anthropic | Live fetches | unverifiable |
| anthropic-ai | Anthropic | AI training | unverifiable |
| PerplexityBot | Perplexity | AI indexing | published IPs |
| Perplexity-User | Perplexity | Live fetches | published IPs |
| Google-Extended | AI training | unverifiable | |
| Google-CloudVertexBot | AI training | reverse DNS | |
| GoogleOther | AI training | reverse DNS | |
| Google-NotebookLM | Live fetches | published IPs | |
| GoogleAgent-URLContext | Live fetches | published IPs | |
| Gemini-Deep-Research | Live fetches | published IPs | |
| Applebot | Apple | AI indexing | published IPs |
| Applebot-Extended | Apple | AI training | unverifiable |
| Amazonbot | Amazon | AI indexing | reverse DNS |
| DuckAssistBot | DuckDuckGo | AI indexing | published IPs |
| MistralAI-User | Mistral AI | Live fetches | published IPs |
| CCBot | Common Crawl | AI training | published IPs |
| meta-externalagent | Meta | AI training | unverifiable |
| meta-externalfetcher | Meta | Live fetches | unverifiable |
| FacebookBot | Meta | AI training | unverifiable |
| Bytespider | ByteDance | AI training | unverifiable |
| cohere-ai | Cohere | Live fetches | unverifiable |
| cohere-training-data-crawler | Cohere | AI training | unverifiable |
| YouBot | You.com | AI indexing | unverifiable |
| DeepSeekBot | DeepSeek | AI training | unverifiable |
| Diffbot | Diffbot | AI training | unverifiable |
| AI2Bot | Ai2 | AI training | unverifiable |
| Timpibot | Timpi | AI training | unverifiable |
| kagi-fetcher | Kagi | Live fetches | unverifiable |
The current list, as JSON, is public at /api/v1/signatures.