# AI crawlers and verification

> Which AI crawlers we recognise, why they come (indexing, live fetches, training) and how we check they are who they claim.

**An AI crawler is a program run by an AI company that requests your pages.** We recognise it by the name it
introduces itself with (the user agent) and verify it by the address it comes from.

## Why they come: the three purposes

| Purpose | What it does | Examples | If you block it |
|---|---|---|---|
| **AI indexing** | Indexes pages for the assistant's search, like a search engine | OAI-SearchBot, Claude-SearchBot, PerplexityBot, Applebot | The assistant no longer finds your pages when it searches |
| **Live fetches** | Opens a page right now, for someone asking the assistant | ChatGPT-User, Claude-User, Perplexity-User | The assistant cannot quote the page in that answer |
| **AI training** | Collects text to train models | GPTBot, ClaudeBot, Google-Extended, CCBot | You leave the training data; search is not affected |

One operator usually runs a crawler per purpose. You can block training (GPTBot) and leave search (OAI-SearchBot)
open.

## Verification

Anyone can send a request with the user agent "GPTBot". So we check the IP of every hit:

- **Published IP ranges** — the operator (OpenAI, Anthropic, Perplexity, Google, Apple, DuckDuckGo, Mistral, Common
  Crawl) publishes its addresses. The IP must be in the list.
- **Reverse DNS** — the address's name must end in the operator's domain and lead back to the same address (Amazon,
  some Google crawlers).
- **Unverifiable** — the operator publishes no method (Meta, ByteDance, Cohere, DeepSeek…).

Every hit gets one of these statuses:

| Status | Meaning | Counted |
|---|---|---|
| Verified | the IP belongs to the operator | yes |
| Unverifiable | the operator publishes no way to verify it | only with "Include unverifiable" |
| Unchecked | the integration sent no IP, or the operator's list could not be read at the time | only with "Include unverifiable" |
| Spoofed | the IP does **not** belong to the operator: someone is pretending to be that crawler | never |

Spoofed hits stay in **AI crawlers → Request log**, marked, so you can see who tries.

If the site sits behind Cloudflare or a load balancer and the integration does not see the real IP, every hit comes
out unchecked or spoofed. Each install guide says how to set it.

## Crawlers we do not count

- **Googlebot and Bingbot.** They also feed AI Overviews or Copilot, but they are classic search crawlers; counted
  here, they would drown every other number.
- **Crawlers removed from the list.** When a crawler leaves the list its old hits stay stored but no longer count.
- **Bots that are not AI** (SEO tools, uptime monitors, scripts). They are not in the
  [daily total](https://survelium.com/docs/ai-traffic/traffic-share.md) either.

The list is updated weekly. Integrations download it daily; even with an old copy, we match every hit again when it
arrives.

## Crawler list

| Crawler | Operator | Purpose | Verification |
|---|---|---|---|
| GPTBot | OpenAI | AI training | published IPs |
| OAI-SearchBot | OpenAI | AI indexing | published IPs |
| ChatGPT-User | OpenAI | Live fetches | published IPs |
| ClaudeBot | Anthropic | AI training | published IPs |
| Claude-SearchBot | Anthropic | AI indexing | published IPs |
| Claude-User | Anthropic | Live fetches | published IPs |
| Claude-Web | Anthropic | Live fetches | unverifiable |
| anthropic-ai | Anthropic | AI training | unverifiable |
| PerplexityBot | Perplexity | AI indexing | published IPs |
| Perplexity-User | Perplexity | Live fetches | published IPs |
| Google-Extended | Google | AI training | unverifiable |
| Google-CloudVertexBot | Google | AI training | reverse DNS |
| GoogleOther | Google | AI training | reverse DNS |
| Google-NotebookLM | Google | Live fetches | published IPs |
| GoogleAgent-URLContext | Google | Live fetches | published IPs |
| Gemini-Deep-Research | Google | Live fetches | published IPs |
| Applebot | Apple | AI indexing | published IPs |
| Applebot-Extended | Apple | AI training | unverifiable |
| Amazonbot | Amazon | AI indexing | reverse DNS |
| DuckAssistBot | DuckDuckGo | AI indexing | published IPs |
| MistralAI-User | Mistral AI | Live fetches | published IPs |
| CCBot | Common Crawl | AI training | published IPs |
| meta-externalagent | Meta | AI training | unverifiable |
| meta-externalfetcher | Meta | Live fetches | unverifiable |
| FacebookBot | Meta | AI training | unverifiable |
| Bytespider | ByteDance | AI training | unverifiable |
| cohere-ai | Cohere | Live fetches | unverifiable |
| cohere-training-data-crawler | Cohere | AI training | unverifiable |
| YouBot | You.com | AI indexing | unverifiable |
| DeepSeekBot | DeepSeek | AI training | unverifiable |
| Diffbot | Diffbot | AI training | unverifiable |
| AI2Bot | Ai2 | AI training | unverifiable |
| Timpibot | Timpi | AI training | unverifiable |
| kagi-fetcher | Kagi | Live fetches | unverifiable |

The current list, as JSON, is public at [`/api/v1/signatures`](/api/v1/signatures).