AI traffic
AI traffic
How we see the AI crawlers reading your site and the people assistants send you, and what data leaves your site.
The AI traffic pages answer two questions: which AI crawlers read your site (and what they get back) and how many people AI assistants send you. The data comes from your own site, through an integration you install once.
Why it needs a server integration
AI crawlers (GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot…) do not run JavaScript. A script in the page, like Google Analytics or our beacon, never sees them. Only the server answering them does, so that is where the integration runs:
| Platform | What you install | Sees AI crawlers | Sees people from AI |
|---|---|---|---|
| WordPress | the Survelium plugin | yes | yes |
| Laravel | the Composer package | yes | yes |
| OpenCart | a folder next to index.php |
yes | yes |
| Any other PHP site | the PHP core | yes | yes |
| Shopify, Wix, Webflow, static sites | the JavaScript beacon | no | yes |
Every package is open source (MIT licence), at github.com/survelium.
How the data reaches us
- You create a key in Brand settings → Integrations and put it in the integration's settings. The key ties the site to the brand.
- On every page view the integration looks at the user agent (who is asking for the page) and the referrer (where the visitor came from), after the page has gone out to the visitor. Nobody waits on it.
- Matches go into a local queue, sent to us in batches every few minutes.
- We match every hit against our crawler list again and verify its IP against the ranges the operator publishes.
If the site cannot reach us (network, revoked key, plan without AI traffic), the data stays queued and goes out later. Hits older than 7 days are no longer accepted.
What gets recorded
| Sent | Never sent | |
|---|---|---|
| AI crawler hit | time, user agent, page path, status code, IP (to verify it is really that crawler) | — |
| Visitor from an AI assistant | time, assistant domain (e.g. chatgpt.com), page path, user agent |
IP, full referrer, cookies |
| Every other page view | one total per day, for the AI traffic share | anything about the visit |
Crawler IPs are deleted after 30 days. We store no IPs of people.
The AI traffic pages
- Overview — crawler hits by purpose, people from AI and the AI traffic share, with the change against the previous period.
- Pages — every page of the site: who read it, how many people landed on it and what errors crawlers got.
- AI crawlers — every crawler, with its request log.
- Referrals — people sent by assistants, per day, per assistant and per landing page, with a CSV export.
- Integration — keys and install instructions. The "Connected" badge shows when data last arrived.
Rules on every page
- Days are UTC days, like the daily total your site sends. Charts can be a few hours off your local time.
- Only verified crawlers count by default. The "Include unverifiable" switch adds the crawlers that cannot be verified. Requests pretending to be another crawler never count.
- There is no model filter: crawlers and visitors are not tied to the models your prompts monitor.
- The period (7, 28 or 90 days) is compared with the same length of time just before it.