AI traffic
Pages and errors
Which pages AI crawlers read, how many people they bring and what errors search crawlers get.
The AI traffic → Pages page shows, for every page of the site, who read it, how many people landed on it from assistants and what the last crawler to request it got back.
Columns
| Column | What it shows |
|---|---|
| Page | The page path, without parameters: /product?utm_source=chatgpt.com is the same page as /product |
| AI indexing, Live fetches, AI training | Crawler hits on the page, by purpose |
| People from AI | Visits from assistants that landed on the page |
| Errors | Requests from indexing and live-fetch crawlers answered with a 4xx or 5xx code |
| Last status | The code the most recent crawler got for the page |
| Last read | When an AI crawler last requested it |
Errors
errors = requests from AI indexing and Live fetches with status ≥ 400
Errors returned to training crawlers do not count: a 403 for GPTBot is usually your choice, not a problem. For search crawlers and live fetches, a 403 or 404 means the assistant cannot quote the page.
When at least one page has errors, an alert shows above the table, with a button that keeps only those pages. Common causes:
- a firewall or security plugin blocking crawlers (Cloudflare Bot Fight Mode, Wordfence);
- pages deleted or moved without a redirect;
robots.txtor server rules refusing the user agent;- 5xx errors when crawlers request many pages at once.
Example
In the last 28 days, /services got 30 requests from OAI-SearchBot (all 200), 6 from ChatGPT-User (4 with 200, 2 with
503) and 10 from GPTBot (all 403):
AI indexing = 30 · Live fetches = 6 · AI training = 10 · Errors = 2
The 10 responses of 403 to GPTBot are not errors here.
Where it appears
- AI traffic → Pages — the full table, with search and sorting.
- AI traffic → Overview — "Top pages", with crawler hits and people from AI.