Optimization
Crawl Checker
Whether AI crawlers can read a page of your site, what they get from it, and how to read the result.
Crawl Checker asks one page of your site the way AI crawlers do, and tells you whether they get in and what they find. Nothing in it uses AI: every verdict is a rule over the answers our requests got.
What it checks
AI crawler access, for twenty crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot…):
- robots.txt — read the way crawlers read it: a group can name several crawlers, a crawler follows its own group
and only falls back to
User-agent: *when none names it, and the most specific rule wins. A robots.txt that cannot be downloaded (server error, timeout) counts as "everything disallowed", because that is what crawlers do. - The server — we request the page with each crawler's user agent. A refusal (403, 401, a challenge page such as Cloudflare's "Just a moment…") is Blocked; a 429 is a rate limit, shown apart; no answer is Unreachable. A second request as an ordinary browser tells a crawler block from a site that is down.
Google-Extended, Applebot-Extended and anthropic-ai only exist in robots.txt (the real crawler announces itself under another name), so they are judged on robots.txt alone.
The page: whether the text is in the HTML or loaded by script (most AI crawlers do not run JavaScript),
noindex, structured data (schema.org, FAQPage in particular), title, meta description, the H1 and the order of
headings, the canonical address, Open Graph and the declared language.
The site: robots.txt, llms.txt and the sitemap.
Reading a "Blocked"
Our requests leave from our server, not from OpenAI's or Anthropic's addresses. Some hosts and firewalls let the real GPTBot in (they check its address) and stop anything else that calls itself GPTBot. Such a site blocks us and still serves the real crawler.
When the AI traffic integration is installed, compare with what real crawlers got:
| Crawl Checker | AI traffic shows | Most likely |
|---|---|---|
| Blocked | verified visits of that crawler | protection against fake bots — nothing to fix |
| Blocked | no visit at all for weeks | the host blocks the crawler — ask them to allow it |
| Blocked | integration not installed | cannot tell — install it to be sure |
Shared hosting often blocks AI crawlers at the firewall, before the request reaches the site, so a plugin on the site never sees them. Crawl Checker does.
Watched pages
Up to ten pages of your site are re-checked on their own: the ones you pinned or added, the homepage, the pages AI answers cite most, then the pages AI crawlers read most. The list follows what AI cites, so it changes over time; a page you remove is never picked again.
The homepage, pages AI cites and pinned pages are checked every week, the others every month. Scheduled checks go easier on your site: every crawler on the homepage, three (GPTBot, ClaudeBot, PerplexityBot) on the other pages, since a hosting block covers the whole site.
Recommendations
Every problem a check finds becomes a recommendation, with what is wrong, why it matters and how to fix it. They close by themselves: when the next check of that page no longer finds the problem, the recommendation is marked resolved with the date. Dismiss one you do not want to fix (blocking a training crawler can be a choice) and the checks will not bring it back. Site-wide rules (robots.txt, llms.txt, sitemap, server blocks) are judged on homepage checks; the others on the page checked.
Recommendations also hold the comparison with cited pages: for the prompts where AI leaves your brand out, what the pages it cites have and yours does not.
Limits
Only pages of your brand's domain and its subdomains. Each manual check sends about twenty requests to your site, one at a time; a workspace can run 50 a day.
The free AI Crawl Checker
The same check, without an account and for any site, is on survelium.com/ai-crawl-checker: the crawlers, the page and the site, with a link to share the result. It does not know your real crawler visits, so a "Blocked" there cannot be told apart from a protection against fake bots; in the app it can.