# Crawl Checker

> Whether AI crawlers can read a page of your site, what they get from it, and how to read the result.

**Crawl Checker** asks one page of your site the way AI crawlers do, and tells you whether they get in and what they
find. Nothing in it uses AI: every verdict is a rule over the answers our requests got.

## What it checks

**AI crawler access**, for twenty crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot…):

- **robots.txt** — read the way crawlers read it: a group can name several crawlers, a crawler follows its own group
  and only falls back to `User-agent: *` when none names it, and the most specific rule wins. A robots.txt that cannot
  be downloaded (server error, timeout) counts as "everything disallowed", because that is what crawlers do.
- **The server** — we request the page with each crawler's user agent. A refusal (403, 401, a challenge page such as
  Cloudflare's "Just a moment…") is **Blocked**; a 429 is a rate limit, shown apart; no answer is **Unreachable**.
  A second request as an ordinary browser tells a crawler block from a site that is down.

Google-Extended, Applebot-Extended and anthropic-ai only exist in robots.txt (the real crawler announces itself under
another name), so they are judged on robots.txt alone.

**The page**: whether the text is in the HTML or loaded by script (most AI crawlers do not run JavaScript),
`noindex`, structured data (schema.org, `FAQPage` in particular), title, meta description, the H1 and the order of
headings, the canonical address, Open Graph and the declared language.

**The site**: robots.txt, `llms.txt` and the sitemap.

## Reading a "Blocked"

Our requests leave from our server, not from OpenAI's or Anthropic's addresses. Some hosts and firewalls let the real
GPTBot in (they check its address) and stop anything else that calls itself GPTBot. Such a site blocks us and still
serves the real crawler.

When the [AI traffic](https://survelium.com/docs/ai-traffic.md) integration is installed, compare with what real crawlers got:

| Crawl Checker | AI traffic shows | Most likely |
|---|---|---|
| Blocked | verified visits of that crawler | protection against fake bots — nothing to fix |
| Blocked | no visit at all for weeks | the host blocks the crawler — ask them to allow it |
| Blocked | integration not installed | cannot tell — install it to be sure |

Shared hosting often blocks AI crawlers at the firewall, before the request reaches the site, so a plugin on the site
never sees them. Crawl Checker does.

## Watched pages

Up to ten pages of your site are re-checked on their own: the ones you pinned or added, the homepage, the pages AI
answers cite most, then the pages AI crawlers read most. The list follows what AI cites, so it changes over time; a
page you remove is never picked again.

The homepage, pages AI cites and pinned pages are checked every week, the others every month. Scheduled checks go easier
on your site: every crawler on the homepage, three (GPTBot, ClaudeBot, PerplexityBot) on the other pages, since a
hosting block covers the whole site.

## Recommendations

Every problem a check finds becomes a recommendation, with what is wrong, why it matters and how to fix it. They
close by themselves: when the next check of that page no longer finds the problem, the recommendation is marked resolved
with the date. Dismiss one you do not want to fix (blocking a training crawler can be a choice) and the checks will not
bring it back. Site-wide rules (robots.txt, llms.txt, sitemap, server blocks) are judged on homepage checks; the others
on the page checked.

Recommendations also hold the [comparison with cited pages](https://survelium.com/docs/optimization/page-comparison.md): for the prompts where
AI leaves your brand out, what the pages it cites have and yours does not.

## Limits

Only pages of your brand's domain and its subdomains. Each manual check sends about twenty requests to your site, one at
a time; a workspace can run 50 a day.

## The free AI Crawl Checker

The same check, without an account and for any site, is on [survelium.com/ai-crawl-checker](/ai-crawl-checker): the
crawlers, the page and the site, with a link to share the result. It does not know your real crawler visits, so a
"Blocked" there cannot be told apart from a protection against fake bots; in the app it can.