Skip to content

Free tools for SEO, AI search, Google Ads and local businesses. No sign-up. See all tools

DigitalSnot

AI Crawler Checker: can ChatGPT, Perplexity and Google read your site?

See which AI crawlers can read your page, and what they see without JavaScript.

  • Free to use
  • Nothing to install
  • No sign-up

How to use the AI Crawler Checker

Read what the guide says

This is the AI Crawler Checker.

It shows which AI crawlers may read your page, and what they see without JavaScript.

If AI search crawlers can't fetch a page, AI answers can't cite it.

Use it on your homepage, or a key service page.

Type a page address. Here, it's a sample site for a made-up dental clinic.

Then press Check crawlers.

We read the site's robots.txt and test the page itself.

Here, one check needs fixing, and five passed.

Start with Do this first. Here, robots.txt blocks PerplexityBot, a search crawler.

Open How to fix to see why it matters and the steps.

Remove the rule that blocks it, or add this Allow rule above the others.

On WordPress, edit robots.txt in Yoast or Rank Math, as shown here.

It also tests for noindex, and for firewalls that block AI crawlers. Both passed.

This is what a crawler that does not run JavaScript gets.

Here, the title, H1 and text are all in the HTML.

And every crawler, one by one, with the rule that decides.

Blocking training crawlers like GPTBot is your choice, and does not remove you from AI answers.

To change it, pick a policy and copy a ready robots.txt.

Make the change, then check the page again.

You can also email yourself a PDF of the result.

Share this free tool

About the AI Crawler Checker

Enter any page. We read the site's robots.txt and show, crawler by crawler, whether ChatGPT search, Perplexity, Claude, Google, Bing and AI training crawlers may fetch it and which rule decides. We also test for page-level noindex, firewalls and bot checks that refuse AI crawlers, and show what a crawler that does not run JavaScript actually sees: the text, links, title and H1 in the raw HTML. Then copy the exact robots.txt for the policy you choose.

How it works

  1. Step 1

    Enter a page address, such as your homepage or a key service page.

  2. Step 2

    We fetch /robots.txt from that site and apply the same matching rules Google documents (RFC 9309).

  3. Step 3

    Each crawler shows Allowed or Blocked, the rule that applied and a link to the company's own documentation.

Search crawlers and training crawlers are different

Companies use separate crawlers for answering questions and for training models. OpenAI, for example, documents OAI-SearchBot for ChatGPT search and GPTBot for training, and says the two settings are independent. You can block training and still appear in AI answers.

What "user" agents mean

Agents such as ChatGPT-User fetch a page only when a person asks the assistant to open it. OpenAI notes robots.txt rules may not apply to these user-initiated visits.

Robots.txt is a request, not a lock

Reputable crawlers follow robots.txt, but it does not protect private content. Use authentication for anything that must stay private.

AI crawlers and JavaScript

Google renders JavaScript, but studies of crawler traffic (Vercel, 2024) found that the main AI crawlers, including GPTBot, ClaudeBot and PerplexityBot, download the HTML without running it. If your text, links or headings are added by JavaScript, those crawlers see an empty or partial page. The "What AI crawlers see" preview shows exactly what is in the HTML.

Frequently asked questions

Should I block GPTBot?

It is a business decision. Blocking GPTBot asks OpenAI not to use your content for training; according to OpenAI it does not remove you from ChatGPT search, which uses OAI-SearchBot. If being recommended in ChatGPT matters to you, keep OAI-SearchBot allowed.

Does blocking Google-Extended remove me from AI Overviews?

No. Google-Extended controls whether content is used for Gemini training and grounding; Google states it does not affect Search. AI Overviews are part of Search and rely on Googlebot.

My site has no robots.txt. Is that a problem?

No. Without a robots.txt file, crawlers may fetch every page. You only need one if you want to block something or point crawlers to your sitemap.

Why does a crawler show Blocked when I never mentioned it?

A rule under "User-agent: *" applies to every crawler that has no group of its own. A broad "Disallow: /" under * blocks all of them.

How quickly do changes take effect?

Crawlers re-read robots.txt regularly; OpenAI says changes can take about 24 hours for search. Google usually caches robots.txt for up to a day.

My site is built with React, Vue or Angular. Can AI crawlers read it?

Only if the content is in the HTML the server sends, through server-side rendering, static generation or prerendering. If the check shows only a few words in the HTML, AI crawlers that do not run JavaScript see little more than an empty page.

Related reading

Rather have us do it for you?

Tell us about your business and what you want to fix. We reply within one business day.

Talk to us Our service: AI search optimisation (GEO)

More free tools

All tools

Want us to fix what the tool found?

Tell us about your business and what you want to grow. We reply within one business day with next steps, not a sales script.

  • A reply within one business day
  • A quick look at your site and search visibility before we talk
  • Your details are only used to answer your enquiry

Prefer email or a call?

hello@digitalsnot.com

+91 70079 75411

Popular