# Free AI Crawler Access Check: Can ChatGPT, Claude and Perplexity Read Your Site?

AI assistants can only recommend what they can read. Enter your website and see which AI crawlers your robots.txt lets in, and whether you have the files that help them understand you.

Tool: https://sethshoultes.com/tools/ai-crawler-access-check/

## What this checks

**robots.txt.** Whether GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot and Google-Extended are allowed to read your site, blocked from all of it, or blocked from some paths.

**llms.txt.** A plain file at the top of your site that tells AI assistants what you do and where to look.

**A markdown version.** A clean text copy of your page that is easy for AI to read.

**JSON-LD.** The code that spells out your business details in a form machines understand.

The tool reads your public pages the way a crawler would. It only reports what it finds, and when it cannot tell, it says so.

## How to read the result

**Open** means the crawler may read your whole site. **Partly open** means it may read your pages but some paths are closed, which is usually fine for admin areas. **Blocked** means your robots.txt tells that crawler to stay out of everything.

If a crawler is blocked and you want to be recommended by that assistant, ask whoever manages your site to remove or change that rule.

Google-Extended is the switch for Gemini. Blocking it does not remove you from Google Search.

## Why it matters

Some hosts and security plugins block AI crawlers by default, and the site owner never finds out. If the crawler cannot read you, the assistant has nothing to say about you.

If you want me to fix what this finds, [book a call](/consulting/#contact).

## Questions

### Is this really free?

Yes. There is no sign-up and nothing to pay. To stop abuse, each visitor gets a limited number of checks an hour.

### What does the tool fetch from my site?

Your robots.txt, your llms.txt, your home page, and a possible markdown copy of it, once each, the way a crawler would. It does not log in or change anything.

### Do you keep my website address?

No. The address is used for the check and is not saved.

### Does allowed mean ChatGPT will recommend me?

No. It means the door is open. Whether an assistant names you depends on what it can find and trust about your business.

### My robots.txt allows everything but a crawler shows as blocked. Why?

The check follows the robots.txt rules the way crawlers do: a rule written for one crawler by name replaces the general rules for that crawler. Read the rule shown under the result.

### A check says it could not read something. What now?

The site may have blocked the tool, been slow, or been down. Try again in a few minutes, or book a call and I will look at it by hand.

### Is there a WordPress plugin that does this automatically?

Yes, CrawlPress, my plugin. It checks every day, shows which AI crawlers visit your site and writes your llms.txt.

## On WordPress?

CrawlPress does this automatically, every day: https://crawlpress.com/

## Work with me

Book a call: https://sethshoultes.com/consulting/#contact

All free tools: https://sethshoultes.com/tools/