Free AI Crawler Access Check: Can ChatGPT, Claude and Perplexity Read Your Site?
AI assistants can only recommend what they can read. Enter your website and see which AI crawlers your robots.txt lets in, and whether you have the files that help them understand you.
What this checks
robots.txt. Whether GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot and Google-Extended are allowed to read your site, blocked from all of it, or blocked from some paths.
llms.txt. A plain file at the top of your site that tells AI assistants what you do and where to look.
A markdown version. A clean text copy of your page that is easy for AI to read.
JSON-LD. The code that spells out your business details in a form machines understand.
The tool reads your public pages the way a crawler would. It only reports what it finds, and when it cannot tell, it says so.
How to read the result
Open means the crawler may read your whole site. Partly open means it may read your pages but some paths are closed, which is usually fine for admin areas. Blocked means your robots.txt tells that crawler to stay out of everything.
If a crawler is blocked and you want to be recommended by that assistant, ask whoever manages your site to remove or change that rule.
Google-Extended is the switch for Gemini. Blocking it does not remove you from Google Search.
Why it matters
Some hosts and security plugins block AI crawlers by default, and the site owner never finds out. If the crawler cannot read you, the assistant has nothing to say about you.
If you want me to fix what this finds, book a call.
Questions
Straight answers.
Is this really free?
Yes. There is no sign-up and nothing to pay. To stop abuse, each visitor gets a limited number of checks an hour.
What does the tool fetch from my site?
Your robots.txt, your llms.txt, your home page, and a possible markdown copy of it, once each, the way a crawler would. It does not log in or change anything.
Do you keep my website address?
No. The address is used for the check and is not saved.
Does allowed mean ChatGPT will recommend me?
No. It means the door is open. Whether an assistant names you depends on what it can find and trust about your business.
My robots.txt allows everything but a crawler shows as blocked. Why?
The check follows the robots.txt rules the way crawlers do: a rule written for one crawler by name replaces the general rules for that crawler. Read the rule shown under the result.
A check says it could not read something. What now?
The site may have blocked the tool, been slow, or been down. Try again in a few minutes, or book a call and I will look at it by hand.
Is there a WordPress plugin that does this automatically?
Yes, CrawlPress, my plugin. It checks every day, shows which AI crawlers visit your site and writes your llms.txt.
Want it done for you?
I help small businesses get found in search and in AI assistants. Tell me where you are and we will work out a plan.