Free tool · No signup · Instant result
Is your site blocking GPTBot and other AI crawlers?
ChatGPT, Perplexity, Claude and Google's AI answers read the web with crawlers like GPTBot, PerplexityBot and ClaudeBot — and if your robots.txt blocks them, AI assistants can't mention or recommend you. This free AI crawler checker reads your robots.txt live, test-loads your homepage as the bots themselves, and shows, in seconds, exactly which AI crawlers have access to your site and which are blocked. No signup, and you'll get a ready-to-paste robots.txt fix for any bot you need to allow.
We read your robots.txt live and test-load your homepage as a normal browser and as GPTBot, ClaudeBot and PerplexityBot. Nothing is stored.
One CMS update can quietly block AI bots again
RankAgent re-audits your sites every week — AI crawler access, llms.txt, schema and the rest of the technical checks — and turns each problem it finds into a fix you can hand over. Free for 14 days, no card.
Frequently asked questions
How do I check if my site is blocking GPTBot?
Enter your domain in the AI crawler checker above. It reads your robots.txt live, test-loads your homepage as GPTBot, ClaudeBot and PerplexityBot, and shows whether GPTBot — along with PerplexityBot, ClaudeBot, Google-Extended, CCBot and others — is allowed or blocked. It also flags when a firewall or CDN like Cloudflare is blocking AI bots even though robots.txt allows them. It's a one-shot, live check, so you can re-run it any time you change your rules.
What does a GPTBot robots.txt rule look like?
To allow OpenAI's crawler, add a line 'User-agent: GPTBot' followed by 'Allow: /' to the robots.txt file at your domain root; to block it, use 'Disallow: /' instead. Our checker reads your actual robots.txt and gives you the exact lines to delete and add for the crawlers it found blocked — so you can unblock the ones that power AI answers while keeping training crawlers out if you prefer.
Should I block AI crawlers or allow them?
If you want your business to be mentioned and recommended inside ChatGPT, Perplexity and Google's AI answers, allow them — blocking AI crawlers rules out any chance of being cited there. Allowing them makes you eligible but doesn't guarantee a mention; the AI still chooses what to surface. Only block AI crawlers if you specifically don't want your content used for AI answers or training.
Can this tool monitor my AI crawler access over time?
This free checker is a one-shot, live check — run it whenever you want a fresh read on which AI crawlers can access your site. Ongoing weekly re-audits that include AI crawler access are part of the paid RankAgent product, not this free tool.
Related free tools
Most freelancers run a few of these together. Keep going with a related check — all free, no login.