Free tool · No signup · Instant result
Is your site blocking GPTBot and other AI crawlers?
ChatGPT, Perplexity, Claude and Google's AI answers read the web with crawlers like GPTBot, PerplexityBot and ClaudeBot — and if your robots.txt blocks them, AI assistants can't mention or recommend you. This free AI crawler checker reads your robots.txt and headers live and shows, in seconds, exactly which AI crawlers have access to your site and which are blocked. No signup, and you'll get a ready-to-paste robots.txt fix for any bot you need to allow.
We read your robots.txt and check the real AI crawler rules. We never store your site — this runs live each time.
Crawler access is one piece of the puzzle
Letting AI bots in is step one. Run a free, full SEO scorecard on your homepage — 20 checks, a 0–100 score, and a plain-language list of what to fix first so you actually get found and recommended. No signup to see your score.
Frequently asked questions
How do I check if my site is blocking GPTBot?
Enter your domain in the AI crawler checker above. It fetches your robots.txt and response headers live and shows whether GPTBot — along with PerplexityBot, ClaudeBot, Google-Extended, CCBot and others — is allowed or blocked. It also flags when a firewall or CDN like Cloudflare is blocking AI bots even though robots.txt allows them. It's a one-shot, live check, so you can re-run it any time you change your rules.
What does a GPTBot robots.txt rule look like?
To allow OpenAI's crawler, add a line 'User-agent: GPTBot' followed by 'Allow: /' to the robots.txt file at your domain root; to block it, use 'Disallow: /' instead. Our checker generates a ready-to-paste snippet that allows every major AI crawler in one go — GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, Google-Extended, CCBot and more.
Should I block AI crawlers or allow them?
If you want your business to be mentioned and recommended inside ChatGPT, Perplexity and Google's AI answers, allow them — blocking AI crawlers rules out any chance of being cited there. Allowing them makes you eligible but doesn't guarantee a mention; the AI still chooses what to surface. Only block AI crawlers if you specifically don't want your content used for AI answers or training.
Can this tool monitor my AI crawler access over time?
This free checker is a one-shot, live check — run it whenever you want a fresh read on which AI crawlers can access your site. Ongoing weekly monitoring, with alerts when your AI crawler access or rankings change, is part of the paid RankAgent product, not this free tool.
Related free tools
Most freelancers run a few of these together. Keep going with a related check — all free, no login.