Loading CheckWebs…
Loading CheckWebs…
Public AI Access tool
Audit your robots.txt for AI/LLM crawler directives. Detect whether GPTBot, ClaudeBot, Google-Extended, PerplexityBot, and other AI agents can access public content, and separate retrieval, training, and classic search crawler policy.
What it checks
Check if AI crawlers (GPTBot, ClaudeBot) are blocked in robots.txt.
Review semantic structure, metadata, schema, and crawl permissions.
Look for clear answers, factual density, authorship, and source context.
Prioritize changes that help answer engines quote the right content.
Methodology
Next steps
Interpretation
Use cases
Improve crawl access, extractability, structured data clarity, and source context.
Check whether pages are structured for snippets, answers, and source references.
Align robots, metadata, schema, and llms.txt with your public AI crawl policy.
Popular checks
Next paths
Use these pages when the result points to a broader check, a fix workflow, or a related technical question.
Fix workflows
Fix robots.txt rules that block Googlebot from important pages while preserving protection for private routes.
Open fix guideSeparate AI retrieval, AI training, and classic search crawler rules so your policy matches what should be public.
Open fix guideFrequently asked questions
No. Enter a URL or domain and run the check in your browser. Some results can be partial if the target site blocks requests or hides the data being checked.
The AI Crawler Audit runs live checks against the target website. Results reflect what was reachable at the time of the test, so retest after changing DNS, redirects, headers, or page content.
Allow only the crawlers that match your public-content policy. Separate search or retrieval crawlers from training controls, keep private paths blocked, and document why each rule exists.
Some crawlers separate these functions. For example, blocking Google-Extended prevents Gemini training but Google Search still indexes your site normally. ChatGPT-User handles live browsing while GPTBot handles training.
Crawler access is only one part of public AI access. If important public pages are blocked, AI systems may have less usable content to retrieve; if everything is open without policy, private or low-value areas may be exposed unnecessarily.