Public AI Access

AI crawler appears blocked

Check whether AI retrieval or training crawlers are blocked intentionally, accidentally, or through wildcard robots rules.

Run AI Crawler Audit Browse workflows

Workflow shape

1Confirm the symptom

2Collect live evidence

3Fix in the right order

4Retest the same URL

What this means

AI crawler policy should be intentional. A wildcard Disallow can block useful public retrieval while bot-specific rules can be misunderstood if the site does not separate crawler types.

Use this when

AI Crawler Audit reports blocked or partially blocked status.
You are unsure whether robots.txt rules block retrieval, training, or search crawlers.
A policy change was deployed and needs production validation.

Evidence

What to collect before changing anything

robots.txt content from production, including wildcard and named user-agent groups.
Crawler names affected, such as GPTBot, ChatGPT-User, ClaudeBot, Google-Extended, or PerplexityBot.
Public paths that should be accessible and private paths that should remain blocked.
Sitemap and llms.txt references for public content.

Diagnosis

Run the checks in this order

1Run AI Crawler Audit and note which crawlers are blocked, allowed, or unclear.
2Run robots.txt Analyzer and compare wildcard rules with bot-specific rules.
3Decide whether each crawler category matches your business policy.
4Run Public AI Access Check on pages you want visible to public AI retrieval.
5Document intentional restrictions so future changes do not reverse them accidentally.

Fix plan

What to fix first

1Keep classic search crawler rules separate from AI crawler rules.
2Allow public retrieval paths only when that matches your policy.
3Keep admin, account, API, report, and private paths blocked.
4Avoid broad Disallow: / rules unless the site truly should not be crawled.
5Retest after deploy and review logs if available.

Retest

How to confirm the fix worked

Re-run AI Crawler Audit and robots.txt Analyzer.
Fetch robots.txt from production, not staging.
Check that sitemap and llms.txt still point to public, crawlable pages.

FAQ

Common questions

Should every AI crawler be allowed?+

No. This is a business policy decision. The important part is separating intentional restrictions from accidental blocks.

Does allowing an AI crawler guarantee visibility?+

No. It only removes an access barrier. Content quality, structure, trust, and demand still matter.

Loading CheckWebs…