# AI Crawler / Robots.txt Strategy Audit

Generated: 2026-06-11T01:27:52.270Z

## Scope

- Target: https://example.com
- Robots URL: https://example.com/robots.txt
- Goal: allow_search
- Evidence URL: fixture://tests/fixtures/acme-robots.txt

## Findings

| Risk | Signal | Interpretation | Recommended Action |
|---|---|---|---|
| red | OpenAI OAI-SearchBot | disallow:/ for search_visibility | Allow OAI-SearchBot if the business goal is AI search visibility. |
| unknown | OpenAI ChatGPT-User | no-specific-rule for user_triggered_fetch | Review this signal against the current visibility goal before customer delivery. |
| unknown | Perplexity Perplexity-User | no-specific-rule for user_triggered_fetch | Review this signal against the current visibility goal before customer delivery. |
| unknown | Anthropic Claude-User | no-specific-rule for user_triggered_fetch | Review this signal against the current visibility goal before customer delivery. |
| unknown | Cloudflare Cloudflare-Managed-Robots-Txt | no-specific-rule for policy_advisory | Review this signal against the current visibility goal before customer delivery. |

## Bot Policy Results

| Provider | Bot | Category | Observed Rule | Risk |
|---|---|---|---|---|
| OpenAI | OAI-SearchBot | search_visibility | disallow:/ | red |
| OpenAI | GPTBot | training_crawl | disallow:/ | green |
| OpenAI | ChatGPT-User | user_triggered_fetch | no-specific-rule | unknown |
| Perplexity | PerplexityBot | search_visibility | allow:/ | green |
| Perplexity | Perplexity-User | user_triggered_fetch | no-specific-rule | unknown |
| Anthropic | ClaudeBot | training_crawl | disallow:/ | green |
| Anthropic | Claude-User | user_triggered_fetch | no-specific-rule | unknown |
| Anthropic | Claude-SearchBot | search_visibility | allow:/ | green |
| Cloudflare | Cloudflare-Managed-Robots-Txt | policy_advisory | no-specific-rule | unknown |
| Cloudflare | Content-Signal | policy_advisory | no-specific-rule | green |

## Recommended Robots.txt Patch

```txt
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

Content-Signal: search=yes, ai-train=no
```

## Caveats

- This checker reviews robots.txt policy signals only; it does not guarantee AI ranking, citations, traffic, leads, or revenue.
- Provider crawler names and behavior can drift; recheck official docs before external customer delivery.
- Do not treat this local report as customer delivery unless a paid order and delivery approval exist.

## Next Actions

- Confirm business goal: allow AI search visibility while blocking training crawls.
- Review recommended robots.txt changes with the site owner before editing production robots.txt.
- Re-run the checker after any robots.txt change and save evidence.