Should You Block AI Crawlers? The Real Trade-off
Updated on 8 September 2026
For a restaurant trying to be seen, blocking generative AI crawlers (GPTBot, ClaudeBot, PerplexityBot) in robots.txt means voluntarily excluding yourself from ChatGPT, Claude, and Perplexity's answers. Barring a specific privacy or content-protection reason, letting these crawlers access the site is the default position consistent with a visibility goal.
The crawlers involved
| User-agent | Publisher | Use |
|---|---|---|
GPTBot | OpenAI | Training and content retrieval for ChatGPT |
OAI-SearchBot | OpenAI | Real-time search for ChatGPT |
ChatGPT-User | OpenAI | Browsing triggered by a ChatGPT user |
ClaudeBot | Anthropic | Training and retrieval for Claude |
Claude-SearchBot | Anthropic | Real-time search for Claude |
PerplexityBot | Perplexity | Content retrieval for Perplexity answers |
Google-Extended | Controls content use for Google's generative features (distinct from classic Search crawling) | |
Applebot-Extended | Apple | Controls content use for Apple's AI features |
What blocking these crawlers actually prevents
A crawler blocked in robots.txt can't read your pages' content. In practice, your business can't be cited, mentioned, or accurately described by the corresponding model, since it simply doesn't have access to your listing's or website's text to form an informed view.
This is different from accidental blocking: many sites block these crawlers without knowing it, through generic server-level anti-bot rules (WAF, fail2ban, overly broad nginx rules) that don't distinguish a legitimate AI crawler from a malicious bot. This is the most common silent failure: the site "works" for a human visitor, but AI crawlers get a 403 error without anyone noticing.
How to check your crawlers aren't blocked
- Check your
robots.txtfile directly (available atyourdomain.com/robots.txt) and confirm there's noDisallow: /rule for these user-agents. - Check server access logs (nginx
access_log) and search for these user-agents withgrep -E "GPTBot|ChatGPT-User|ClaudeBot|PerplexityBot" access.log— their presence confirms they're actually accessing the site. - If you use a WAF or third-party anti-bot service, check its configuration specifically for these user-agents, since some providers' default rules block them globally.
The rare cases where blocking makes sense
- Strictly licensed content you don't want reused to train third-party models.
- Pages containing sensitive or private information that should never be publicly synthesized.
- A deliberate business strategy of staying invisible through AI channels (rare, and generally not relevant for a local business trying to attract customers).
For a growing restaurant, none of these usually apply — visibility comes first.
For the full context on why this visibility matters, see get your restaurant recommended by ChatGPT.
SEOresto makes sure your online presence stays accessible to the crawlers that matter, AI and Google alike. From €39/month. See pricing.
Sources: public documentation for GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot (Perplexity), Google-Extended (Google), Applebot-Extended (Apple) user-agents.
Author: Dmitrii Portnov, founder of SEOresto.
