The Day

theday.com · United States · measured 9 Oct 2026

Google Search

Allowed

Google-Extended

Blocked

OpenAI

Blocked

Anthropic

Blocked

Perplexity

Blocked

robots.txt

As read on 9 Oct 2026, 957 bytes. 3 rule groups name AI crawlers; shown first.

User-agent: ChatGPT-User
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ChatGPT-User/2.0
User-agent: anthropic-ai
User-agent: ClaudeBot
User-agent: claude-web
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Amazonbot
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Bytespider
User-agent: DuckAssistBot
User-agent: cohere-ai
User-agent: AI2Bot
User-agent: CCBot
User-agent: Diffbot
User-agent: omgili
User-agent: TimpiBot
User-agent: YouBot
User-agent: MistralAI-User
Disallow: /

User-agent: *
Disallow: /images/
Disallow: /mce/
Disallow: /multifileupload/
Disallow: /oauth/
Disallow: /support/
Disallow: /typefaces/
Disallow: /zeroclipboard/
Disallow: /user/login/
Disallow: /user/register/

User-agent: facebookexternalhit
Allow: /

User-agent: Slurp
Disallow: /

User-agent: AdsBot-Google
Allow: /

History

Changes recorded since June 2026.

No change recorded.

In context

United States

63.2% block at least one AI crawler

Behind Cloudflare