CrawlerPulse
CrawlerPulse / North America / United States / thedailybeast.com

thedailybeast.com

thedailybeast.com · United States · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Allowed
OpenAI
Allowed
Anthropic
Allowed
Perplexity
Allowed
Meta
Allowed
Common Crawl
Allowed
ByteDance
Allowed
Apple
Allowed
Amazon
Allowed

robots.txt

As read on 8 Oct 2026, 547 bytes. No rule names an AI crawler.

user-agent: *
allow: /
disallow: /_server-islands/*
disallow: /account/
disallow: /membership/
disallow: /the-scorched-earth-wellness-war-between-tanya-zuckerbrot-emily-gellis-lande-and-ingrid-de-la-mare-kenny,inside-the-mean-girl-army-going-to-war-for-a-celebrity-dietician
Sitemap: https://www.thedailybeast.com/arc/outboundfeeds/sitemap-index?outputType=xml
Sitemap: https://www.thedailybeast.com/arc/outboundfeeds/sitemap-news-index?outputType=xml
Sitemap: https://www.thedailybeast.com/arc/outboundfeeds/sitemap-section-index?outputType=xml

History

Changes recorded since June 2026.

No change recorded.

In context

United States
63.2% block at least one AI crawler
Behind Cloudflare
Yes