CrawlerPulse
CrawlerPulse / North America / United States / cbsnews.com

CBS News

cbsnews.com · United States · Paramount Skydance · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Allowed
OpenAI
Blocked
Anthropic
Allowed
Perplexity
Allowed
Meta
Allowed
Common Crawl
Allowed
ByteDance
Allowed
Apple
Allowed
Amazon
Allowed

robots.txt

As read on 8 Oct 2026, 950 bytes. 1 rule group name AI crawlers; shown first.

User-agent: GPTBot
Disallow: /

User-agent: *
Disallow: */search/*
Disallow: /index.php/*
Disallow: /election-results-data/*
Disallow: /common
Disallow: /htdocs/*
Disallow: /user/login*
Disallow: /app/?*
Disallow: /network
Disallow: /assets
Disallow: /stories
Disallow: /sections
Disallow: */feed/
Disallow: /1318
Disallow: /1319
Disallow: /1328
Disallow: /1344
Disallow: /1770
Disallow: /2240
Disallow: /2300
Disallow: /2303
Disallow: /2304
Disallow: /2305
Disallow: /2306
Disallow: /2307
Disallow: /2308
Disallow: /2332
Disallow: /2741
Disallow: /2994
Disallow: /8301
Disallow: /8324
Disallow: /8601
Disallow: /8614
Disallow: /8618
Disallow: /9742
Disallow: /9744
Sitemap: https://www.cbsnews.com/xml-sitemap/index.xml

User-agent: MAZBot
Disallow: /

User-agent: panscient.com
Disallow: /

User-agent: proximic
Disallow:

History

Changes recorded since June 2026.

No change recorded.

In context

United States
63.2% block at least one AI crawler
Behind Cloudflare
No
Same group
cbs.com, cbssports.com