csmonitor.com · United States · measured 8 Oct 2026
As read on 8 Oct 2026, 2,351 bytes. 18 rule groups name AI crawlers; shown first.
User-agent: anthropic-ai Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Bytespider Disallow: / User-agent: CCBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Claude-Web Disallow: / User-agent: cohere-ai Disallow: / User-agent: Diffbot Disallow: / User-agent: FacebookBot Disallow: / User-agent: Google-CloudVertexBot Disallow: / Allow: /wirecutter/ User-agent: Google-Extended Disallow: / User-agent: GPTBot Disallow: / User-agent: ImagesiftBot Disallow: / User-agent: Meta-ExternalAgent User-agent: meta-externalagent Disallow: / User-agent: omgili Disallow: / User-agent: omgilibot Disallow: / User-agent: PerplexityBot Disallow: / User-agent: Timpibot Disallow: / User-agent: Mediapartners-Google* Disallow: /includes/ Disallow: /keepalive.html User-agent: NetShelter ContentScan Disallow: / User-agent: NetShelter Disallow: / User-agent: ^NetShelter Disallow: / User-agent: sentibot Disallow: / User-agent: AwarioRssBot User-agent: AwarioSmartBot Disallow: / User-agent: FriendlyCrawler Disallow: / User-agent: magpie-crawler Disallow: / User-agent: MyCentralAIScraperBot Disallow: / User-agent: NewsNow Disallow: / User-agent: peer39_crawler User-agent: peer39_crawler/1.0 Disallow: / User-agent: Poseidon Research Crawler Disallow: / User-agent: quillbot.com Disallow: / User-agent: Scrapy Disallow: / User-agent: Reflectionbot Disallow: / User-agent: * Disallow: /sudoku Disallow: /404 Disallow: /text_edition/ Disallow: /digitalpdf/ Disallow: /content/search Disallow: /content/advancedsearch Disallow: /content/download/ Disallow: /layout/set/text/ Disallow: /layout/set/print/ Disallow: /layout/set/newsletter/ Disallow: /layout/set/ap/ Disallow: /layout/set/doc/ Disallow: /layout/set/htm/ Disallow: /layout/set/lmt/ Disallow: /layout/set/newsml/ Disallow: /layout/set/nfb/ Disallow: /layout/set/nitf/ Disallow: /layout/set/txt/ Disallow: /layout/set/txt_pq/ Disallow: /layout/set/xml/ Disallow: /layout/set/csm/ Disallow: /layout/set/basic/ Disallow: /layout/set/rssfull/ Disallow: /layout/set/tabletkiosk/ Disallow: /style/set/json/ Disallow: /content/view/full/ Disallow: /1010203/ Disallow: /tags/topic/ Disallow: /csmlists/topic/ Disallow: /csmlists/addbookmark/ Disallow: /services/ Disallow: /auth/ Crawl-delay: 1 Sitemap: https://www.csmonitor.com/sitemap-index.xml
Changes recorded since June 2026.
No change recorded.