tucsonsentinel.com · United States · measured 8 Oct 2026
As read on 8 Oct 2026, 6,913 bytes. 12 rule groups name AI crawlers; shown first.
User-agent: Applebot-Extended Disallow: / User-agent: Google-Extended Disallow: / User-agent: CCBot/2.0 Disallow: / User-agent: CCBot Disallow: / User-agent: claudebot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: GPTBot Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Amazonbot/0.1 Disallow: / User-agent: Amazonbot Disallow: / User-agent: Bytespider Disallow: / Sitemap: https://www.tucsonsentinel.com/news_sitemap.php Sitemap: https://www.tucsonsentinel.com/sitemap.php Sitemap: https://www.tucsonsentinel.com/sitemap_index.xml Sitemap: https://www.tucsonsentinel.com/sitemap-archive-index.xml User-Agent: * Content-Signal: search=yes, ai-input=no, ai-train=no, Allow: / User-agent: * Disallow: /events/ User-agent: * Disallow: /calendar/ User-agent: * Disallow: /forums/ User-agent: * Disallow: /local/comments/ User-agent: * Disallow: /arts/comments/ User-agent: * Disallow: /nationworld/comments/ User-agent: * Disallow: /opinion/comments/ User-agent: * Disallow: /sports/comments/ User-agent: * Crawl-delay: 2 User-agent: Mediapartners-Google Allow: / User-agent: Google-Display-Ads-Bot Allow: / User-agent: facebookexternalhit Allow: / User-agent: NextdoorBot Allow: / User-agent: Applebot Crawl-delay: 30 User-agent: AppleBot Crawl-delay: 30 User-agent: Amzn-SearchBot/0.1 Disallow: / User-agent: Amzn-SearchBot Disallow: / User-agent: Amzn-User Disallow: / User-agent: msnbot Disallow: / User-agent: AhrefsBot disallow: / User-agent: MauiBot Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: SemrushBot Disallow: / User-agent: SEMrushBot Disallow: / User-agent: BLP_bbot/0.1 Disallow: / User-agent: Go 1.1 package http Disallow: / User-Agent: The Knowledge AI Disallow: / User-Agent: PiplBot Disallow: / User-agent: grub-client Disallow: / User-agent: NPBot Disallow: / User-agent: Veooz Disallow: / User-agent: Veoozbot Disallow: / User-agent: GarlikCrawler Disallow: / User-agent: 008 Disallow: / User-agent: Speedy Disallow: / User-agent: Sogou web spider Disallow: / User-agent: Ezooms Disallow: / User-agent: magpie-crawler Disallow: / User-agent: YandexImages Disallow: / User-agent: MJ12bot Disallow: / User-agent: YodaoBot Disallow: / User-agent: NerdByNature.Bot Disallow: / User-agent: discobot Disallow: / User-agent: knowaboutBot Disallow:/ User-agent: UnwindFetchor Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: Zealbot Disallow: / User-agent: MSIECrawler Disallow: / User-agent: WebReaper Disallow: / User-agent: SiteSnagger Disallow: / User-agent: WebStripper Disallow: / User-agent: WebCopier Disallow: / User-agent: Fetch Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: WebZIP Disallow: / User-agent: linko Disallow: / User-agent: HTTrack Disallow: / User-agent: Microsoft.URL.Control Disallow: / User-agent: Xenu Disallow: / User-agent: larbin Disallow: / User-agent: libwww Disallow: / User-agent: ZyBORG Disallow: / User-agent: Download Ninja Disallow: / User-agent: BLEXBot Disallow: / User-agent: bingbot Crawl-delay: 10 User-agent: Slurp Crawl-delay: 60 User-agent: YahooMobile User-agent: YahooCacheSystem User-agent: Yahoo! Slurp/Site Explorer User-agent: Mozilla/4.05 [en] User-agent: LTI/LemurProject User-agent: Yahoo-Blogs User-agent: Yahoo-Blogs/v3.9 User-agent: Yahoo-MMCrawler User-agent: Yahoo-MMCrawler/3.x User-agent: YahooYSMcm User-agent: YahooYSMcm/2.0.0 User-agent: Yahoo-Test User-agent: Yahoo! Mindset User-agent: Y!J-BSC User-agent: Y!J-BSC/1.0 User-agent: Y!J-BSC/1.0(http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) User-agent: y!j-bsc User-agent: y!j-bsc/1.0 User-agent: y!j-bsc/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) User-agent: Y!J User-agent: Y!J/1.0 User-agent: Y!J/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) User-agent: y!j User-agent: y!j/1.0 User-agent: y!j/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) User-agent: Mozilla/4.0 (compatible; Y!J; for robot study; keyoshid) User-agent: Mozilla/4.0 (compatible; y!j; for robot study; keyoshid) User-agent: Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html) User-agent: Mozilla/5.0 (compatible; Yahoo! DE Slurp; http://help.yahoo.com/help/us/ysearch/slurp) User-agent: Mozilla/5.0 (Yahoo-Test/4.0 mailto:vertical-crawl-support@yahoo-inc.com) User-agent: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) Disallow: /
Changes recorded since June 2026.
No change recorded.