usmagazine.com · United States · McClatchy · measured 8 Oct 2026
As read on 8 Oct 2026, 5,510 bytes. 14 rule groups name AI crawlers; shown first.
User-agent: FacebookBot Crawl-delay: 3 User-agent: Applebot-Extended Disallow: / User-agent: Bytespider Disallow: / User-agent: CCBot Disallow: / User-agent: cohere-ai Disallow: / User-agent: Diffbot Disallow: / User-agent: Meta-ExternalAgent Disallow: / User-agent: MistralAI-User Disallow: / User-agent: MistralAI-User/1.0 Disallow: / User-agent: omgili Disallow: / User-agent: omgilibot Disallow: / User-agent: PetalBot Disallow: / User-agent: Timpibot Disallow: / User-agent: YouBot Disallow: / Sitemap: https://www.usmagazine.com/sitemap.xml Sitemap: https://www.usmagazine.com/feed/video-sitemap/ Sitemap: https://www.usmagazine.com/gallery-sitemap.xml Sitemap: https://www.usmagazine.com/feed/google-news-sitemap/ Sitemap: https://www.usmagazine.com/authors-sitemap.xml Sitemap: https://www.usmagazine.com/celebrity-sitemap.xml Sitemap: https://www.usmagazine.com/espanol-sitemap.xml Sitemap: https://www.usmagazine.com/espanol-gallery-sitemap.xml Sitemap: https://www.usmagazine.com/espanol-google-news-sitemap.xml User-agent: * Disallow: /*?share= Disallow: /*/dick-van-dyke/celebrities/ Disallow: /*/forever21.com/ Disallow: /sso-dashboard.js Disallow: /article-preview/ Disallow: /wp-json/ Disallow: /_static/*.js Disallow: /*<Macro*profile_link* Disallow: /*%3CMacro*profile_link* Disallow: /top-stories/*embed_id= User-agent: A6-Indexer Disallow: / User-agent: AddSearchBot Disallow: / User-agent: aiHitBot Disallow: / User-agent: Andibot Disallow: / User-agent: ankiro.com Disallow: / User-agent: archive.org_bot Disallow: / User-agent: AspiegelBot Disallow: / User-agent: Baiduspider Disallow: / User-agent: Barkrowler Disallow: / User-agent: bigsur.ai Disallow: / User-agent: BLP_bbot Disallow: / User-agent: Brightbot 1.0 Disallow: / User-agent: Buck Disallow: / User-agent: ChatGLM-Spider Disallow: / User-agent: CheckMarkNetwork Disallow: / User-agent: coccocbot-web Disallow: / User-agent: cohere-training-data-crawler Disallow: / User-agent: Cotoyogi Disallow: / User-agent: crawler4j Disallow: / User-agent: Crawlspace Disallow: / User-agent: dataprovider Disallow: / User-agent: Datenbank Crawler Disallow: / User-agent: EasouSpider Disallow: / User-agent: Echobot Bot Disallow: / User-agent: EchoboxBot Disallow: / User-agent: ecoResearch Disallow: / User-agent: FirecrawlAgent Disallow: / User-agent: Genieo Disallow: / User-agent: https://hada.news Disallow: / User-agent: ia_archiver Disallow: / User-agent: ia_archiver-web.archive.org Disallow: / User-agent: iaskspider/2.0 Disallow: / User-agent: ICC-Crawler Disallow: / User-agent: img2dataset Disallow: / User-agent: JenkersBot Disallow: / User-agent: Jetslide Disallow: / User-agent: Kangaroo Bot Disallow: / User-agent: KunatoCrawler Disallow: / User-agent: LivelapBot Disallow: / User-agent: magpie-crawler Disallow: / User-agent: Magus Bot Disallow: / User-agent: MauiBot Disallow: / User-agent: MAZBot Disallow: / User-agent: Meta-ExternalFetcher Disallow: / User-agent: Mojeek Disallow: / User-agent: MojeekBot Disallow: / User-agent: MyCentralAIScraperBot Disallow: / User-agent: Netvibes Disallow: / User-agent: news-please Disallow: / User-agent: NewsNow Disallow: / User-agent: nlcrawler Disallow: / User-agent: NovaAct Disallow: / User-agent: Oracle Secure Enterprise Search Disallow: / User-agent: panscient.com Disallow: / User-agent: PanguBot Disallow: / User-agent: Poseidon Research Crawler Disallow: / User-agent: Primalbot Disallow: / User-agent: QuillBot Disallow: / User-agent: quillbot.com Disallow: / User-agent: R6_CommentReader Disallow: / User-agent: Riddler Disallow: / User-agent: SBIntuitionsBot Disallow: / User-agent: Scrapy Disallow: / User-agent: Seekport Disallow: / User-agent: Seekr Disallow: / User-agent: SeekrBot Disallow: / User-agent: SeznamBot Disallow: / User-agent: SeznamHomepageCrawler Disallow: / User-agent: SemrushBot Disallow: / User-agent: Sidetrade indexer bot Disallow: / User-agent: Spinn3r Disallow: / User-agent: TaraGroup Intelligent Bot Disallow: / User-agent: Thinkbot Disallow: / User-agent: VelenPublicWebCrawler Disallow: / User-agent: Verity Disallow: / User-agent: Verity/1.1 Disallow: / User-agent: ViennaTinyBot Disallow: / User-agent: voxgov.com Disallow: / User-agent: WebVac Disallow: / User-agent: webintegration.at Disallow: / User-agent: Webzio-Extended Disallow: / User-agent: WebZip Disallow: / User-agent: WRTNBot Disallow: / User-agent: yacy Disallow: / User-agent: yacybot Disallow: / User-agent: Yandex Disallow: / User-agent: * Disallow:
Changes recorded since June 2026.
No change recorded.