vox.com · United States · Vox Media · measured 8 Oct 2026
As read on 8 Oct 2026, 7,548 bytes. 24 rule groups name AI crawlers; shown first.
User-agent: Amazonbot Allow: / User-agent: GPTBot Allow: / User-agent: anthropic-ai Disallow: / Allow: /sp/ Allow: /strategist User-agent: Applebot-Extended Disallow: / Allow: /sp/ Allow: /strategist User-agent: Bytespider Disallow: / Allow: /sp/ Allow: /strategist User-agent: CCBot Disallow: / Allow: /sp/ Allow: /strategist User-agent: ChatGPT-User Disallow: / Allow: /sp/ Allow: /strategist User-agent: Claude-Web Disallow: / Allow: /sp/ Allow: /strategist User-agent: ClaudeBot Disallow: / Allow: /sp/ Allow: /strategist User-agent: cohere-ai Disallow: / Allow: /sp/ Allow: /strategist User-agent: DeepSeekBot Disallow: / Allow: /sp/ Allow: /strategist User-agent: Diffbot Disallow: / Allow: /sp/ Allow: /strategist User-agent: FacebookBot Disallow: / Allow: /sp/ Allow: /strategist User-agent: Google-CloudVertexBot Disallow: / Allow: /sp/ Allow: /strategist User-agent: Google-Extended Disallow: / Allow: /sp/ Allow: /strategist User-agent: ImagesiftBot Disallow: / Allow: /sp/ Allow: /strategist User-agent: meta-externalagent Disallow: / Allow: /sp/ Allow: /strategist User-agent: MistralAI-User Disallow: / Allow: /sp/ Allow: /strategist User-agent: omgili Disallow: / Allow: /sp/ Allow: /strategist User-agent: Omgilibot Disallow: / Allow: /sp/ Allow: /strategist User-agent: PerplexityBot Disallow: / Allow: /sp/ Allow: /strategist User-agent: PetalBot Disallow: / Allow: /sp/ Allow: /strategist User-agent: Timpibot Disallow: / Allow: /sp/ Allow: /strategist User-agent: YouBot Disallow: / Allow: /sp/ Allow: /strategist User-agent: Applebot Allow: / User-agent: A6-Indexer Disallow: / Allow: /sp/ Allow: /strategist User-agent: AhrefsBot Disallow: / Allow: /sp/ Allow: /strategist User-agent: AI2Bot Disallow: / Allow: /sp/ Allow: /strategist User-agent: AI2Bot-DeepResearchEval Disallow: / Allow: /sp/ Allow: /strategist User-agent: Ai2Bot-Dolma Disallow: / Allow: /sp/ Allow: /strategist User-agent: aiHitBot Disallow: / Allow: /sp/ Allow: /strategist User-agent: Anchor Browser Disallow: / Allow: /sp/ Allow: /strategist User-agent: ApifyBot Disallow: / Allow: /sp/ Allow: /strategist User-agent: ApifyWebsiteContentCrawler Disallow: / Allow: /sp/ Allow: /strategist User-agent: Baiduspider Disallow: / Allow: /sp/ Allow: /strategist User-agent: betaBot Disallow: / Allow: /sp/ Allow: /strategist User-agent: BLEXBot Disallow: / Allow: /sp/ Allow: /strategist User-agent: botify Disallow: / Allow: /sp/ Allow: /strategist User-agent: brands-bot-logo Disallow: / Allow: /sp/ Allow: /strategist User-agent: Brightbot Disallow: / Allow: /sp/ Allow: /strategist User-agent: calculon Disallow: / Allow: /sp/ Allow: /strategist User-agent: ChatGLM-Spider Disallow: / Allow: /sp/ Allow: /strategist User-agent: Claude-SearchBot Disallow: / Allow: /sp/ Allow: /strategist User-agent: Claude-User Disallow: / Allow: /sp/ Allow: /strategist User-agent: CloudVertexBot Disallow: / Allow: /sp/ Allow: /strategist User-agent: cohere-training-data-crawler Disallow: / Allow: /sp/ Allow: /strategist User-agent: collection@infegy.com Disallow: / Allow: /sp/ Allow: /strategist User-agent: Cotoyogi Disallow: / Allow: /sp/ Allow: /strategist User-agent: Crawl4AI Disallow: / Allow: /sp/ Allow: /strategist User-agent: CrunchBot Disallow: / Allow: /sp/ Allow: /strategist User-agent: deepcrawl Disallow: / Allow: /sp/ Allow: /strategist User-agent: DirBuster Disallow: / Allow: /sp/ Allow: /strategist User-agent: DTS Disallow: / Allow: /sp/ Allow: /strategist User-agent: DuckAssistBot Disallow: / Allow: /sp/ Allow: /strategist User-agent: EasouSpider Disallow: / Allow: /sp/ Allow: /strategist User-agent: FirecrawlAgent Disallow: / Allow: /sp/ Allow: /strategist User-agent: FriendlyCrawler Disallow: / Allow: /sp/ Allow: /strategist User-agent: GlutenCrawler Disallow: / Allow: /sp/ Allow: /strategist User-agent: GrapeshotCrawler Disallow: / Allow: /sp/ Allow: /strategist User-agent: iaskspider Disallow: / Allow: /sp/ Allow: /strategist User-agent: ICC-Crawler Disallow: / Allow: /sp/ Allow: /strategist User-agent: img2dataset Disallow: / Allow: /sp/ Allow: /strategist User-agent: JackCrawl Disallow: / Allow: /sp/ Allow: /strategist User-agent: Jetslide Disallow: / Allow: /sp/ Allow: /strategist User-agent: linkapedia_bot Disallow: / Allow: /sp/ Allow: /strategist User-agent: Mediatoolkitbot Disallow: / Allow: /sp/ Allow: /strategist User-agent: MegaIndex Disallow: / Allow: /sp/ Allow: /strategist User-agent: meta-externalfetcher Disallow: / Allow: /sp/ Allow: /strategist User-agent: Microsoft BITS Disallow: / Allow: /sp/ Allow: /strategist User-agent: Microsoft URL Control Disallow: / Allow: /sp/ Allow: /strategist User-agent: Microsoft-WebDAV-MiniRedir Disallow: / Allow: /sp/ Allow: /strategist User-agent: news-please Disallow: / Allow: /sp/ Allow: /strategist User-agent: Novellum Disallow: / Allow: /sp/ Allow: /strategist User-agent: NTENTbot Disallow: / Allow: /sp/ Allow: /strategist User-agent: Nutch Disallow: / Allow: /sp/ Allow: /strategist User-agent: OpenLinkProfiler Disallow: / Allow: /sp/ Allow: /strategist User-agent: Perplexity-User Disallow: / Allow: /sp/ Allow: /strategist User-agent: Primalbot Disallow: / Allow: /sp/ Allow: /strategist User-agent: python-requests Disallow: / Allow: /sp/ Allow: /strategist User-agent: Re-Animator Disallow: / Allow: /sp/ Allow: /strategist User-agent: Riddler Disallow: / Allow: /sp/ Allow: /strategist User-agent: SafeDNSBot Disallow: / Allow: /sp/ Allow: /strategist User-agent: SBIntuitionsBot Disallow: / Allow: /sp/ Allow: /strategist User-agent: Scrapy Disallow: / Allow: /sp/ Allow: /strategist User-agent: SearchAtlas Disallow: / Allow: /sp/ Allow: /strategist User-agent: SemanticScholarBot Disallow: / Allow: /sp/ Allow: /strategist User-agent: SemrushBot Disallow: / Allow: /sp/ Allow: /strategist User-agent: SentiBot Disallow: / Allow: /sp/ Allow: /strategist User-agent: Sogou Disallow: / Allow: /sp/ Allow: /strategist User-agent: Spider-Pig Disallow: / Allow: /sp/ Allow: /strategist User-agent: sqlmap Disallow: / Allow: /sp/ Allow: /strategist User-agent: StorygizeBot Disallow: / Allow: /sp/ Allow: /strategist User-agent: Test Certificate Info Disallow: / Allow: /sp/ Allow: /strategist User-agent: TikTokSpider Disallow: / Allow: /sp/ Allow: /strategist User-agent: Turnitin Disallow: / Allow: /sp/ Allow: /strategist User-agent: uipbot Disallow: / Allow: /sp/ Allow: /strategist User-agent: vBSEO Disallow: / Allow: /sp/ Allow: /strategist User-agent: VegeBot Disallow: / Allow: /sp/ Allow: /strategist User-agent: VelenPublicWebCrawler Disallow: / Allow: /sp/ Allow: /strategist User-agent: Web Link Validator Disallow: / Allow: /sp/ Allow: /strategist User-agent: WebCruiser Disallow: / Allow: /sp/ Allow: /strategist User-agent: webzio-extended Disallow: / Allow: /sp/ Allow: /strategist User-agent: Xenu Disallow: / Allow: /sp/ Allow: /strategist User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php User-agent: Googlebot-News Disallow: /ad/ Disallow: /sponsored/ Disallow: /sp/ User-agent: * Allow: /sp/ Disallow: /admin Disallow: /newfanshot Disallow: /users/*/replies Disallow: /users/*/comments Disallow: /login Disallow: /account Disallow: /auth/* Disallow: /chorus_auth Disallow: /sso Disallow: /search Disallow: /the-highlight$ Disallow: /share$ Disallow: /share/* Disallow: /share?* Sitemap: https://www.vox.com/sitemaps/google_news Sitemap: https://www.vox.com/sitemaps
Changes recorded since June 2026.
No change recorded.