sbnation.com · United States · Vox Media · measured 8 Oct 2026
As read on 8 Oct 2026, 5,650 bytes. 24 rule groups name AI crawlers; shown first.
User-agent: Amazonbot Allow: / User-agent: GPTBot Allow: / User-agent: anthropic-ai Allow: /sp/ Disallow: / User-agent: Applebot-Extended Allow: /sp/ Disallow: / User-agent: Bytespider Allow: /sp/ Disallow: / User-agent: CCBot Allow: /sp/ Disallow: / User-agent: ChatGPT-User Allow: /sp/ Disallow: / User-agent: Claude-Web Allow: /sp/ Disallow: / User-agent: ClaudeBot Allow: /sp/ Disallow: / User-agent: cohere-ai Allow: /sp/ Disallow: / User-agent: DeepSeekBot Allow: /sp/ Disallow: / User-agent: Diffbot Allow: /sp/ Disallow: / User-agent: FacebookBot Allow: /sp/ Disallow: / User-agent: Google-CloudVertexBot Allow: /sp/ Disallow: / User-agent: Google-Extended Allow: /sp/ Disallow: / User-agent: ImagesiftBot Allow: /sp/ Disallow: / User-agent: meta-externalagent Allow: /sp/ Disallow: / User-agent: MistralAI-User Allow: /sp/ Disallow: / User-agent: omgili Allow: /sp/ Disallow: / User-agent: Omgilibot Allow: /sp/ Disallow: / User-agent: PerplexityBot Allow: /sp/ Disallow: / User-agent: PetalBot Allow: /sp/ Disallow: / User-agent: Timpibot Allow: /sp/ Disallow: / User-agent: YouBot Allow: /sp/ Disallow: / User-agent: Applebot Allow: / User-agent: A6-Indexer Allow: /sp/ Disallow: / User-agent: AhrefsBot Allow: /sp/ Disallow: / User-agent: AI2Bot Allow: /sp/ Disallow: / User-agent: AI2Bot-DeepResearchEval Allow: /sp/ Disallow: / User-agent: Ai2Bot-Dolma Allow: /sp/ Disallow: / User-agent: aiHitBot Allow: /sp/ Disallow: / User-agent: Anchor Browser Allow: /sp/ Disallow: / User-agent: ApifyBot Allow: /sp/ Disallow: / User-agent: ApifyWebsiteContentCrawler Allow: /sp/ Disallow: / User-agent: Baiduspider Allow: /sp/ Disallow: / User-agent: betaBot Allow: /sp/ Disallow: / User-agent: BLEXBot Allow: /sp/ Disallow: / User-agent: botify Allow: /sp/ Disallow: / User-agent: brands-bot-logo Allow: /sp/ Disallow: / User-agent: Brightbot Allow: /sp/ Disallow: / User-agent: calculon Allow: /sp/ Disallow: / User-agent: ChatGLM-Spider Allow: /sp/ Disallow: / User-agent: Claude-SearchBot Allow: /sp/ Disallow: / User-agent: Claude-User Allow: /sp/ Disallow: / User-agent: CloudVertexBot Allow: /sp/ Disallow: / User-agent: cohere-training-data-crawler Allow: /sp/ Disallow: / User-agent: collection@infegy.com Allow: /sp/ Disallow: / User-agent: Cotoyogi Allow: /sp/ Disallow: / User-agent: Crawl4AI Allow: /sp/ Disallow: / User-agent: CrunchBot Allow: /sp/ Disallow: / User-agent: deepcrawl Allow: /sp/ Disallow: / User-agent: DirBuster Allow: /sp/ Disallow: / User-agent: DTS Allow: /sp/ Disallow: / User-agent: DuckAssistBot Allow: /sp/ Disallow: / User-agent: EasouSpider Allow: /sp/ Disallow: / User-agent: FirecrawlAgent Allow: /sp/ Disallow: / User-agent: FriendlyCrawler Allow: /sp/ Disallow: / User-agent: GlutenCrawler Allow: /sp/ Disallow: / User-agent: GrapeshotCrawler Allow: /sp/ Disallow: / User-agent: iaskspider Allow: /sp/ Disallow: / User-agent: ICC-Crawler Allow: /sp/ Disallow: / User-agent: img2dataset Allow: /sp/ Disallow: / User-agent: JackCrawl Allow: /sp/ Disallow: / User-agent: Jetslide Allow: /sp/ Disallow: / User-agent: linkapedia_bot Allow: /sp/ Disallow: / User-agent: Mediatoolkitbot Allow: /sp/ Disallow: / User-agent: MegaIndex Allow: /sp/ Disallow: / User-agent: meta-externalfetcher Allow: /sp/ Disallow: / User-agent: Microsoft BITS Allow: /sp/ Disallow: / User-agent: Microsoft URL Control Allow: /sp/ Disallow: / User-agent: Microsoft-WebDAV-MiniRedir Allow: /sp/ Disallow: / User-agent: news-please Allow: /sp/ Disallow: / User-agent: Novellum Allow: /sp/ Disallow: / User-agent: NTENTbot Allow: /sp/ Disallow: / User-agent: Nutch Allow: /sp/ Disallow: / User-agent: OpenLinkProfiler Allow: /sp/ Disallow: / User-agent: Perplexity-User Allow: /sp/ Disallow: / User-agent: Primalbot Allow: /sp/ Disallow: / User-agent: python-requests Allow: /sp/ Disallow: / User-agent: Re-Animator Allow: /sp/ Disallow: / User-agent: Riddler Allow: /sp/ Disallow: / User-agent: SafeDNSBot Allow: /sp/ Disallow: / User-agent: SBIntuitionsBot Allow: /sp/ Disallow: / User-agent: Scrapy Allow: /sp/ Disallow: / User-agent: SearchAtlas Allow: /sp/ Disallow: / User-agent: SemanticScholarBot Allow: /sp/ Disallow: / User-agent: SemrushBot Allow: /sp/ Disallow: / User-agent: SentiBot Allow: /sp/ Disallow: / User-agent: Sogou Allow: /sp/ Disallow: / User-agent: Spider-Pig Allow: /sp/ Disallow: / User-agent: sqlmap Allow: /sp/ Disallow: / User-agent: StorygizeBot Allow: /sp/ Disallow: / User-agent: Test Certificate Info Allow: /sp/ Disallow: / User-agent: TikTokSpider Allow: /sp/ Disallow: / User-agent: Turnitin Allow: /sp/ Disallow: / User-agent: uipbot Allow: /sp/ Disallow: / User-agent: vBSEO Allow: /sp/ Disallow: / User-agent: VegeBot Allow: /sp/ Disallow: / User-agent: VelenPublicWebCrawler Allow: /sp/ Disallow: / User-agent: Web Link Validator Allow: /sp/ Disallow: / User-agent: WebCruiser Allow: /sp/ Disallow: / User-agent: webzio-extended Allow: /sp/ Disallow: / User-agent: Xenu Allow: /sp/ Disallow: / User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php User-agent: Googlebot-News Disallow: /ad/ Disallow: /sponsored/ Disallow: /sp/ User-agent: * Allow: /sp/ Disallow: /admin Disallow: /newfanshot Disallow: /users/*/replies Disallow: /users/*/comments Disallow: /login Disallow: /account Disallow: /auth/* Disallow: /chorus_auth Disallow: /sso Disallow: /search Disallow: /the-highlight$ Disallow: /share$ Disallow: /share/* Disallow: /share?* Sitemap: https://sbnation.com/sitemaps/google_news Sitemap: https://sbnation.com/sitemaps
Changes recorded since June 2026.
No change recorded.