sportsbusinessjournal.com · United States · measured 9 Oct 2026
As read on 9 Oct 2026, 6,488 bytes. 16 rule groups name AI crawlers; shown first.
User-agent: GPTBot Disallow: / User-agent: Bytespider Disallow: / User-agent: PetalBot Disallow: / User-agent: Amazonbot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: OAI-SearchBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: anthropic-ai Disallow: / User-agent: PerplexityBot Disallow: / User-agent: CCBot Disallow: / User-agent: meta-externalagent Disallow: / User-agent: MistralAI-User Disallow: / User-agent: YouBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: cohere-ai Disallow: / User-agent: AwarioSmartBot Disallow: / User-agent: Dalvik/2.1.0 Disallow: / User-agent: Edge Health Probe Disallow: / User-agent: IPTV Smarters/1.3.3 Disallow: / User-agent: GnowitNewsbot Disallow: / User-agent: MeltwaterNews Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729) Disallow: / User-agent: TikTokSpider Disallow: / User-agent: AhrefsBot Disallow: / User-agent: SemrushBot Disallow: / User-agent: NetworkingExtension/8623.2.7.110.1 Network/5569.82.5 iOS/26.3.1 Disallow: / User-agent: Sogou web spider Disallow: / User-agent: TerraNews Crawler/1.0 Disallow: / User-agent: jptt/3.1 Disallow: / User-agent: Perplexity-User Disallow: / User-agent: DuckAssistBot Disallow: / User-agent: Google-NotebookLM Disallow: / User-agent: LinkupBot Disallow: / User-agent: Aranet-SearchBot Disallow: / User-agent: Intelligensia Disallow: / User-agent: YeowBot Disallow: / User-agent: AIWebIndex Disallow: / User-agent: OmniCorpusPrep Disallow: / User-agent: OmniCorpus-Preprocessor Disallow: / User-agent: QuillBot Disallow: / User-agent: MarkdownWorker Disallow: / User-agent: Manus-User Disallow: / User-agent: ShapBot Disallow: / User-agent: Shap-User Disallow: / User-agent: SleepBot Disallow: / User-agent: ABEvalBot Disallow: / User-agent: ClueWeb-Crawler Disallow: / User-agent: OWLer Disallow: / User-agent: SiteAuditBot Disallow: / User-agent: SERankingBacklinksBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: DotBot Disallow: / User-agent: DataForSeoBot Disallow: / User-agent: Paqlebot Disallow: / User-agent: Barkrowler Disallow: / User-agent: BitSightBot Disallow: / User-agent: VelenPublicWebCrawler Disallow: / User-agent: SemanticVisionsBot Disallow: / User-agent: AffsignalCrawler Disallow: / User-agent: AudigentAdBot Disallow: / User-agent: Mediatoolkitbot Disallow: / User-agent: Moreover Disallow: / User-agent: trendictionbot Disallow: / User-agent: Buck Disallow: / User-agent: Manzama Disallow: / User-agent: MuckRack Disallow: / User-agent: PressEngineBot Disallow: / User-agent: NewsCred-Aggregator Disallow: / User-agent: ClientNewsroom Disallow: / User-agent: HN-Hundred Intelligence Collector Disallow: / User-agent: TicketingIntelBot Disallow: / User-agent: Dragonfly Disallow: / User-agent: AwarioBot Disallow: / User-agent: SocialStatusBot Disallow: / User-agent: SocialChamp Disallow: / User-agent: Kennect2Read Disallow: / User-agent: NewsSearch Disallow: / User-agent: Games Industry Monitoring Disallow: / User-agent: Baiduspider Disallow: / User-agent: YisouSpider Disallow: / User-agent: 360Spider Disallow: / User-agent: Yeti Disallow: / User-agent: Scrapy Disallow: / User-agent: Googlebot Allow: / User-agent: Bingbot Allow: / User-agent: YandexBot Allow: / User-agent: Claude-SearchBot Disallow: / User-agent: GrokBot Disallow: / User-agent: ExaSearchBot Disallow: / User-agent: Amzn-SearchBot Disallow: / User-agent: Google-Agent Disallow: / User-agent: AI2Bot Disallow: /
Changes recorded since June 2026.
No change recorded.