CrawlerPulse
CrawlerPulse / North America / United States / thestreet.com

The Street

thestreet.com · United States · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Allowed
OpenAI
Allowed
Anthropic
Allowed
Perplexity
Allowed
Meta
Allowed
Common Crawl
Allowed
ByteDance
Allowed
Apple
Allowed
Amazon
Allowed

robots.txt

As read on 8 Oct 2026, 1,060 bytes. No rule names an AI crawler.

User-Agent: *
Allow: /*?page=1$
Allow: /*?page=2$
Allow: /*?page=3$
Allow: /*?page=4$
Allow: /*?page=5$
Disallow: /.api/
Disallow: /api/
Disallow: /*?_rsc=*
Disallow: /account$
Disallow: /cgi-bin
Disallow: /pf/
Disallow: /json/
Disallow: /m/pf/
Disallow: /p/pf/
Disallow: /i/
Disallow: /b/
Disallow: /print/
Disallow: /find/
Disallow: /email/
Disallow: /press-releases/
Disallow: /partner/
Disallow: /storycomments/
Disallow: /newsanalysis/
Disallow: /_yahoo/
Disallow: /_googlepi/
Disallow: /_aol/
Disallow: /_cnnmoney/
Disallow: /_catholic/
Disallow: /p/
Disallow: /commerce/subscribeToNewsletter.action
Disallow: /static-full-na/
Disallow: /embed/
Disallow: /_tsc*
Disallow: /_rms/*
Disallow: /*/gdprStatus
Disallow: /ci0*
Disallow: /comment/
Disallow: /s/
Disallow: /funds/
Disallow: /bn/
Disallow: /_dm/
Disallow: /user/
Disallow: /*/2018-*
Disallow: /keyword/
Disallow: /signup/
Disallow: /tsc/*
Disallow: /util/
Disallow: /k/tscir/*
Disallow: /*?page=
Sitemap: https://www.thestreet.com/sitemaps.xml
Sitemap: https://www.thestreet.com/news-sitemap.xml

History

Changes recorded since June 2026.

No change recorded.

In context

United States
63.2% block at least one AI crawler
Behind Cloudflare
No