CrawlerPulse
CrawlerPulse / North America / United States / investors.com

Investor's Business Daily

investors.com · United States · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Blocked
OpenAI
Allowed
Anthropic
Blocked
Perplexity
Blocked
Meta
Blocked
Common Crawl
Blocked
ByteDance
Blocked
Apple
Allowed
Amazon
Blocked

robots.txt

As read on 8 Oct 2026, 2,509 bytes. 1 rule group name AI crawlers; shown first.

User-agent: googlebot
User-agent: googlebot-image
User-agent: GoogleOther
User-agent: Googlebot-Video
User-agent: Google-InspectionTool
User-agent: AdsBot-Google
User-agent: AdsBot-Google-Mobile
User-agent: AdsBot-Google-Mobile-Apps
User-agent: Storebot-Google
User-agent: google-read-aloud
User-agent: bingbot
User-agent: msnbot
User-agent: bingpreview
User-agent: slurp
User-agent: yahoo
User-agent: baiduspider
User-agent: Pinterestbot
User-agent: Yeti
User-agent: MojeekBot
User-agent: 360Spider
User-agent: google-cloudvertexbot
User-agent: duckduckbot
User-agent: Applebot
User-agent: flipboard
User-agent: qwantbot
User-agent: SeznamBot
User-agent: proximic
User-agent: admantx
User-agent: thetradedesk
User-agent: outbrain
User-agent: ias_crawler
User-agent: AmazonAdBot
User-agent: pubmatic
User-agent: smartologybot
User-agent: parselybot
User-agent: BomboraBot
User-agent: IllumaBot/1.0
User-agent: Publication-Access-for-Facebook
User-agent: Slackbot
User-agent: Screaming Frog SEO Spider
User-agent: AhrefsBot
User-agent: SemrushBot
User-agent: SimilarWebBot
User-agent: SISTRIX
User-agent: botify
User-agent: Chrome-Lighthouse
User-agent: ChatGPT-User
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: facebookexternalhit
User-agent: facebot
User-agent: twitterbot
User-agent: linkedinbot
User-agent: snapchat
User-agent: sentry
User-agent: Iframely
User-agent: Vocabtracker
User-agent: EpvzCrawl6194680250
User-agent: Citoid
User-agent: ZoteroTranslationServer
User-agent: CloudFlare-AlwaysOnline
User-agent: Cloudflare-Healthchecks
User-agent: Cloudflare-Traffic-Manager
User-agent: CloudFlare-Prefetch
User-agent: Cloudflare-SSLDetector
User-agent: Cloudflare-diagnostics
Allow: /
Disallow: /Article.aspx
Disallow: /offer/splash.aspx?id=subOffers
Disallow: /offer/splash.aspx?id=subGifts
Disallow: /fadiscount
Disallow: /StockNews
Disallow: /Search
Disallow: /SiteAds
Disallow: /et/dfs
Disallow: /photopopup.aspx*
Disallow: /articleredirect.aspx*
Disallow: /print/
Disallow: *-branch/
Disallow: *-branch-1/
Disallow: *-branch-2/
Disallow: *-branch-3/
Disallow: *-branch-4/
Disallow: *-branch-5/
Disallow: *-branch-6/
Disallow: *-branch-7/
Disallow: *-branch-8/
Disallow: *-branch-9/
Sitemap: https://www.investors.com/sitemaps/web/ibd/en/sitemap_ibd_en_index.xml
Sitemap: https://www.investors.com/news-sitemap.xml

# robots.txt

User-agent: *
Disallow: /

History

Changes recorded since June 2026.

No change recorded.

In context

United States
63.2% block at least one AI crawler
Behind Cloudflare
Yes