CrawlerPulse
CrawlerPulse / North America / United States / wsls.com

WSLS

wsls.com · United States · Graham Media Group · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Allowed
OpenAI
Blocked
Anthropic
Blocked
Perplexity
Allowed
Meta
Allowed
Common Crawl
Allowed
ByteDance
Blocked
Apple
Allowed
Amazon
Blocked

robots.txt

As read on 8 Oct 2026, 1,139 bytes. 5 rule groups name AI crawlers; shown first.

User-Agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: YisouSpider
User-agent: Amzn-SearchBot
User-agent: AhrefsBot
User-agent: SemrushBot
User-agent: SemrushBot-EO
User-agent: SemrushBot-BA
User-agent: DotBot
User-agent: Sogou web spider
User-agent: ArachnysSpider
User-agent: DataForSeoBot
User-agent: MJ12bot
User-agent: PetalBot
User-agent: SeekportBot
User-agent: Baiduspider-render
Disallow: /
Sitemap: https://www.wsls.com/sitemap.xml
Sitemap: https://community.wsls.com/sitemap.xml
Sitemap: https://www.wsls.com/arc/outboundfeeds/sitemap/?outputType=xml
Sitemap: https://www.wsls.com/arc/outboundfeeds/news-sitemap/?outputType=xml
Sitemap: https://www.wsls.com/arc/outboundfeeds/google-news-feed/?outputType=xml

User-Agent: *
Disallow: /appfeeds/*
Disallow: /meta/*
Disallow: /climate-collaborative/*
Disallow: /*outputType=rawHTML
Disallow: /*outputType=appfeeds

User-Agent: Baiduspider
Disallow: /

User-Agent: YisouSpider
Disallow: /

User-agent: Amzn-SearchBot
Disallow: /

History

Changes recorded since June 2026.

No change recorded.

In context

United States
63.2% block at least one AI crawler
Behind Cloudflare
No
Same group