CrawlerPulse
CrawlerPulse / North America / United States / lehighvalleylive.com

lehighvalleylive.com

lehighvalleylive.com · United States · Advance Local · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Blocked
OpenAI
Blocked
Anthropic
Blocked
Perplexity
Blocked
Meta
Allowed
Common Crawl
Blocked
ByteDance
Blocked
Apple
Blocked
Amazon
Blocked

robots.txt

As read on 8 Oct 2026, 2,811 bytes. 2 rule groups name AI crawlers; shown first.

User-agent: ia_archiver
User-agent: archive.org_bot
User-agent: heritrix
User-agent: My-heritrix-crawler
User-agent: Timpibot
User-agent: omgili
User-agent: Bytespider
User-agent: DuckAssistBot
User-agent: Amazonbot
User-agent: Applebot-Extended
User-agent: Google-Extended
User-agent: Google-CloudVertexBot
User-agent: GoogleOther
User-agent: GPTBot
User-agent: CCBot
User-agent: FacebookBot
User-agent: cohere-ai
User-agent: cohere-training-data-crawler
User-agent: Omgilibot
User-agent: ClaudeBot
User-agent: Diffbot
User-agent: PanguBot
User-agent: MistralAI-User
User-agent: webz.io
User-agent: DeepSeekBot
User-agent: Bravebot
User-agent: Brightbot 1.0
User-agent: factset_spyderbot
User-agent: FirecrawlAgent
User-agent: Scrapy
User-agent: Shap-User
User-agent: ShapBot
User-agent: Yisouspider
User-agent: YouBot
Disallow: /

User-agent: Amzn-SearchBot
User-agent: Amzn-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PetalBot
User-agent: Meta-ExternalFetcher
User-agent: Meta-WebIndexer
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Disallow: /
Allow: /shopping/
Allow: /shopping-deals/
Allow: /tv/
Allow: /live-entertainment/
Allow: /sports/event-tickets/
Allow: /betting/
Allow: /sports-betting/
Allow: /casinos/
Allow: /online-casinos/
Allow: /sweepstakes/
Allow: /sweepstakes-casinos/

Sitemap: https://www.lehighvalleylive.com/arc/outboundfeeds/rss-latest/?outputType=xml
Sitemap: https://www.lehighvalleylive.com/arc/outboundfeeds/sitemap-index/?outputType=xml
Sitemap: https://www.lehighvalleylive.com/arc/outboundfeeds/news-sitemap-index/?outputType=xml
Sitemap: https://www.lehighvalleylive.com/arc/outboundfeeds/sitemap-index-by-day/
Sitemap: https://www.lehighvalleylive.com/arc/outboundfeeds/evg_updates/
Sitemap: https://www.lehighvalleylive.com/arc/outboundfeeds/google-news-feed/

User-agent: *
Disallow: /search/*
Disallow: /sponsor-content/*
Disallow: /sponsor-content-n/*
Disallow: /sponsor-content-test/
Disallow: /sponsor-content-index-test/
Disallow: /sponsor-content-p-test/
Disallow: /digitalsubscription/imagedownload/
Disallow: /workspace-betting/*
Disallow: /home-beta/
Disallow: /test/
Disallow: /preview/
Disallow: /staging/
Disallow: /clients/
Disallow: /auctions/
Disallow: /cgi-bin/
Disallow: /printer/
Disallow: /*/print.html*
Disallow: /*/print.ssf*
Disallow: /landingpages/
Disallow: /test_team/
Disallow: /set-local-edition/
Disallow: /toprailapp/
Disallow: /*mt-preview-*
Disallow: /puzzles-games/
Disallow: /comics/
Disallow: /puzzle-society/
Disallow: /go-comics/
Disallow: /comics-kingdom/
Disallow: /*?*sru=
Disallow: /archives/sweepstakes/
Allow: /archives/sweepstakes/2026/*
Allow: /archives/sweepstakes/2027/*
Allow: /archives/sweepstakes/2028/*

User-agent: AmazonAdBot
Allow: /

History

Changes recorded since June 2026.

No change recorded.

In context

United States
63.2% block at least one AI crawler
Behind Cloudflare
No
Same group