CrawlerPulse
CrawlerPulse / North America / United States / telegraphherald.com

Telegraph Herald

telegraphherald.com · United States · Woodward Communications, Inc. · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Blocked
OpenAI
Blocked
Anthropic
Blocked
Perplexity
Blocked
Meta
Blocked
Common Crawl
Blocked
ByteDance
Blocked
Apple
Blocked
Amazon
Allowed

robots.txt

As read on 8 Oct 2026, 4,935 bytes. 21 rule groups name AI crawlers; shown first.

User-agent: Applebot-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: Claude-Web
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Diffbot
Disallow: /

User-agent: FacebookBot
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: ImagesiftBot
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: PetalBot
Disallow: /

User-agent: Timpibot
Disallow: /

User-agent: YouBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: cohere-ai
Disallow: /

User-agent: omgili
Disallow: /

User-agent: omgilibot
Disallow: /

User-agent: AI2Bot
Disallow: /

User-agent: Ai2Bot-Dolma
Disallow: /

User-agent: FriendlyCrawler
Disallow: /

User-agent: ICC-Crawler
Disallow: /

User-agent: Scrapy
Disallow: /

User-agent: VelenPublicWebCrawler
Disallow: /

User-agent: Webzio-Extended
Disallow: /

User-agent: iaskspider/2.0
Disallow: /

User-agent: img2dataset
Disallow: /

User-agent: *
Disallow: /tncms/tracking/
Disallow: /_services/
Disallow: /eedition_demo/
Disallow: /fix/
Disallow: /test/nxt/
Disallow: /test/testform/
Disallow: /tncms/admin/
Disallow: /tncms/dmp/
Disallow: /tncms/openid2/
Disallow: /tncms/counter/
Disallow: /tncms/gtm/
Disallow: /tncms/openweb/*
Disallow: /tncms/track/
Disallow: /tncms/webservice/
Disallow: /tncms/search/
Disallow: /tncms/csrf/
Disallow: /tncms/weather/
Disallow: /tncms/messaging/
Disallow: /tncms/media/
Disallow: /tncms/auth/
Disallow: /tncms/tax/
Disallow: /tncms/block/
Disallow: /content/tncms/assets/v3/form/
Disallow: /content/tncms/form/
Disallow: /content/tncms/
Disallow: /tncms/user/
Disallow: /users/admin/
Disallow: /users/*/?
Disallow: /marketplace/*action=srch
Disallow: /tncms/calendar/
Disallow: /biztimes/events/search/
Disallow: /calendar/search/
Disallow: /calendar/adult_events/search/
Disallow: /calendar/bingo/search/
Disallow: /calendar/biztimes/search/
Disallow: /calendar/blood_drive/search/
Disallow: /calendar/clubs/search/
Disallow: /calendar/family_events/search/
Disallow: /calendar/fundraisers/search/
Disallow: /calendar/govt_meetings/search/
Disallow: /calendar/health_awareness/search/
Disallow: /calendar/kids_events/search/
Disallow: /calendar/music/search/
Disallow: /calendar/nightlife/search/
Disallow: /calendar/senior_activities/search/
Disallow: /calendar/singles_events/search/
Disallow: /calendar/support_groups/search/
Disallow: /calendar/travel_and_tourism/search/
Disallow: /calendar/tri-college_calendar_-_arts_and_culture/search/
Disallow: /calendar/tri-college_calendar_-_athletics_and_recreation/search/
Disallow: /calendar/tri-college_calendar_-_scholarship-lectures/search/
Disallow: /calendar/tri-college_calendar_-_worship/search/
Disallow: /festivals/search/
Disallow: /tncms/disqus/
Disallow: /classifieds/*?
Disallow: /classifieds/community/*?
Disallow: /classifieds/housing/*?
Disallow: /classifieds/job/*?
Disallow: /classifieds/legals/*?
Disallow: /classifieds/personals/*?
Disallow: /classifieds/sale/*?
Disallow: /classifieds/service/*?
Disallow: /classifieds/vehicle/*?
Sitemap: https://www.telegraphherald.com/sitemap.xml
Sitemap: https://www.telegraphherald.com/tncms/sitemap/editorial.xml
Sitemap: https://www.telegraphherald.com/tncms/sitemap/editorial.xml?year=1999
Sitemap: https://www.telegraphherald.com/tncms/sitemap/editorial.xml?year=2009
Sitemap: https://www.telegraphherald.com/tncms/sitemap/editorial.xml?year=2010
Sitemap: https://www.telegraphherald.com/tncms/sitemap/editorial.xml?year=2011
Sitemap: https://www.telegraphherald.com/tncms/sitemap/editorial.xml?year=2012
Sitemap: https://www.telegraphherald.com/tncms/sitemap/editorial.xml?year=2013
Sitemap: https://www.telegraphherald.com/tncms/sitemap/editorial.xml?year=2014
Sitemap: https://www.telegraphherald.com/tncms/sitemap/editorial.xml?year=2015
Sitemap: https://www.telegraphherald.com/tncms/sitemap/editorial.xml?year=2016
Sitemap: https://www.telegraphherald.com/tncms/sitemap/editorial.xml?year=2017
Sitemap: https://www.telegraphherald.com/tncms/sitemap/editorial.xml?year=2018
Sitemap: https://www.telegraphherald.com/tncms/sitemap/editorial.xml?year=2019
Sitemap: https://www.telegraphherald.com/tncms/sitemap/editorial.xml?year=2020
Sitemap: https://www.telegraphherald.com/tncms/sitemap/editorial.xml?year=2021
Sitemap: https://www.telegraphherald.com/tncms/sitemap/editorial.xml?year=2022
Sitemap: https://www.telegraphherald.com/tncms/sitemap/editorial.xml?year=2023
Sitemap: https://www.telegraphherald.com/tncms/sitemap/editorial.xml?year=2024
Sitemap: https://www.telegraphherald.com/tncms/sitemap/editorial.xml?year=2025

History

Changes recorded since June 2026.

No change recorded.

In context

United States
63.2% block at least one AI crawler
Behind Cloudflare
No