CrawlerPulse
CrawlerPulse / Africa / South Africa / georgeherald.com

George Herald

georgeherald.com · South Africa · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Allowed
OpenAI
Allowed
Anthropic
Allowed
Perplexity
Allowed
Meta
Allowed
Common Crawl
Allowed
ByteDance
Allowed
Apple
Allowed
Amazon
Allowed

robots.txt

As read on 8 Oct 2026, 1,868 bytes. 1 rule group name AI crawlers; shown first.

User-agent: Googlebot
User-agent: Googlebot-News
User-agent: Bingbot
User-agent: GPTBot
User-agent: Google-Extended
User-agent: CCBot
User-agent: Bytespider
User-agent: Amazonbot
User-agent: ClaudeBot
User-agent: FacebookBot
User-agent: PerplexityBot
User-agent: Claude-Web
Allow: /

User-agent: *
Allow: /

User-agent: trendictionbot
User-agent: Qwantify
User-agent: LinksCrawler
User-agent: DeuSu
User-agent: oBot
User-agent: Exabot
User-agent: DotBot
User-agent: archive.org_bot
User-agent: YandexBot
User-agent: special_archiver
User-agent: BLEXBot
User-agent: MJ12bot
User-agent: MegaIndex.ru
User-agent: netEstate
Disallow: /

User-agent: *
Disallow: /search
Disallow: /admin
Disallow: /login
Disallow: /Account
Disallow: /backend
- [Main Sitemap - Auto Updates](https://www.georgeherald.com/RSS/sitemapnews)
- [Google News 2026](https://www.georgeherald.com/SiteMaps/GH/GH_GoogleSiteMapNews2026.xml)
- [Google News 2025](https://www.georgeherald.com/SiteMaps/GH/GH_GoogleSiteMapNews2025.xml)
- [Google News 2024](https://www.georgeherald.com/SiteMaps/GH/GH_GoogleSiteMapNews2024.xml)
- [Google News 2023](https://www.georgeherald.com/SiteMaps/GH/GH_GoogleSiteMapNews2023.xml)
- [Google News 2022](https://www.georgeherald.com/SiteMaps/GH/GH_GoogleSiteMapNews2022.xml)
- [Google News 2021](https://www.georgeherald.com/SiteMaps/GH/GH_GoogleSiteMapNews2021.xml)
- [Google News 2020](https://www.georgeherald.com/SiteMaps/GH/GH_GoogleSiteMapNews2020.xml)
- [Google News 2019](https://www.georgeherald.com/SiteMaps/GH/GH_GoogleSiteMapNews2019.xml)

History

Changes recorded since June 2026.

No change recorded.

In context

South Africa
22.1% block at least one AI crawler
Behind Cloudflare
Yes