CrawlerPulse
CrawlerPulse / Africa / South Africa / citizen.co.za

The Citizen

citizen.co.za · South Africa · CAXTON AND CTP PUBLISHERS AND PRINTERS LTD · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Allowed
OpenAI
Allowed
Anthropic
Allowed
Perplexity
Allowed
Meta
Allowed
Common Crawl
Allowed
ByteDance
Allowed
Apple
Allowed
Amazon
Allowed

robots.txt

As read on 8 Oct 2026, 4,632 bytes. 6 rule groups name AI crawlers; shown first.

User-agent: FacebookBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: GPTBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Amazonbot
Allow: /
Sitemap: https://www.citizen.co.za/sitemap_index.xml
Sitemap: https://www.citizen.co.za/network-news/sitemap_index.xml
Sitemap: https://www.citizen.co.za/middelburg-observer/sitemap_index.xml
Sitemap: https://www.citizen.co.za/highvelder-news/sitemap_index.xml
Sitemap: https://www.citizen.co.za/die-pos/sitemap_index.xml
Sitemap: https://www.citizen.co.za/heidelberg-nigel-heraut/sitemap_index.xml
Sitemap: https://www.citizen.co.za/kathorus-mail/sitemap_index.xml
Sitemap: https://www.citizen.co.za/sedibeng-ster/sitemap_index.xml
Sitemap: https://www.citizen.co.za/northern-natal-news/sitemap_index.xml
Sitemap: https://www.citizen.co.za/springs-advertiser/sitemap_index.xml
Sitemap: https://www.citizen.co.za/african-reporter/sitemap_index.xml
Sitemap: https://www.citizen.co.za/soweto-urban/sitemap_index.xml
Sitemap: https://www.citizen.co.za/comaro-chronicle/sitemap_index.xml
Sitemap: https://www.citizen.co.za/alex-news/sitemap_index.xml
Sitemap: https://www.citizen.co.za/southlands-sun/sitemap_index.xml
Sitemap: https://www.citizen.co.za/randburg-sun/sitemap_index.xml
Sitemap: https://www.citizen.co.za/northcliff-melville-times/sitemap_index.xml
Sitemap: https://www.citizen.co.za/vaalweekblad/sitemap_index.xml
Sitemap: https://www.citizen.co.za/letaba-herald/sitemap_index.xml
Sitemap: https://www.citizen.co.za/germiston-city-news/sitemap_index.xml
Sitemap: https://www.citizen.co.za/mpumalanga-news/sitemap_index.xml
Sitemap: https://www.citizen.co.za/fourways-review/sitemap_index.xml
Sitemap: https://www.citizen.co.za/southern-courier/sitemap_index.xml
Sitemap: https://www.citizen.co.za/randfontein-herald/sitemap_index.xml
Sitemap: https://www.citizen.co.za/south-coast-sun/sitemap_index.xml
Sitemap: https://www.citizen.co.za/bedfordview-edenvale-news/sitemap_index.xml
Sitemap: https://www.citizen.co.za/rosebank-killarney-gazette/sitemap_index.xml
Sitemap: https://www.citizen.co.za/midrand-reporter/sitemap_index.xml
Sitemap: https://www.citizen.co.za/berea-mail/sitemap_index.xml
Sitemap: https://www.citizen.co.za/benoni-city-times/sitemap_index.xml
Sitemap: https://www.citizen.co.za/alberton-record/sitemap_index.xml
Sitemap: https://www.citizen.co.za/kempton-express/sitemap_index.xml
Sitemap: https://www.citizen.co.za/sandton-chronicle/sitemap_index.xml
Sitemap: https://www.citizen.co.za/carletonville-herald/sitemap_index.xml
Sitemap: https://www.citizen.co.za/ridge-times/sitemap_index.xml
Sitemap: https://www.citizen.co.za/north-glen-news/sitemap_index.xml
Sitemap: https://www.citizen.co.za/brakpan-herald/sitemap_index.xml
Sitemap: https://www.citizen.co.za/krugersdorp-news/sitemap_index.xml
Sitemap: https://www.citizen.co.za/roodepoort-record/sitemap_index.xml
Sitemap: https://www.citizen.co.za/potchefstroom-herald/sitemap_index.xml
Sitemap: https://www.citizen.co.za/north-coast-courier/sitemap_index.xml
Sitemap: https://www.citizen.co.za/review-online/sitemap_index.xml
Sitemap: https://www.citizen.co.za/highway-mail/sitemap_index.xml
Sitemap: https://www.citizen.co.za/rekord/sitemap_index.xml
Sitemap: https://www.citizen.co.za/zululand-observer/sitemap_index.xml
Sitemap: https://www.citizen.co.za/parys-gazette/sitemap_index.xml
Sitemap: https://www.citizen.co.za/lowvelder/sitemap_index.xml
Sitemap: https://www.citizen.co.za/witbank-news/sitemap_index.xml
Sitemap: https://www.citizen.co.za/boksburg-advertiser/sitemap_index.xml
Sitemap: https://www.citizen.co.za/south-coast-herald/sitemap_index.xml

User-agent: *
Allow: /
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /old-images/
Disallow: /oovvuu/
Disallow: /wp-content/ftp/
Disallow: /wp-content/upgrade/
Disallow: /wp-content/upgrade-temp-backup/
Disallow: /wp-content/uploads/ShortpixelBackups/
Disallow: /wp-content/uploads/colorado/
Disallow: /wp-content/uploads/logs/
Disallow: /wp-content/uploads/pongrass/
Disallow: /wp-content/uploads/redux/
Disallow: /wp-content/uploads/wp-content/
Disallow: /wp-content/uploads/wp-file-manager-pro/
Disallow: /wp-content/uploads/wpallexport/
Disallow: /wp-content/uploads/wpseo-redirects/
Disallow: /?_attachment_id
Disallow: /?q=

User-agent: Googlebot
Allow: /

User-agent: Googlebot-News
Allow: /

History

Changes recorded since June 2026.

No change recorded.

In context

South Africa
22.1% block at least one AI crawler
Behind Cloudflare
Yes