CrawlerPulse
CrawlerPulse / Africa / South Africa / northwestnewspapers.co.za

northwestnewspapers.co.za

northwestnewspapers.co.za · South Africa · measured 8 Oct 2026

Google Search
Blocked
Google-Extended
Blocked
OpenAI
Blocked
Anthropic
Blocked
Perplexity
Blocked
Meta
Blocked
Common Crawl
Blocked
ByteDance
Blocked
Apple
Blocked
Amazon
Blocked

robots.txt

As read on 8 Oct 2026, 5,171 bytes. No rule names an AI crawler.

User-agent: *
Disallow: /

User-agent: *
Disallow: /britspos/administrator/
Disallow: /britspos/bin/
Disallow: /britspos/cache/
Disallow: /britspos/cli/
Disallow: /britspos/components/
Disallow: /britspos/includes/
Disallow: /britspos/installation/
Disallow: /britspos/language/
Disallow: /britspos/layouts/
Disallow: /britspos/libraries/
Disallow: /britspos/logs/
Disallow: /britspos/modules/
Disallow: /britspos/plugins/
Disallow: /britspos/tmp/

User-agent: *
Disallow: /gemsbok/administrator/
Disallow: /gemsbok/bin/
Disallow: /gemsbok/cache/
Disallow: /gemsbok/cli/
Disallow: /gemsbok/components/
Disallow: /gemsbok/includes/
Disallow: /gemsbok/installation/
Disallow: /gemsbok/language/
Disallow: /gemsbok/layouts/
Disallow: /gemsbok/libraries/
Disallow: /gemsbok/logs/
Disallow: /gemsbok/modules/
Disallow: /gemsbok/plugins/
Disallow: /gemsbok/tmp/

User-agent: *
Disallow: /herald/administrator/
Disallow: /herald/bin/
Disallow: /herald/cache/
Disallow: /herald/cli/
Disallow: /herald/components/
Disallow: /herald/includes/
Disallow: /herald/installation/
Disallow: /herald/language/
Disallow: /herald/layouts/
Disallow: /herald/libraries/
Disallow: /herald/logs/
Disallow: /herald/modules/
Disallow: /herald/plugins/
Disallow: /herald/tmp/

User-agent: *
Disallow: /heraldbonus/administrator/
Disallow: /heraldbonus/bin/
Disallow: /heraldbonus/cache/
Disallow: /heraldbonus/cli/
Disallow: /heraldbonus/components/
Disallow: /heraldbonus/includes/
Disallow: /heraldbonus/installation/
Disallow: /heraldbonus/language/
Disallow: /heraldbonus/layouts/
Disallow: /heraldbonus/libraries/
Disallow: /heraldbonus/logs/
Disallow: /heraldbonus/modules/
Disallow: /heraldbonus/plugins/
Disallow: /heraldbonus/tmp/

User-agent: *
Disallow: /klerksdorprecord/administrator/
Disallow: /klerksdorprecord/bin/
Disallow: /klerksdorprecord/cache/
Disallow: /klerksdorprecord/cli/
Disallow: /klerksdorprecord/components/
Disallow: /klerksdorprecord/includes/
Disallow: /klerksdorprecord/installation/
Disallow: /klerksdorprecord/language/
Disallow: /klerksdorprecord/layouts/
Disallow: /klerksdorprecord/libraries/
Disallow: /klerksdorprecord/logs/
Disallow: /klerksdorprecord/modules/
Disallow: /klerksdorprecord/plugins/
Disallow: /klerksdorprecord/tmp/

User-agent: *
Disallow: /lenstwe/administrator/
Disallow: /lenstwe/bin/
Disallow: /lenstwe/cache/
Disallow: /lenstwe/cli/
Disallow: /lenstwe/components/
Disallow: /lenstwe/includes/
Disallow: /lenstwe/installation/
Disallow: /lenstwe/language/
Disallow: /lenstwe/layouts/
Disallow: /lenstwe/libraries/
Disallow: /lenstwe/logs/
Disallow: /lenstwe/modules/
Disallow: /lenstwe/plugins/
Disallow: /lenstwe/tmp/

User-agent: *
Disallow: /mafikengmail/administrator/
Disallow: /mafikengmail/bin/
Disallow: /mafikengmail/cache/
Disallow: /mafikengmail/cli/
Disallow: /mafikengmail/components/
Disallow: /mafikengmail/includes/
Disallow: /mafikengmail/installation/
Disallow: /mafikengmail/language/
Disallow: /mafikengmail/layouts/
Disallow: /mafikengmail/libraries/
Disallow: /mafikengmail/logs/
Disallow: /mafikengmail/modules/
Disallow: /mafikengmail/plugins/
Disallow: /mafikengmail/tmp/

User-agent: *
Disallow: /midweek/administrator/
Disallow: /midweek/bin/
Disallow: /midweek/cache/
Disallow: /midweek/cli/
Disallow: /midweek/components/
Disallow: /midweek/includes/
Disallow: /midweek/installation/
Disallow: /midweek/language/
Disallow: /midweek/layouts/
Disallow: /midweek/libraries/
Disallow: /midweek/logs/
Disallow: /midweek/modules/
Disallow: /midweek/plugins/
Disallow: /midweek/tmp/

User-agent: *
Disallow: /noordwester/administrator/
Disallow: /noordwester/bin/
Disallow: /noordwester/cache/
Disallow: /noordwester/cli/
Disallow: /noordwester/components/
Disallow: /noordwester/includes/
Disallow: /noordwester/installation/
Disallow: /noordwester/language/
Disallow: /noordwester/layouts/
Disallow: /noordwester/libraries/
Disallow: /noordwester/logs/
Disallow: /noordwester/modules/
Disallow: /noordwester/plugins/
Disallow: /noordwester/tmp/

User-agent: *
Disallow: /stellalander/administrator/
Disallow: /stellalander/bin/
Disallow: /stellalander/cache/
Disallow: /stellalander/cli/
Disallow: /stellalander/components/
Disallow: /stellalander/includes/
Disallow: /stellalander/installation/
Disallow: /stellalander/language/
Disallow: /stellalander/layouts/
Disallow: /stellalander/libraries/
Disallow: /stellalander/logs/
Disallow: /stellalander/modules/
Disallow: /stellalander/plugins/
Disallow: /stellalander/tmp/

History

Changes recorded since June 2026.

No change recorded.

In context

South Africa
22.1% block at least one AI crawler
Behind Cloudflare
No