news24.com · South Africa · Media24 · measured 8 Oct 2026
As read on 8 Oct 2026, 2,701 bytes. 19 rule groups name AI crawlers; shown first.
User-agent: chatgpt-user Disallow: / User-agent: bytespider Disallow: / User-agent: ccbot Disallow: / User-agent: diffbot Disallow: / User-agent: facebookbot Disallow: / User-agent: google-extended Disallow: / User-agent: gptbot Disallow: / User-agent: omgili Disallow: / User-agent: Meta-ExternalAgent Disallow: / User-agent: amazonbot Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: perplexitybot Disallow: / User-agent: youbot Disallow: / User-agent: OAI-SearchBot Disallow: / User-agent: anthropic-ai Disallow: / User-agent: claude-web Disallow: / User-agent: claudebot Disallow: / User-agent: cohere-ai Disallow: / User-agent: Bytespider Disallow: / Sitemap: https://www.news24.com/sitemap user-agent: * Allow: / Allow: */cricket/test/ Allow: */cricketworldcup2019/Test/ Disallow: /toets/ Disallow: */toets/* Disallow: /test/ Disallow: */_test/* Disallow: */testpolar/* Disallow: */test/* Disallow: /xArchive/Archive/Illegal-liquor-export-20010319 Disallow: /.well-known/ Disallow: /assetlinks.json User-agent: Twitterbot Allow: / User-agent: ia_archiver Disallow: /BreakingNewsSms User-Agent: MauiBot Disallow: / User-agent: Claude-User Disallow: / User-agent: Perplexity-User Disallow: / User-agent: Meta-ExternalFetcher Disallow: / User-agent: DuckAssistBot Disallow: / User-agent: Meta-WebIndexer Disallow: / User-agent: news-please Disallow: / User-agent: quillbot.com Disallow: / User-agent: Quora-Bot Disallow: / User-agent: SeekrBot Disallow: / User-agent: TurnitinBot Disallow: / User-agent: Claude-SearchBot Disallow: / User-agent: 008 Disallow: / User-agent: dataprovider-com Disallow: / User-agent: dcrawl Disallow: / User-agent: httrack Disallow: / User-agent: httrack-3-0 Disallow: / User-agent: metainspector Disallow: / User-agent: newspaper Disallow: / User-agent: nutch Disallow: / User-agent: offline-explorer Disallow: / User-agent: scrapy Disallow: / User-agent: NewsNow Disallow: / User-agent: archive.org_bot Disallow: / User-agent: ahrefsbot Disallow: / User-agent: barkrowler Disallow: / User-agent: blexbot Disallow: / User-agent: dataforseobot Disallow: / User-agent: domainstatsbot Disallow: / User-agent: dotbot Disallow: / User-agent: hypestat Disallow: / User-agent: linkdexbot Disallow: / User-agent: mj12bot Disallow: / User-agent: screaming-frog-seo-spider Disallow: / User-agent: semrushbot Disallow: / User-agent: semrushbot-ba Disallow: / User-agent: semrushbot-ct Disallow: / User-agent: semrushbot-si Disallow: / User-agent: semrushbot-swa Disallow: / User-agent: serpstatbot Disallow: / User-agent: zoombot Disallow: /
Changes recorded since June 2026.
No change recorded.