snl24.com · South Africa · Media24 · measured 8 Oct 2026
As read on 8 Oct 2026, 2,341 bytes. 16 rule groups name AI crawlers; shown first.
User-agent: chatgpt-user Disallow: / User-agent: bytespider Disallow: / User-agent: ccbot Disallow: / User-agent: diffbot Disallow: / User-agent: facebookbot Disallow: / User-agent: google-extended Disallow: / User-agent: gptbot Disallow: / User-agent: omgili Disallow: / User-agent: amazonbot Disallow: / User-agent: perplexitybot Disallow: / User-agent: youbot Disallow: / User-agent: anthropic-ai Disallow: / User-agent: claude-web Disallow: / User-agent: claudebot Disallow: / User-agent: cohere-ai Disallow: / User-agent: Bytespider Disallow: / Sitemap: https://www.snl24.com/arc/outboundfeeds/sitemap-index/?outputType=xml Sitemap: https://www.snl24.com/arc/outboundfeeds/sitemap-news-index/?outputType=xml Sitemap: https://www.snl24.com/arc/outboundfeeds/sitemap-section-index/?outputType=xml user-agent: * Allow: / Allow: */cricket/test/ Allow: */cricketworldcup2019/Test/ Disallow: /toets/ Disallow: */toets/* Disallow: /test/ Disallow: */_test/* Disallow: */testpolar/* Disallow: */test/* Disallow: /xArchive/Archive/Illegal-liquor-export-20010319 Disallow: /.well-known/ Disallow: /assetlinks.json User-agent: Twitterbot Allow: / User-agent: ia_archiver Disallow: /BreakingNewsSms User-Agent: MauiBot Disallow: / User-agent: applebot Disallow: / User-agent: 008 Disallow: / User-agent: dataprovider-com Disallow: / User-agent: dcrawl Disallow: / User-agent: httrack Disallow: / User-agent: httrack-3-0 Disallow: / User-agent: metainspector Disallow: / User-agent: newspaper Disallow: / User-agent: nutch Disallow: / User-agent: offline-explorer Disallow: / User-agent: scrapy Disallow: / User-agent: ahrefsbot Disallow: / User-agent: barkrowler Disallow: / User-agent: blexbot Disallow: / User-agent: dataforseobot Disallow: / User-agent: domainstatsbot Disallow: / User-agent: dotbot Disallow: / User-agent: hypestat Disallow: / User-agent: linkdexbot Disallow: / User-agent: mj12bot Disallow: / User-agent: screaming-frog-seo-spider Disallow: / User-agent: semrushbot Disallow: / User-agent: semrushbot-ba Disallow: / User-agent: semrushbot-ct Disallow: / User-agent: semrushbot-si Disallow: / User-agent: semrushbot-swa Disallow: / User-agent: serpstatbot Disallow: / User-agent: zoombot Disallow: /
Changes recorded since June 2026.
No change recorded.