eyethunews.co.za · South Africa · measured 8 Oct 2026
As read on 8 Oct 2026, 1,802 bytes. 1 rule group name AI crawlers; shown first.
User-agent: ccbot Disallow: / User-agent: * Allow: / Disallow: /specials/ Disallow: /shop/ Disallow: /biz/ Disallow: /looklocal/ Disallow: /homes/ Disallow: /love/ Disallow: /jobs/ Disallow: /guzzle/ Disallow: /adops-test-page/ Disallow: /builders/ Disallow: /cgi-bin/ Disallow: /wp-admin/ Disallow: /trackback/ Disallow: /comments/ Disallow: */trackback/ Disallow: */comments/ Disallow: wp-login.php Disallow: wp-signup.php Disallow: /wp-json/ User-agent: Googlebot-News Allow: / Disallow: /afp/ Disallow: /lnn/ Disallow: /category/news/mandela-day/ Disallow: /motoring/ Disallow: /category/galleries/ Disallow: /galleries/ Disallow: /events/event/ Disallow: /classifieds-2/ Disallow: /directory/ Disallow: /specials/ Disallow: /shop/ Disallow: /biz/ Disallow: /looklocal/ Disallow: /homes/ Disallow: /love/ Disallow: /jobs/ Disallow: /guzzle/ Disallow: /adops-test-page/ Disallow: /builders/ User-Agent: trendictionbot Disallow: / User-agent: AhrefsBot Disallow: / User-agent: Baiduspider Disallow: / User-agent: Qwantify Disallow: / User-agent: LinksCrawler Disallow: / User-agent: DeuSu Disallow: / User-agent: oBot Disallow: / User-agent: Exabot Disallow: / User-agent: MSIE8.0 Disallow: / User-agent: DotBot Disallow: / User-agent: archive.org_bot Disallow: / User-agent: YandexBot Disallow: / User-agent: special_archiver Disallow: / User-agent: BLEXBot Disallow: / User-agent: DuckDuckGo-Favicons-Bot Disallow: / User-agent: MJ12bot Disallow: / User-agent: MegaIndex.ru Disallow: / User-agent: SemrushBot Disallow: / User-agent: netEstate Disallow: / User-agent: TurnitinBot Disallow: / User-agent: Mail.RU_Bot Disallow: / User-agent: Yandex Disallow: /
Changes recorded since June 2026.
No change recorded.