maroelamedia.co.za · South Africa · measured 8 Oct 2026
As read on 8 Oct 2026, 1,964 bytes. 11 rule groups name AI crawlers; shown first.
User-agent: anthropic-ai Disallow: / User-agent: bytespider Disallow: / User-agent: ccbot Disallow: / User-agent: chatgpt-user Disallow: / User-agent: claude-web Disallow: / User-agent: claudebot Disallow: / User-agent: cohere-ai Disallow: / User-agent: diffbot Disallow: / User-agent: gptbot Disallow: / User-agent: omgili Disallow: / User-agent: petalbot Disallow: / User-Agent: * Allow: / Disallow: /readme.html Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: /wp-signup.php?* Disallow: /xmlrpc.php Disallow: /comments/feed/ Disallow: /*?utm_source= Disallow: /*?gclid= Disallow: /*?sa= Disallow: /*?origin= User-agent: ias_crawler Disallow: / User-agent: proximic Disallow: / User-agent: 008 Disallow: / User-agent: dataprovider-com Disallow: / User-agent: dcrawl Disallow: / User-agent: httrack Disallow: / User-agent: httrack-3-0 Disallow: / User-agent: metainspector Disallow: / User-agent: newspaper Disallow: / User-agent: nutch Disallow: / User-agent: offline-explorer Disallow: / User-agent: peer39_crawler Disallow: / User-agent: scrapy Disallow: / User-agent: 360Spider Disallow: / User-agent: baiduspider Disallow: / User-agent: barkrowler Disallow: / User-agent: blexBot Disallow: / User-agent: criteobot/0.1 Disallow: / User-agent: dataforseobot Disallow: / User-agent: domainstatsbot Disallow: / User-agent: dotbot Disallow: / User-agent: hypestat Disallow: / User-agent: linkdexbot Disallow: / User-agent: mj12bot Disallow: / User-agent: screaming-frog-seo-spider Disallow: / User-agent: screaming Disallow: / User-agent: semrushbot Disallow: / User-agent: siteauditbot Disallow: / User-agent: semrushbot-ba Disallow: / User-agent: semrushbot-si Disallow: / User-agent: semrushbot-swa Disallow: / User-agent: semrushbot-ct Disallow: / User-agent: splitsignalbot Disallow: / User-agent: semrushbot-coub Disallow: / User-agent: yandexbot Disallow: / User-agent: zoombot Disallow: / Sitemap: https://maroelamedia.co.za/sitemap_index.xml
Changes recorded since June 2026.
No change recorded.