dexerto.com · United Kingdom · measured 9 Oct 2026
As read on 9 Oct 2026, 654 bytes. 1 rule group name AI crawlers; shown first.
User-agent: "008" User-agent: Amazonbot User-agent: Applebot User-agent: Applebot-extended User-agent: Bytespider User-agent: CCBot User-agent: ChatGPT-User User-agent: Claude-Web User-agent: ClaudeBot User-agent: Diffbot User-agent: FacebookBot User-agent: GPTBot User-agent: HTTrack User-agent: Nutch User-agent: Offline Explorer User-agent: Scrapy User-agent: YouBot User-agent: anthropic-ai User-agent: cohere-ai User-agent: omgili Disallow: / Sitemap: https://www.dexerto.com/sitemap_index.xml Sitemap: https://www.dexerto.com/news-sitemap.xml User-Agent: * Disallow: /search Disallow: /cdn-cgi/ Allow: /cdn-cgi/image/
Changes recorded since June 2026.
No change recorded.