techxplore.com · Isle of Man · measured 9 Oct 2026
As read on 9 Oct 2026, 1,153 bytes. 9 rule groups name AI crawlers; shown first.
User-Agent: ClaudeBot Disallow: /news/ Disallow: /partners/ Disallow: /journals/ Disallow: /tags/ User-Agent: GPTBot Disallow: /news/ Disallow: /partners/ Disallow: /journals/ Disallow: /tags/ User-agent: FacebookBot Disallow: /news/ Disallow: /partners/ Disallow: /journals/ Disallow: /tags/ User-agent: CCbot Disallow: /news/ Disallow: /partners/ Disallow: /journals/ Disallow: /tags/ User-Agent: Bytespider Disallow: /news/ Disallow: /partners/ Disallow: /journals/ Disallow: /tags/ User-Agent: cohere-ai Disallow: /news/ Disallow: /partners/ Disallow: /journals/ Disallow: /tags/ User-agent: Google-Extended Disallow: /news/ Disallow: /tags/ User-agent: Applebot-Extended Disallow: /news/ Disallow: /tags/ User-Agent: ImagesiftBot Disallow: /news/ Sitemap: https://techxplore.com/sitemap/indx/ User-agent: * Allow: / Disallow: /search/ Disallow: /rss-feed/search/ Disallow: /rss-feed/breaking/search/ Disallow: /rss-feed/tags/ Disallow: /*/sort/ User-agent: AhrefsBot Disallow: /tags/ User-Agent: Scrapy Disallow: / User-Agent: FriendlyCrawler Disallow: / User-Agent: ChatGLM-Spider Disallow: / User-agent: DataForSeoBot Disallow: /
Changes recorded since June 2026.
No change recorded.