ard.de · Germany · measured 8 Oct 2026
As read on 8 Oct 2026, 2,182 bytes. 12 rule groups name AI crawlers; shown first.
User-agent: applebot-extended Disallow: / User-agent: bytespider Disallow: / User-agent: ccbot Disallow: / User-agent: claudebot Disallow: / User-agent: google-extended Disallow: / User-agent: deepseekbot Disallow: / User-agent: diffbot Disallow: / User-agent: facebookbot Disallow: / User-agent: meta-externalagent Disallow: / User-agent: gptbot Disallow: / User-agent: omgili Disallow: / User-agent: omgilibot Disallow: / User-agent: * Allow: / User-agent: cohere-training-data-crawler Disallow: / User-agent: deepseek Disallow: / User-agent: deepseek-crawler Disallow: / User-agent: grokbot Disallow: / User-agent: img2dataset Disallow: / User-agent: laiondownloader Disallow: / User-agent: mistralai-crawler Disallow: / User-agent: pangubot Disallow: /
Changes recorded since June 2026.
No change recorded.