tagesschau.de · Germany · measured 8 Oct 2026
As read on 8 Oct 2026, 2,572 bytes. 12 rule groups name AI crawlers; shown first.
User-agent: applebot-extended Disallow: / User-agent: bytespider Disallow: / User-agent: ccbot Disallow: / User-agent: claudebot Disallow: / User-agent: google-extended Disallow: / User-agent: deepseekbot Disallow: / User-agent: diffbot Disallow: / User-agent: facebookbot Disallow: / User-agent: meta-externalagent Disallow: / User-agent: gptbot Disallow: / User-agent: omgili Disallow: / User-agent: omgilibot Disallow: / User-agent: * Disallow: /ardimport/ Disallow: /*~player.html Disallow: /suche2.html?* User-agent: webzip Disallow: / User-agent: AhrefsBot Disallow: / User-agent: webwhacker Disallow: / User-agent: websauger Disallow: / User-agent: webcapture Disallow: / User-agent: teleport Disallow: / User-agent: sitesnagger Disallow: / User-agent: net attache Disallow: / User-agent: httrack Disallow: / User-agent: cohere-training-data-crawler Disallow: / User-agent: deepseek Disallow: / User-agent: deepseek-crawler Disallow: / User-agent: grokbot Disallow: / User-agent: img2dataset Disallow: / User-agent: laiondownloader Disallow: / User-agent: mistralai-crawler Disallow: / User-agent: pangubot Disallow: /
Changes recorded since June 2026.
No change recorded.