CrawlerPulse
CrawlerPulse / Europe / Germany / mittelbayerische.de

Mittelbayerische Zeitung

mittelbayerische.de · Germany · Medien Digital-Bayern GmbH · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Blocked
OpenAI
Blocked
Anthropic
Blocked
Perplexity
Allowed
Meta
Blocked
Common Crawl
Blocked
ByteDance
Blocked
Apple
Blocked
Amazon
Blocked

robots.txt

As read on 8 Oct 2026, 4,002 bytes. 3 rule groups name AI crawlers; shown first.

User-agent: AhrefsBot
User-agent: backlink-check.de
User-agent: BacklinkCrawler
User-agent: DataForSeoBot
User-agent: deepcrawl
User-agent: dotbot
User-agent: ExtractorPro
User-agent: Fasterfox
User-agent: GPTBot
User-agent: LinkChecker
User-agent: LinkextractorPro
User-agent: LinkWalker
User-agent: Majestic
User-agent: Majestic12
User-agent: MJ12bot
User-agent: OnPageBot
User-agent: Openbot
User-agent: OpenindexSpider
User-agent: Optimizer
User-agent: rogerbot
User-agent: RyteBot
User-agent: searchpreview
User-agent: SemrushBot
User-agent: SemrushBot-SA
User-agent: Semrushbot-SI
User-agent: Seobility
User-agent: SEODAT
User-agent: SEOdiver
User-agent: SEOENGBot
User-agent: SEOkicks
User-agent: SEOkicks-Robot
User-agent: SiteAuditBot
User-agent: True_Robot
User-agent: URL Control
User-agent: URL_Spider_Pro
User-agent: xovi
Disallow: /

User-agent: 360Spider
User-agent: AhrefsBot
User-agent: aiHitBot
User-agent: Amazonbot
User-agent: anthropic-ai
User-agent: Applebot-Extended
User-agent: archive.org_bot
User-agent: Baiduspider
User-agent: ByteDance crawler
User-agent: Bytespider
User-agent: CCBot
User-agent: certytags
User-agent: ChatGPT-User
User-agent: Chatwork crawler
User-agent: ClaudeBot
User-agent: Claude-Web
User-agent: coccocbot
User-agent: cohere-ai
User-agent: Daumoa
User-agent: Diffbot
User-agent: DuckAssistBot
User-agent: FacebookBot
User-agent: FriendlyCrawler
User-agent: Google-Extended
User-agent: GoogleOther
User-agent: GPTBot
User-agent: GumGum Bot
User-agent: ia_archiver
User-Agent: ImagesiftBot
User-agent: img2dataset
User-Agent: meta-externalagent
User-agent: omgili
User-agent: omgilibot
User-agent: peer39_crawler
User-agent: peer39_crawler/1.0
User-agent: Perplexity-User
User-agent: um-IC
User-agent: YouBot
Disallow: /

User-agent: arquivo.pt
User-agent: arquivo-web-crawler
User-agent: Barkrowler
User-agent: BLEXBot
User-agent: browsertrix
User-agent: brozzler
User-agent: BuiltWith
User-agent: Cincraw
User-agent: coccocbot
User-agent: contao/crawler
User-agent: Dmbot
User-agent: DomainStatsBot
User-agent: DotBot
User-agent: dotbot
User-agent: fluid
User-agent: HaosouSpider
User-agent: HappyWing
User-agent: harsilbot
User-agent: hatena antenna
User-agent: Heritrix
User-agent: ImagesiftBot
User-agent: kazbtbot
User-agent: Kraken
User-agent: LCC
User-agent: linkdebot
User-agent: Linkfluence Yak Bot
User-agent: magpie-crawler
User-agent: mail.RU_Bot
User-agent: MetaJobBot
User-agent: MJ12bot
User-agent: Monsidobot
User-agent: netEstate
User-agent: OGDWCtcxCrawler
User-agent: PetalBot
User-agent: ResearchBot
User-agent: Riddler
User-agent: rogerbot
User-agent: Semanticbot
User-agent: SemanticScholarBot
User-agent: SentiBot
User-agent: SirdataBot
User-agent: spbot
User-agent: special_archiver
User-agent: SplitSignalBot
User-agent: Tag-Crawler
User-agent: Testcrawler
User-agent: thinkers-bot
User-agent: Toplistbot
User-agent: TurnitinBot
User-agent: uipbot/1.0
User-agent: urlsuma
User-agent: VsuSearchSpider
User-agent: weborama-fetcher
User-agent: WiseGuys Robot
User-agent: wpbot
User-agent: Yeti
Disallow: /
Sitemap: https://mittelbayerische.de/sitemap.xml

User-agent: Googlebot
User-agent: Googlebot-News
User-agent: Googlebot-Video
User-agent: Googlebot-Image
User-agent: Google-Sitemaps
User-agent: Google_InspectionTool
Disallow: /print/
Disallow: /app/
Disallow: /amp/

User-agent: bingbot
User-agent: Lizard
User-agent: msnbot
User-agent: msnbot-media
Crawl-delay: 5
Disallow: /print/
Disallow: /app/
Disallow: /amp/

User-agent: Baiduspider
User-agent: Cliqzbot
User-agent: Flamingo_SearchEngine
User-agent: SeznamBot
User-agent: sogou spider
User-agent: Yandex
Disallow: /

History

Changes recorded since June 2026.

No change recorded.

In context

Germany
71.7% block at least one AI crawler
Behind Cloudflare
No
Same group