CrawlerPulse
CrawlerPulse / Europe / Germany / bw24.de

bw24

bw24.de · Germany · Ippen Digital GmbH & Co. KG · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Blocked
OpenAI
Blocked
Anthropic
Blocked
Perplexity
Allowed
Meta
Blocked
Common Crawl
Blocked
ByteDance
Blocked
Apple
Blocked
Amazon
Blocked

robots.txt

As read on 8 Oct 2026, 1,540 bytes. 1 rule group name AI crawlers; shown first.

user-agent: Googlebot
user-agent: Googlebot-Image
user-agent: Googlebot-Video
user-agent: Googlebot-News
user-agent: Storebot-Google
user-agent: Google-InspectionTool
user-agent: Google-CloudVertexBot
user-agent: APIs-Google
user-agent: AdsBot-Google-Mobile
user-agent: AdsBot-Google
user-agent: Mediapartners-Google
User-Agent: iTMS
user-agent: bingbot
user-agent: AdIdxBot
user-agent: BingPreview
user-agent: DuckDuckBot
user-agent: Qwantify
user-agent: facebookexternalhit
user-agent: Twitterbot
user-agent: LinkedInBot
user-agent: Pinterestbot
user-agent: Slackbot
user-agent: TelegramBot
user-agent: WhatsApp
user-agent: Discordbot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: AmazonAdBot
User-agent: trendictionbot
Allow: /
Disallow: /sub/landing
Disallow: /sub/paywall/js/
Disallow: /lightweight-ajax
Disallow: /suche/
Disallow: /test/
Disallow: /fdn/bootstrap/
Disallow: /bi/bootstrap/
Disallow: /bi/doop/
Disallow: /bi/dev/
Disallow: /sso/
Disallow: /west/assets/common/js/fallback.js

user-agent: *
Allow: /ueber-uns/
Allow: /advertorials/
Disallow: /
Sitemap: https://www.bw24.de/news.xml

History

Changes recorded since June 2026.

No change recorded.

In context

Germany
71.7% block at least one AI crawler
Behind Cloudflare
No
Same group