CrawlerPulse
CrawlerPulse / Europe / Germany / boersen-zeitung.de

boersen-zeitung.de

boersen-zeitung.de · Germany · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Blocked
OpenAI
Blocked
Anthropic
Blocked
Perplexity
Blocked
Meta
Blocked
Common Crawl
Blocked
ByteDance
Blocked
Apple
Blocked
Amazon
Blocked

robots.txt

As read on 8 Oct 2026, 1,858 bytes. 13 rule groups name AI crawlers; shown first.

User-agent: GPTBot
Allow: /specials/
Disallow: /

User-agent: ChatGPT-User
Allow: /specials/
Disallow: /

User-agent: OAI-SearchBot
Allow: /specials/
Disallow: /

User-agent: CCBot
Allow: /specials/
Disallow: /

User-agent: Google-Extended
Allow: /specials/
Disallow: /

User-agent: Applebot-Extended
Allow: /specials/
Disallow: /

User-agent: Bytespider
Allow: /specials/
Disallow: /

User-agent: PerplexityBot
Allow: /specials/
Disallow: /

User-agent: ClaudeBot
Allow: /specials/
Disallow: /

User-agent: anthropic-ai
Allow: /specials/
Disallow: /

User-agent: YouBot
Allow: /specials/
Disallow: /

User-agent: Amazonbot
Allow: /specials/
Disallow: /

User-agent: Meta-ExternalAgent
Allow: /specials/
Disallow: /

User-agent: *
Disallow: /*?preview=true
Disallow: /null
Disallow: /schreibweise
Disallow: /suche
Disallow: /msn-rss
Disallow: /startseite
Disallow: /Uncategorized
Disallow: /uncategorized
Disallow: /ticker
Disallow: /bz-update
Disallow: /unternehmen/berichte/
Disallow: /unternehmen/uebersicht/
Disallow: /unternehmen/analysen/
Disallow: /unternehmen/dividenden/
Disallow: /unternehmen/kurse/
Disallow: /*?*page-set

User-agent: MistralAI
Allow: /specials/
Disallow: /

User-agent: ExaBot
Allow: /specials/
Disallow: /

User-agent: AndiBot
Allow: /specials/
Disallow: /

User-agent: PhindBot
Allow: /specials/
Disallow: /

User-agent: KagiBot
Allow: /specials/
Disallow: /

User-agent: Meta-ExternalFetcher
Allow: /specials/
Disallow: /
Sitemap: https://www.boersen-zeitung.de/sitemap/index.xml.gz
Sitemap: https://www.boersen-zeitung.de/sitemap/news.xml.gz
Sitemap: https://www.boersen-zeitung.de/sitemap/unternehmensseiten_sitemap_index.xml.gz
Sitemap: https://feeds.purplemanager.com/3bb5104f-f2d6-4138-b889-ac5e5ca06778/alle-news-ohne-agenturmeldungen
Sitemap: https://feeds.purplemanager.com/3bb5104f-f2d6-4138-b889-ac5e5ca06778/ticker

History

Changes recorded since June 2026.

No change recorded.

In context

Germany
71.7% block at least one AI crawler
Behind Cloudflare
No