CrawlerPulse
CrawlerPulse / Europe / Sweden / di.se

Dagens Industri

di.se · Sweden · Bonnier SE · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Blocked
OpenAI
Blocked
Anthropic
Blocked
Perplexity
Blocked
Meta
Blocked
Common Crawl
Blocked
ByteDance
Blocked
Apple
Allowed
Amazon
Blocked

robots.txt

As read on 8 Oct 2026, 5,689 bytes. 20 rule groups name AI crawlers; shown first.

User-agent: CCBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Google-CloudVertexBot
Disallow: /

User-agent: Omgilibot
Disallow: /

User-agent: Omgili
Disallow: /

User-agent: FacebookBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Diffbot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: cohere-ai
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

User-agent: Timpibot
Disallow: /

User-agent: YouBot
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

User-agent: MistralAI-User
Disallow: /

User-agent: Claude-SearchBot
Disallow: /

User-agent: Claude-User
Disallow: /

User-agent: DuckAssistBot
Disallow: /

User-agent: Perplexity-User
Disallow: /

User-agent: cohere-training-data-crawler
Disallow: /

User-agent: Meta-ExternalFetcher
Disallow: /

User-agent: Meta-WebIndexer
Disallow: /

User-agent: Webzio-Extended
Disallow: /

User-agent: VelenPublicWebCrawler
Disallow: /

User-agent: Gemini-Deep-Research
Disallow: /

User-agent: ICC-Crawler
Disallow: /

User-agent: *
Disallow: /_alive/
Disallow: /_alive
Disallow: /login*
Disallow: /search*
Disallow: /bors/search*
Disallow: /*?tab=
Disallow: /konto/*
Disallow: /insider-bolag/
Disallow: /finansiell-information/pressreleaser-per-foretag/
Disallow: /finansiell-information/pressreleaser/?page=
Disallow: /insider-person/
Disallow: /Comments/WebServices/
Disallow: /*?fhtab=
Disallow: /*&fhtab=
Disallow: /*?allakommentarer=
Disallow: /*&allakommentarer=
Disallow: /*?timestamp=
Disallow: /*&timestamp=
Disallow: /*?t=
Disallow: /*&t=
Disallow: /*?flik=
Disallow: /*&flik=
Disallow: /*?qr=
Disallow: /*&qr=
Disallow: /*?screenwidth
Disallow: /*&screenwidth
Disallow: /*?screenheight
Disallow: /*&screenheight
Disallow: /*?ctl
Disallow: /*&ctl
Disallow: /*?currentIndex=
Disallow: /*?ns_
Disallow: /*&ns_
Disallow: /bip/
Disallow: /bip-callback*
Disallow: /register*
Disallow: /part*
Disallow: /akademi/anmalan/*
Disallow: /akademi/info-utbildningar/
Disallow: /34405621/bn/*
Disallow: /di-fonder/*
Disallow: /_akademi/
Disallow: /_akademi
Disallow: /logout
Sitemap: https://www.di.se/sitemap.xml
Sitemap: https://www.di.se/sitemap-news.xml

History

Changes recorded since June 2026.

No change recorded.

In context

Sweden
59.0% block at least one AI crawler
Behind Cloudflare
No
Same group