CrawlerPulse
CrawlerPulse / Europe / Spain / diariodemallorca.es

Diario de Mallorca

diariodemallorca.es · Spain · Prensa Ibérica · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Allowed
OpenAI
Blocked
Anthropic
Blocked
Perplexity
Blocked
Meta
Blocked
Common Crawl
Allowed
ByteDance
Allowed
Apple
Allowed
Amazon
Allowed

robots.txt

As read on 8 Oct 2026, 2,743 bytes. 9 rule groups name AI crawlers; shown first.

User-agent: GPTBot
Allow: /*-bc-
Allow: /ideas/
Disallow: /

User-agent: OAI-SearchBot
Allow: /*-bc-
Allow: /ideas/
Disallow: /

User-agent: ChatGPT-User
Allow: /*-bc-
Allow: /ideas/
Disallow: /

User-agent: PerplexityBot
Allow: /*-bc-
Allow: /ideas/
Disallow: /

User-agent: ClaudeBot
Allow: /*-bc-
Allow: /ideas/
Disallow: /

User-agent: Claude-Web
Allow: /*-bc-
Allow: /ideas/
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

User-agent: Google-Extended
Allow: /

User-agent: Google-CloudVertexBot
Allow: /

User-agent: 008
Disallow: /tic

User-agent: UnisterBot
Disallow: /

User-agent: JikeSpider
Disallow: /

User-agent: Zealbot
Disallow: /

User-agent: WebStripper
Disallow: /

User-agent: WebCopier
Disallow: /

User-agent: Fetch
Disallow: /

User-agent: Teleport
Disallow: /

User-agent: HTTrack
Disallow: /

User-agent: magpie-crawler
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: DotBot
Disallow: /

User-agent: spbot
Disallow: /

User-agent: BLEXBot
Disallow: /

User-agent: BUbiNG
Disallow: /

User-agent: Yandex
Disallow: /

User-agent: Baiduspider
Disallow: /

User-agent: Perplexity-User
Allow: /*-bc-
Allow: /ideas/
Disallow: /

User-agent: Claude-User
Allow: /*-bc-
Allow: /ideas/
Disallow: /

User-agent: Claude-SearchBot
Allow: /*-bc-
Allow: /ideas/
Disallow: /

User-agent: AmazonAdBot
Allow: /

User-agent: *
Allow: /*-bc-
Allow: /*-sh/index.html$
Allow: /*-rm/index.html$
Disallow: /tags/$
Disallow: /tags/letra/
Disallow: /medio-ambiente/
Disallow: /viajes/
Disallow: /nacional/
Disallow: /internacional/
Disallow: /economia/activos/
Disallow: /tiempo/
Disallow: /blogs/
Disallow: /blogs/pep-ribas-ribas/
Disallow: /deportes/futbol/
Disallow: /deportes/baloncesto/
Disallow: /clip/
Disallow: /deportes/ed/
Disallow: /api/
Disallow: /ocio/agenda/eventos/
Disallow: /ocio/hosteleria/hoteles/
Disallow: /ocio/planes/planes/
Disallow: /ocio/gastronomia/vinos/mejores/
Disallow: /ocio/gastronomia/cocteles/
Disallow: /ocio/hosteleria/hoteles/
Disallow: /ocio/hosteleria/restaurantes/
Disallow: /olimpiadas/disciplinas/
Disallow: /olimpiadas/calendario/
Disallow: /olimpiadas/medallero/
Disallow: /olimpiadas/resultados/
Disallow: /hemeroteca/
Disallow: /buscador/
Disallow: /*?p=
Disallow: /*CANONICAL_URL
Disallow: /fonts/
Disallow: /elecciones/municipales/28m/42ds/fonts/
Disallow: /elecciones/municipales/28m/

User-agent: Twitterbot
Allow: /

User-agent: facebookexternalhit
Allow: /
Sitemap: https://www.diariodemallorca.es/sitemapNews.xml
Sitemap: https://www.diariodemallorca.es/sitemapMonth.xml
Sitemap: https://www.diariodemallorca.es/sitemapSection.xml

History

Changes recorded since June 2026.

No change recorded.

In context