CrawlerPulse
CrawlerPulse / Europe / Spain / as.com

Diario AS

as.com · Spain · PRISA MEDIA SAU · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Allowed
OpenAI
Allowed
Anthropic
Blocked
Perplexity
Blocked
Meta
Allowed
Common Crawl
Blocked
ByteDance
Blocked
Apple
Allowed
Amazon
Blocked

robots.txt

As read on 8 Oct 2026, 4,616 bytes. 8 rule groups name AI crawlers; shown first.

User-agent: PetalBot
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: DeepSeekBot
Disallow: /

User-agent: MistralAI-User
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: bedrock-agentcore-browser
Disallow: /

User-agent: Amazon-Bot
Disallow: /

User-agent: AmazonProductDiscoveryBot
Disallow: /

User-agent: Amzn-SearchBot
Disallow: /

User-agent: Amzn-User
Disallow: /

User-agent: Arquivo-web-crawler
Disallow: /

User-agent: bnf.fr_bot
Disallow: /

User-agent: Claude-Extension
Disallow: /

User-agent: Claude-User
Disallow: /

User-agent: Claude-SearchBot
Disallow: /

User-agent: Comet
Disallow: /

User-agent: CloudFlare-AlwaysOnline
Disallow: /

User-agent: Cloudflare-Healthchecks
Disallow: /

User-agent: CloudflareBrowserRendering
Disallow: /

User-agent: Cloudflare-Radar
Disallow: /

User-agent: DuckAssistBot
Disallow: /

User-agent: archive.org_bot
Disallow: /

User-agent: ia_archiver
Disallow: /

User-agent: Moreover
Disallow: /

User-agent: Adbeat
Disallow: /

User-agent: Argus
Disallow: /

User-agent: Baiduspider
Disallow: /

User-agent: Baiduspider-image
Disallow: /

User-agent: Bitdefender
Disallow: /

User-agent: Blogtrottr
Disallow: /

User-agent: BloombergBot
Disallow: /

User-agent: BoardReader
Disallow: /

User-agent: checkmy.ws
Disallow: /

User-agent: Clickagy
Disallow: /

User-agent: CookieBot
Disallow: /

User-agent: cXensebot
Disallow: /

User-agent: DataObserver
Disallow: /

User-agent: DomainStatsBot
Disallow: /

User-agent: Embedly
Disallow: /

User-agent: Feedspotbot
Disallow: /

User-agent: FlipboardProxy
Disallow: /

User-agent: Getintent
Disallow: /

User-agent: Google-Apps-Script
Disallow: /

User-agent: GoogleDrive
Disallow: /

User-agent: Grapevine6
Disallow: /

User-agent: Gumgum
Disallow: /

User-agent: HaloBot
Disallow: /

User-agent: HubSpot Crawler
Disallow: /

User-agent: IbouBot
Disallow: /

User-agent: Inout
Disallow: /

User-agent: LinerBot
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: Manus
Disallow: /

User-agent: Meltwater
Disallow: /

User-agent: Metajob
Disallow: /

User-agent: Muckrack
Disallow: /

User-agent: NetSeer
Disallow: /

User-agent: Netvibes
Disallow: /

User-agent: Onyphe
Disallow: /

User-agent: OpenClaw
Disallow: /

User-agent: Opoint
Disallow: /

User-agent: OutbrainBot
Disallow: /

User-agent: PlumAnalytics
Disallow: /

User-agent: ProSitemaps
Disallow: /

User-agent: Perplexity-User
Disallow: /

User-agent: proximic
Disallow: /

User-agent: PubMatic
Disallow: /

User-agent: Quantcastbot
Disallow: /

User-agent: SearchConcepts
Disallow: /

User-agent: Semantic-Visions
Disallow: /

User-agent: SentiBot
Disallow: /

User-agent: Sindup
Disallow: /

User-agent: Sirdata
Disallow: /

User-agent: SmartologyBot
Disallow: /

User-agent: Superfeedr
Disallow: /

User-agent: surlybot
Disallow: /

User-agent: Synaptive
Disallow: /

User-agent: TinEye
Disallow: /

User-agent: Trendiction
Disallow: /

User-agent: TurnitinBot
Disallow: /

User-agent: Ubermetrics
Disallow: /

User-agent: Vebidoo
Disallow: /

User-agent: Webcorp
Disallow: /

User-agent: Weborama
Disallow: /

User-agent: WebPageTest
Disallow: /

User-agent: YandexAccessibilityBot
Disallow: /

User-agent: YandexDirect
Disallow: /

User-agent: YandexDirectDyn
Disallow: /

User-agent: YaDirectFetcher
Disallow: /

User-agent: YandexFavicons
Disallow: /

User-agent: YandexMetrika
Disallow: /

User-agent: YandexMobileBot
Disallow: /

User-agent: YandexNews
Disallow: /

User-agent: YandexRCA
Disallow: /

User-agent: YandexRenderResourcesBot
Disallow: /

User-agent: YandexUserproxy
Disallow: /

User-agent: *
Disallow: /*core_dmena

User-agent: *
Disallow: /pruebas/
Disallow: /publicidad/
Disallow: /notificarelacionadas
Disallow: /*.swf$
Disallow: /eskupTSUpdate
Disallow: /ThreadeskupSimple
Disallow: /OuteskupSimple
Disallow: /Ineskup
Disallow: /Outeskup
Disallow: /OuteskupSimpleMustache
Disallow: /Msgeskup
Disallow: /Profileeskup
Disallow: /Threadeskup
Disallow: /buscador/
Disallow: /kioskoymas/promociones/
Disallow: /pxlctl.gif
Disallow: /pxlctl2.gif
Disallow: /vdpep/1/
Disallow: /estaticos/aviso_legal.html
Disallow: /formularios/
Disallow: /diarioas/politica_privacidad.html
Disallow: /pdf/Condiciones_Generales_Suscripciones_DIARIO_AS.pdf
Disallow: *environment=int*
Disallow: *environment=prod*
Disallow: /encuestas/resultados/
Disallow: /*.js2$

History

Changes recorded since June 2026.

No change recorded.

In context

Spain
34.1% block at least one AI crawler
Behind Cloudflare
No
Same group