CrawlerPulse
CrawlerPulse / North America / Mexico / diariodelistmo.com

Diario Del Istmo

diariodelistmo.com · Mexico · Imagen del Golfo · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Blocked
OpenAI
Blocked
Anthropic
Blocked
Perplexity
Allowed
Meta
Blocked
Common Crawl
Blocked
ByteDance
Blocked
Apple
Allowed
Amazon
Blocked

robots.txt

As read on 8 Oct 2026, 1,278 bytes. 1 rule group name AI crawlers; shown first.

User-agent: Applebot
User-agent: baiduspider
User-agent: Bingbot
User-agent: Discordbot
User-agent: facebookexternalhit
User-agent: FacebookBot
User-agent: Googlebot
User-agent: Googlebot-News
User-agent: Googlebot-Mobile
User-agent: Mediapartners-Google
User-agent: Adsbot-Google
User-agent: AdsBot-Google-Mobile
User-agent: AdsBot-Google-Mobile-Apps
User-agent: Googlebot-Image
User-agent: Googlebot-Video
User-agent: Storebot-Google
User-agent: APIs-Google
User-agent: LinkedInBot
User-agent: msnbot
User-agent: Naverbot
User-agent: Pinterestbot
User-agent: PetalBot
User-agent: seznambot
User-agent: Slurp
User-agent: teoma
User-agent: TelegramBot
User-agent: Twitterbot
User-agent: Yeti
User-agent: Exabot
User-agent: CriteoBot/0.1
User-agent: DuckDuckBot
User-agent: AmazonAdBot
User-agent: AdIdxBot
User-agent: BingPreview
User-agent: MicrosoftPreview
User-agent: GoogleOther
User-agent: GoogleOther-Image
User-agent: GoogleOther-Video
User-agent: Google-InspectionTool
User-agent: OAI-Searchbot
User-agent: PerplexityBot
User-agent: Claude-SearchBot
User-agent: Meta-WebIndexer
User-agent: Meta-ExternalAds
User-agent: BraveBot
Disallow: /_req/
Disallow: /*contenidos/
Disallow: /vipcontent/
Disallow: /23309856066/
Disallow: /buscar
Allow: /

User-agent: *
Disallow: /

History

Changes recorded since June 2026.

No change recorded.

In context

Mexico
11.1% block at least one AI crawler
Behind Cloudflare
No
Same group