CrawlerPulse
CrawlerPulse / North America / Mexico / debate.com.mx

El Debate

debate.com.mx · Mexico · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Allowed
OpenAI
Allowed
Anthropic
Allowed
Perplexity
Allowed
Meta
Allowed
Common Crawl
Blocked
ByteDance
Blocked
Apple
Allowed
Amazon
Blocked

robots.txt

As read on 8 Oct 2026, 1,926 bytes. 2 rule groups name AI crawlers; shown first.

User-agent: Applebot
User-agent: baiduspider
User-agent: Bingbot
User-agent: Discordbot
User-agent: facebookexternalhit
User-agent: FacebookBot
User-agent: Googlebot
User-agent: Googlebot-News
User-agent: Googlebot-Mobile
User-agent: Mediapartners-Google
User-agent: Adsbot-Google
User-agent: AdsBot-Google-Mobile
User-agent: AdsBot-Google-Mobile-Apps
User-agent: Googlebot-Image
User-agent: Googlebot-Video
User-agent: Storebot-Google
User-agent: APIs-Google
User-agent: LinkedInBot
User-agent: msnbot
User-agent: Naverbot
User-agent: Pinterestbot
User-agent: PetalBot
User-agent: seznambot
User-agent: Slurp
User-agent: teoma
User-agent: TelegramBot
User-agent: Twitterbot
User-agent: Yeti
User-agent: Exabot
User-agent: CriteoBot/0.1
User-agent: DuckDuckBot
User-agent: AmazonAdBot
User-agent: AdIdxBot
User-agent: BingPreview
User-agent: MicrosoftPreview
User-agent: GoogleOther
User-agent: GoogleOther-Image
User-agent: GoogleOther-Video
User-agent: Google-InspectionTool
User-agent: OAI-Searchbot
User-agent: PerplexityBot
User-agent: Claude-SearchBot
User-agent: Meta-WebIndexer
User-agent: Meta-ExternalAds
User-agent: grapeshot
User-agent: ChatGPT-User
User-agent: BraveBot
Disallow: /_req/
Disallow: /*contenidos/
Disallow: /vipcontent/
Disallow: /america/
Disallow: /usa/
Disallow: /buscar.html
Disallow: /*buscar
Disallow: /*search
Disallow: /auth/
Disallow: /account/
Allow: /

User-agent: anthropic-ai
User-agent: Applebot-Extended
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: DuckAssistBot
User-agent: Google-Extended
User-agent: Meta-ExternalAgent
Disallow: /_req/
Disallow: /*contenidos/
Disallow: /vipcontent/
Disallow: /22655596684/
Disallow: /noticias/*/
Disallow: /entretenimiento/*/
Disallow: /buscar.html
Disallow: /*buscar
Disallow: /*search
Disallow: /sinaloa/
Disallow: /america/
Disallow: /usa/
Disallow: /auth/
Disallow: /account/
Allow: /

User-agent: *
Disallow: /

History

Changes recorded since June 2026.

No change recorded.

In context

Mexico
11.1% block at least one AI crawler
Behind Cloudflare
No
Same group