CrawlerPulse
CrawlerPulse / North America / Mexico / reporteindigo.com

Reporte Indigo

reporteindigo.com · Mexico · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Blocked
OpenAI
Blocked
Anthropic
Blocked
Perplexity
Allowed
Meta
Blocked
Common Crawl
Blocked
ByteDance
Blocked
Apple
Allowed
Amazon
Blocked

robots.txt

As read on 8 Oct 2026, 2,111 bytes. 1 rule group name AI crawlers; shown first.

User-agent: Applebot
User-agent: baiduspider
User-agent: Bingbot
User-agent: Discordbot
User-agent: facebookexternalhit
User-agent: FacebookBot
User-agent: Googlebot
User-agent: Googlebot-News
User-agent: Googlebot-Mobile
User-agent: Mediapartners-Google
User-agent: Adsbot-Google
User-agent: AdsBot-Google-Mobile
User-agent: AdsBot-Google-Mobile-Apps
User-agent: Googlebot-Image
User-agent: Googlebot-Video
User-agent: Storebot-Google
User-agent: APIs-Google
User-agent: LinkedInBot
User-agent: msnbot
User-agent: Naverbot
User-agent: Pinterestbot
User-agent: PetalBot
User-agent: seznambot
User-agent: Slurp
User-agent: teoma
User-agent: TelegramBot
User-agent: Twitterbot
User-agent: Yeti
User-agent: Exabot
User-agent: CriteoBot/0.1
User-agent: DuckDuckBot
User-agent: AmazonAdBot
User-agent: AdIdxBot
User-agent: BingPreview
User-agent: MicrosoftPreview
User-agent: GoogleOther
User-agent: GoogleOther-Image
User-agent: GoogleOther-Video
User-agent: Google-InspectionTool
User-agent: OAI-Searchbot
User-agent: PerplexityBot
User-agent: Claude-SearchBot
User-agent: Meta-WebIndexer
User-agent: Meta-ExternalAds
User-agent: ias_crawler
User-agent: ias_wombles
User-agent: Slackbot-LinkExpanding
User-agent: Slackbot
User-agent: Slack-ImgProxy
User-agent: BraveBot
Disallow: /_req/
Disallow: /*contenidos/
Disallow: /vipcontent/
Disallow: /tema/
Disallow: /video/
Disallow: /columna/
Disallow: /author/
Disallow: /page/
Disallow: */amp/$
Disallow: /amp/*/*-201[2-9][0-9]*.html$
Disallow: */page/*[1-9][0-9]/$
Disallow: */page/*/page/*
Disallow: /*?noamp=mobile
Disallow: /*?jwsource=
Disallow: /*?amp
Disallow: /22984396690/reporte-indigo-bs/
Disallow: /buscar.html
Disallow: /templates/article/
Disallow: /piensa/
Disallow: /fan/
Disallow: /reporte/
Disallow: /indigo-geek/
Disallow: /indigonomics/
Disallow: /latitud/
Disallow: /*?fb_comment_id=*
Disallow: /edicion-impresa
Disallow: /energia-industria/
Disallow: /videos/video-youtube/
Disallow: /*-$
Disallow: /icons/filled/sprite.svg$
Disallow: /*?utm_source=*
Disallow: /*?mrfhud=true
Disallow: /*%20*
Disallow: /*-t/
Allow: /

User-agent: *
Disallow: /

History

Changes recorded since June 2026.

No change recorded.

In context

Mexico
11.1% block at least one AI crawler
Behind Cloudflare
No