CrawlerPulse
CrawlerPulse / North America / Mexico / vanguardia.com.mx

Vanguardia

vanguardia.com.mx · Mexico · Grupo Editorial Coahuila, S.A. de C.V. · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Allowed
OpenAI
Allowed
Anthropic
Allowed
Perplexity
Allowed
Meta
Allowed
Common Crawl
Allowed
ByteDance
Blocked
Apple
Allowed
Amazon
Allowed

robots.txt

As read on 8 Oct 2026, 3,857 bytes. 3 rule groups name AI crawlers; shown first.

User-agent: Bytespider
Disallow: /

User-agent: PetalBot
Disallow: /

User-agent: ImagesiftBot
Disallow: /

sitemap: https://vanguardia.com.mx/sitemap.xml
sitemap: https://vanguardia.com.mx/sitemapforgoogle.xml
sitemap: https://vanguardia.com.mx/megasitemap.xml

User-agent: *
Disallow: /news-portlet/metalocator/
Disallow: /news-portlet/html/teaser-viewer-portlet/teaser_page.jsp
Disallow: /news-portlet/html/teaser-viewer-portlet/teaser_filter.jsp
Disallow: /news-portlet/filterteaser/
Disallow: /news-portlet/getfilteropts/
Disallow: /tracking-portlet/html/ranking-viewer/ranking_details.jsp
Disallow: /user-portlet/login-with/
Disallow: /user-portlet/edit-user-profile/
Disallow: /user-portlet/reset-credentials/
Disallow: /user-portlet/confirm-email/
Disallow: /user-portlet/refreshuserentitlements/
Disallow: /user-portlet/getEntitlements/
Disallow: /group/
Disallow: /user/
Disallow: /web/
Disallow: /image/
Disallow: /resultados-busqueda/
Disallow: /usuarios/perfil-usuario/
Disallow: /usuarios/login/
Disallow: /usuarios/recuperar-credenciales/
Allow: /binrepository/
Allow: /.well-known/
Allow: /ads.txt

User-agent: AhrefsBot
Disallow: /

User-agent: SemrushBot
Disallow: /

User-agent: SemrushBot-BA
Disallow: /

User-agent: SemrushBot-EO
Disallow: /

User-agent: SERankingBacklinksBot
Disallow: /

User-agent: serpstatbot
Disallow: /

User-agent: DataForSeoBot
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: dotbot
Disallow: /

User-agent: BLEXBot
Disallow: /

User-agent: barkrowler
Disallow: /

User-agent: SleepBot
Disallow: /

User-agent: ImageBot
Disallow: /

User-agent: ExaSearchBot
Disallow: /

User-agent: Baiduspider
Disallow: /

User-agent: Baiduspider-render
Disallow: /

User-agent: Baiduspider-image
Disallow: /

User-agent: Sogou web spider
Disallow: /

User-agent: SeznamBot
Disallow: /

User-agent: coccocbot-web
Disallow: /

User-agent: rednote-websearch-bot
Disallow: /

User-agent: Seekport
Disallow: /

User-agent: AwarioSmartBot
Disallow: /

User-agent: UnzoiNewsBot
Disallow: /

User-agent: Leikibot
Disallow: /

User-agent: ZoominfoBot
Disallow: /

User-agent: MediatoolkitBot
Disallow: /

User-agent: LivelapBot
Disallow: /

User-agent: TrendictionBot
Disallow: /

User-agent: meta-webindexer
Disallow: /

History

Changes recorded since June 2026.

No change recorded.

In context

Mexico
11.1% block at least one AI crawler
Behind Cloudflare
Yes