CrawlerPulse
CrawlerPulse / North America / Mexico / excelsior.com.mx

Grupo Imagen

excelsior.com.mx · Mexico · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Allowed
OpenAI
Allowed
Anthropic
Allowed
Perplexity
Allowed
Meta
Allowed
Common Crawl
Allowed
ByteDance
Allowed
Apple
Allowed
Amazon
Allowed

robots.txt

As read on 8 Oct 2026, 7,796 bytes. 2 rule groups name AI crawlers; shown first.

User-agent: Amazonbot
Crawl-delay: 5

User-agent: PetalBot
Disallow: /clasificados/

User-agent: *
Disallow: /trivia/*

User-agent: Twitterbot
Disallow: *
Allow: /media/

User-agent: *
Allow: /.well-known/amphtml/apikey.pub

User-agent: *
Sitemap: https://www.excelsior.com.mx/sitemap-index.xml
Sitemap: https://www.excelsior.com.mx/sitemap-google-news.xml
Sitemap: https://www.excelsior.com.mx/sitemap-category.xml
Sitemap: https://www.excelsior.com.mx/sitemap-tag.xml
Allow: /misc/*.css$
Allow: /misc/*.css?
Allow: /misc/*.js$
Allow: /misc/*.js?
Allow: /misc/*.gif
Allow: /misc/*.jpg
Allow: /misc/*.jpeg
Allow: /misc/*.png
Allow: /modules/*.css$
Allow: /modules/*.css?
Allow: /modules/*.js$
Allow: /modules/*.js?
Allow: /modules/*.gif
Allow: /modules/*.jpg
Allow: /modules/*.jpeg
Allow: /modules/*.png
Allow: /profiles/*.css$
Allow: /profiles/*.css?
Allow: /profiles/*.js$
Allow: /profiles/*.js?
Allow: /profiles/*.gif
Allow: /profiles/*.jpg
Allow: /profiles/*.jpeg
Allow: /profiles/*.png
Allow: /themes/*.css$
Allow: /themes/*.css?
Allow: /themes/*.js$
Allow: /themes/*.js?
Allow: /themes/*.gif
Allow: /themes/*.jpg
Allow: /themes/*.jpeg
Allow: /themes/*.png
Disallow: /includes/
Disallow: /misc/
Disallow: /modules/
Disallow: /profiles/
Disallow: /scripts/
Disallow: /themes/
Disallow: /CHANGELOG.txt
Disallow: /cron.php
Disallow: /INSTALL.mysql.txt
Disallow: /INSTALL.pgsql.txt
Disallow: /INSTALL.sqlite.txt
Disallow: /install.php
Disallow: /INSTALL.txt
Disallow: /LICENSE.txt
Disallow: /MAINTAINERS.txt
Disallow: /update.php
Disallow: /UPGRADE.txt
Disallow: /xmlrpc.php
Disallow: /admin/
Disallow: /comment/reply/
Disallow: /filter/tips/
Disallow: /node/add/
Disallow: /search/
Disallow: /user/register/
Disallow: /user/password/
Disallow: /user/login/
Disallow: /user/logout/
Disallow: /vota-cumbreimagen-2024
Disallow: /?q=admin/
Disallow: /?q=comment/reply/
Disallow: /?q=filter/tips/
Disallow: /?q=node/add/
Disallow: /?q=search/
Disallow: /?q=user/password/
Disallow: /?q=user/register/
Disallow: /?q=user/login/
Disallow: /?q=user/logout/
Disallow: /buscador
Disallow: /buscador/
Disallow: /feed
Disallow: /feed/*
Disallow: /feeds/
Disallow: /api/
Disallow: /feeds/*
Disallow: /api/*
Disallow: /esi/
Disallow: /esi/*
Disallow: /esi/rubrikk/api/api.php
Disallow: /taxonomy-term-json
Disallow: /taxonomy-term-json/*
Disallow: /nacional-json
Disallow: /nacional-json/*
Disallow: /funcion-json
Disallow: /funcion-json/*
Disallow: /comunidad-json
Disallow: /comunidad-json/*
Disallow: /expresiones-json
Disallow: /expresiones-json/*
Disallow: /hacker-json
Disallow: /hacker-json/*
Disallow: /global-json
Disallow: /global-json/*
Disallow: /especial-json
Disallow: /especial-json/*
Disallow: /taxonomy-term-json-adrenalina
Disallow: /taxonomy-term-json-adrenalina/*
Disallow: /taxonomy-term-json-tv
Disallow: /taxonomy-term-json-tv/*
Disallow: /cron/feeds/dinero/index.html
Disallow: /cron/
Disallow: /cron/*
Disallow: /descuentos/sitemap.xml
Disallow: /descuentos/*
Disallow: /file
Disallow: /file/*
Disallow: /system
Disallow: /system/*
Disallow: /clasificados/*.js
Disallow: /clasificados/*.css
Disallow: /clasificados/*.jpg
Disallow: /clasificados/*?browse=
Disallow: /clasificados/*--*.html
Disallow: /clasificados/*=
Disallow: /clasificados/Click
Disallow: /clasificados/*--expensive
Disallow: /clasificados/*--fair-price
Disallow: /clasificados/*--good-price
Disallow: /clasificados/*--no-price-rating
Disallow: /clasificados/*--super-price
Disallow: /clasificados/*expensive--
Disallow: /clasificados/*fair-price--
Disallow: /clasificados/*good-price--
Disallow: /clasificados/*no-price-rating--
Disallow: /clasificados/*super-price--
Disallow: /clasificados/*--a-bit-pricy
Disallow: /clasificados/*a-bit-pricy--
Disallow: /clasificados/*itemindex*
Disallow: /clasificados/SaveAd/
Disallow: /clasificados/confirm-search/
Disallow: /clasificados/get-filter-results/
Disallow: /clasificados/get-top-box-results/
Disallow: /clasificados/login-saved-searches/
Disallow: /clasificados/manage-saved-searches/
Disallow: /clasificados/recover-password/
Disallow: /clasificados/register-saved-searches/
Disallow: /clasificados/SavedSearch/
Disallow: /clasificados/ss-logout/
Disallow: /clasificados/Subscriptions/
Disallow: /clasificados/update-frequency/
Disallow: /clasificados/update-password/
Disallow: /clasificados/Account/
Disallow: /clasificados/SingleAdPage/
Disallow: /clasificados/SingleAdPage/Home/
Disallow: /clasificados/SingleAdPage/Product/
Disallow: /clasificados/SingleAdPage/Similar/
Disallow: /clasificados/*ajax.svc*
Disallow: /clasificados/*_nif
Disallow: /clasificados/admanagement/*
Disallow: /clasificados/Controls/*
Disallow: /clasificados/data/*
Disallow: /clasificados/external-login/
Disallow: /clasificados/ExternalApi/
Disallow: /clasificados/get-results/*
Disallow: /clasificados/JsonProvider/Components
Disallow: /clasificados/mobile/search/
Disallow: /clasificados/Report/
Disallow: /clasificados/Rss/
Disallow: /clasificados/set-admin-cookie/
Disallow: /clasificados/SitemapHandler/
Disallow: /clasificados/PartnerIntegration/
Disallow: /clasificados/PopularSearches/
Disallow: /clasificados/XmlProvider/
Disallow: /clasificados/api/
Disallow: /clasificados/*.aspx
Disallow: /clasificados/browse/*
Disallow: /clasificados/Reviews/
Disallow: /clasificados/*?
Disallow: /clasificados/serp/

User-agent: AhrefsBot
Disallow: /clasificados/

User-agent: BLEXBot
Disallow: /clasificados/

User-agent: Bloglines/3.1
Disallow: /clasificados/

User-agent: cityreview
Disallow: /clasificados/

User-agent: DOC
Disallow: /clasificados/

User-agent: dotbot
Disallow: /clasificados/

User-agent: Download Ninja
Disallow: /clasificados/

User-agent: Exabot
Disallow: /clasificados/

User-agent: Fetch
Disallow: /clasificados/

User-agent: grapeshot
Disallow: /clasificados/

User-agent: grub-client
Disallow: /clasificados/

User-agent: HTTrack
Disallow: /clasificados/

User-agent: Jyxobot/1
Disallow: /clasificados/

User-agent: k2spider
Disallow: /clasificados/

User-agent: larbin
Disallow: /clasificados/

User-agent: libwww
Disallow: /clasificados/

User-agent: linko
Disallow: /clasificados/

User-agent: Microsoft.URL.Control
Disallow: /clasificados/

User-agent: MSIECrawler
Disallow: /clasificados/

User-agent: NPBot
Disallow: /clasificados/

User-agent: Offline Explorer
Disallow: /clasificados/

User-agent: proximic
Disallow: /clasificados/

User-agent: psbot
Disallow: /clasificados/

User-agent: SemrushBot
Disallow: /clasificados/

User-agent: sitecheck.internetseer.com
Disallow: /clasificados/

User-agent: SiteSnagger
Disallow: /clasificados/

User-agent: Speedy
Disallow: /clasificados/

User-agent: Teleport
Disallow: /clasificados/

User-agent: TeleportPro
Disallow: /clasificados/

User-agent: UbiCrawler
Disallow: /clasificados/

User-agent: WebCopier
Disallow: /clasificados/

User-agent: WebReaper
Disallow: /clasificados/

User-agent: WebStripper
Disallow: /clasificados/

User-agent: WebZIP
Disallow: /clasificados/

User-agent: wget
Disallow: /clasificados/

User-agent: Xenu
Disallow: /clasificados/

User-agent: Zao
Disallow: /clasificados/

User-agent: Zealbot
Disallow: /clasificados/

User-agent: ZyBORG
Disallow: /clasificados/

History

Changes recorded since June 2026.

No change recorded.

In context

Mexico
11.1% block at least one AI crawler
Behind Cloudflare
No
Same group