CrawlerPulse
CrawlerPulse / North America / Mexico / elnoticieroenlinea.com

EL NOTICIERO

elnoticieroenlinea.com · Mexico · EL NOTICIERO | Periodismo comprometido con la Sociedad · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Allowed
OpenAI
Allowed
Anthropic
Allowed
Perplexity
Allowed
Meta
Allowed
Common Crawl
Blocked
ByteDance
Allowed
Apple
Allowed
Amazon
Allowed

robots.txt

As read on 8 Oct 2026, 4,009 bytes. 3 rule groups name AI crawlers; shown first.

User-agent: ShowyouBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: omgili
Disallow: /

User-agent: *
Disallow: /node/
Disallow: /cdb/
Disallow: /wp-content/
Disallow: /sites/
Disallow: /Topicos/
Disallow: /7198/
Disallow: /noticias/
Disallow: /bbtstats/
Disallow: /bbtfile/
Disallow: /feed/
Disallow: /rss7/
Disallow: /rss10/
Disallow: /MediaCenter/
Disallow: /portal/
Disallow: /infografia/*
Disallow: /5644/*
Disallow: /wp-admin
Allow: /wp-admin/admin-ajax.php
Sitemap: https://www.elnoticieroenlinea.com/sitemap/sitemap-articles-index.xml
Sitemap: https://www.elnoticieroenlinea.com/sitemap/sitemap-google-news-index.xml
Sitemap: https://www.elnoticieroenlinea.com/sitemap/sitemap-tags-index.xml
Sitemap: https://www.elnoticieroenlinea.com/sitemap/sitemap-images-index.xml
Sitemap: https://www.elnoticieroenlinea.com/sitemap/sitemap-videos-index.xml

User-agent: Genio
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: ScooperBot
Disallow: /

User-agent: SeekportBot
Disallow: /

User-agent: rogerbot
Disallow: /

User-agent: Flamingo_SearchEngine
Disallow: /

User-agent: Facebot
Disallow: /

User-agent: LuminateBot
Disallow: /

User-agent: Vagabondo
Disallow: /

User-agent: AhrefsBot
Disallow: /

User-agent: SeznamBot
Disallow: /

User-agent: R6_CommentReader
Disallow: /

User-agent: Yeti
Disallow: /

User-agent: heritrix
Disallow: /

User-agent: Baiduspider
Disallow: /

User-agent: GozaikBot
Disallow: /

User-agent: python-requests
Disallow: /

User-agent: QuerySeekerSpider
Disallow: /

User-agent: DotBot
Disallow: /

User-agent: YandexImages
Disallow: /

User-agent: Apache-HttpClient
Disallow: /

User-agent: PiplBot
Disallow: /

User-agent: Scrapy
Disallow: /

User-agent: Buck
Disallow: /

User-agent: WikiDo
Disallow: /

User-agent: ZoominfoBot
Disallow: /

User-agent: Sogou
Disallow: /

User-agent: Zend_Http_Client
Disallow: /

User-agent: robots
Disallow: /

User-agent: arquivo-web-crawler
Disallow: /

User-agent: bidswitchbot
Disallow: /

User-agent: G-i-g-a-b-o-t
Disallow: /

User-agent: Gigabot
Disallow: /

User-agent: GarlikCrawler
Disallow: /

User-agent: CAAM
Disallow: /

User-agent: Clickagy Intelligence Bot
Disallow: /

User-agent: Jersey
Disallow: /

User-agent: libwww-perl
Disallow: /

User-agent: ltx71
Disallow: /

User-agent: PiplBot
Disallow: /

User-agent: Python-urllib
Disallow: /

User-agent: ZoominfoBot
Disallow: /

User-agent: SiteAuditBot
Disallow: /

User-agent: SemrushBot-BA
Disallow: /

User-agent: SemrushBot-SI
Disallow: /

User-agent: SemrushBot-SWA
Disallow: /

User-agent: SemrushBot-CT
Disallow: /

User-agent: SemrushBot-BM
Disallow: /

User-agent: SplitSignalBot
Disallow: /

User-agent: SemrushBot-COUB
Disallow: /
Disallow: /lp
Disallow: /de-de/lp
Disallow: /en-au/lp
Disallow: /en-ca/lp
Disallow: /en-gb/lp
Disallow: /en-in/lp
Disallow: /es-es/lp
Disallow: /es-la/lp
Disallow: /fr-fr/lp
Disallow: /it-it/lp
Disallow: /ja-jp/lp
Disallow: /ko-kr/lp
Disallow: /pt-br/lp
Disallow: /zh-cn/lp
Disallow: /zh-tw/lp
Disallow: /feedback
Disallow: /de-de/feedback
Disallow: /en-au/feedback
Disallow: /en-ca/feedback
Disallow: /en-gb/feedback
Disallow: /en-in/feedback
Disallow: /es-es/feedback
Disallow: /es-la/feedback
Disallow: /fr-fr/feedback
Disallow: /it-it/feedback
Disallow: /ja-jp/feedback
Disallow: /ko-kr/feedback
Disallow: /pt-br/feedback
Disallow: /zh-cn/feedback
Disallow: /zh-tw/feedback

History

Changes recorded since June 2026.

No change recorded.

In context

Mexico
11.1% block at least one AI crawler
Behind Cloudflare
No