CrawlerPulse
CrawlerPulse / North America / Mexico / globalmedia.mx

Global Media el portal de noticias más completo de San Luis Potosí || Noticias de San Luis Potosí

globalmedia.mx · Mexico · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Allowed
OpenAI
Allowed
Anthropic
Allowed
Perplexity
Allowed
Meta
Allowed
Common Crawl
Blocked
ByteDance
Allowed
Apple
Allowed
Amazon
Allowed

robots.txt

As read on 8 Oct 2026, 1,924 bytes. 3 rule groups name AI crawlers; shown first.

User-agent: ShowyouBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: omgili
Disallow: /

User-agent: *
Disallow: /angular/
Disallow: /assets/
Sitemap: https://www.globalmedia.mx/sitemap.xml
Sitemap: https://www.globalmedia.mx/google_news.xml

User-agent: Genio
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: ScooperBot
Disallow: /

User-agent: rogerbot
Disallow: /

User-agent: Flamingo_SearchEngine
Disallow: /

User-agent: Facebot
Disallow: /

User-agent: LuminateBot
Disallow: /

User-agent: Vagabondo
Disallow: /

User-agent: AhrefsBot
Disallow: /

User-agent: SeznamBot
Disallow: /

User-agent: R6_CommentReader
Disallow: /

User-agent: Yeti
Disallow: /

User-agent: heritrix
Disallow: /

User-agent: Baiduspider
Disallow: /

User-agent: GozaikBot
Disallow: /

User-agent: python-requests
Disallow: /

User-agent: QuerySeekerSpider
Disallow: /

User-agent: DotBot
Disallow: /

User-agent: YandexImages
Disallow: /

User-agent: Apache-HttpClient
Disallow: /

User-agent: PiplBot
Disallow: /

User-agent: Scrapy
Disallow: /

User-agent: Buck
Disallow: /

User-agent: WikiDo
Disallow: /

User-agent: ZoominfoBot
Disallow: /

User-agent: Sogou
Disallow: /

User-agent: Zend_Http_Client
Disallow: /

User-agent: robots
Disallow: /

User-agent: arquivo-web-crawler
Disallow: /

User-agent: bidswitchbot
Disallow: /

User-agent: G-i-g-a-b-o-t
Disallow: /

User-agent: Gigabot
Disallow: /

User-agent: GarlikCrawler
Disallow: /

User-agent: CAAM
Disallow: /

User-agent: Clickagy Intelligence Bot
Disallow: /

User-agent: Jersey
Disallow: /

User-agent: libwww-perl
Disallow: /

User-agent: ltx71
Disallow: /

User-agent: PiplBot
Disallow: /

User-agent: Python-urllib
Disallow: /

User-agent: ZoominfoBot
Disallow: /

History

Changes recorded since June 2026.

No change recorded.

In context

Mexico
11.1% block at least one AI crawler
Behind Cloudflare
Yes