CrawlerPulse
CrawlerPulse / North America / Panama / metrolibre.com

metrolibre.com

metrolibre.com · Panama · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Allowed
OpenAI
Allowed
Anthropic
Allowed
Perplexity
Allowed
Meta
Blocked
Common Crawl
Allowed
ByteDance
Blocked
Apple
Allowed
Amazon
Blocked

robots.txt

As read on 8 Oct 2026, 6,704 bytes. 5 rule groups name AI crawlers; shown first.

User-agent: PetalBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: ImagesiftBot
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: meta-externalagent
Disallow: /

sitemap: https://www.metrolibre.com/sitemap.xml
sitemap: https://www.metrolibre.com/sitemapforgoogle.xml
sitemap: https://www.metrolibre.com/megasitemap.xml

User-agent: *
Disallow: /news-portlet/metalocator/
Disallow: /news-portlet/html/teaser-viewer-portlet/teaser_page.jsp
Disallow: /news-portlet/html/teaser-viewer-portlet/teaser_filter.jsp
Disallow: /news-portlet/filterteaser/
Disallow: /news-portlet/getfilteropts/
Disallow: /tracking-portlet/html/ranking-viewer/ranking_details.jsp
Disallow: /user-portlet/login-with/
Disallow: /user-portlet/edit-user-profile/
Disallow: /user-portlet/reset-credentials/
Disallow: /user-portlet/confirm-email/
Disallow: /user-portlet/refreshuserentitlements/
Disallow: /user-portlet/getEntitlements/
Disallow: /group/
Disallow: /user/
Disallow: /web/
Disallow: /image/
Disallow: /busquedas/

User-agent: *
Disallow: /inicio-sesion/
Disallow: /registro/
Disallow: /editar-perfil/
Disallow: /recuperar-contrasena/
Disallow: /false/false/
Disallow: /error-404
Disallow: /css/
Disallow: /img/
Disallow: /scripts/
Disallow: /c/
Disallow: /busqueda/-/search/
Disallow: /search/
Disallow: /includes/
Disallow: /misc/
Disallow: /modules/
Disallow: /profiles/
Disallow: /scripts/
Disallow: /themes/
Disallow: /CHANGELOG.txt
Disallow: /logout/
Disallow: /node/add
Disallow: /search/
Disallow: /search
Disallow: /user/register/
Disallow: /user/register
Disallow: /user/password/
Disallow: /user/password
Disallow: /user/login/
Disallow: /user/login
Disallow: /laboratorio/
Disallow: /laboratorio

User-Agent: The Knowledge AI
Disallow: /

User-Agent: MauiBot
Disallow: /

User-agent: ia_archiver
Disallow: /

User-agent: UbiCrawler
Disallow: /

User-agent: DOC
Disallow: /

User-agent: Zao
Disallow: /

User-agent: sitecheck.internetseer.com
Disallow: /

User-agent: Zealbot
Disallow: /

User-agent: MSIECrawler
Disallow: /

User-agent: SiteSnagger
Disallow: /

User-agent: WebStripper
Disallow: /

User-agent: WebCopier
Disallow: /

User-agent: Fetch
Disallow: /

User-agent: Offline Explorer
Disallow: /

User-agent: Teleport
Disallow: /

User-agent: TeleportPro
Disallow: /

User-agent: WebZIP
Disallow: /

User-agent: linko
Disallow: /

User-agent: HTTrack
Disallow: /

User-agent: Microsoft.URL.Control
Disallow: /

User-agent: Xenu
Disallow: /

User-agent: larbin
Disallow: /

User-agent: libwww
Disallow: /

User-agent: ZyBORG
Disallow: /

User-agent: Download Ninja
Disallow: /

User-agent: Slurp
Disallow: /

User-agent: Maxthon
Disallow: /

User-agent: CNCDialer
Disallow: /

User-agent: JoLAri
Disallow: /

User-agent: ImageBot
Disallow: /

User-agent: img2dataset
Disallow: /

User-agent: AhrefsBot
Disallow: /

User-agent: SemrushBot
Disallow: /

User-agent: SemrushBot-BA
Disallow: /

User-agent: SemrushBot-EO
Disallow: /

User-agent: SERankingBacklinksBot
Disallow: /

User-agent: serpstatbot
Disallow: /

User-agent: DataForSeoBot
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: dotbot
Disallow: /

User-agent: BLEXBot
Disallow: /

User-agent: barkrowler
Disallow: /

User-agent: SleepBot
Disallow: /

User-agent: ExaSearchBot
Disallow: /

User-agent: Baiduspider
Disallow: /

User-agent: Baiduspider-render
Disallow: /

User-agent: Baiduspider-image
Disallow: /

User-agent: Sogou web spider
Disallow: /

User-agent: SeznamBot
Disallow: /

User-agent: coccocbot-web
Disallow: /

User-agent: rednote-websearch-bot
Disallow: /

User-agent: Seekport
Disallow: /

User-agent: AwarioSmartBot
Disallow: /

User-agent: UnzoiNewsBot
Disallow: /

User-agent: Leikibot
Disallow: /

User-agent: ZoominfoBot
Disallow: /

User-agent: MediatoolkitBot
Disallow: /

User-agent: LivelapBot
Disallow: /

User-agent: TrendictionBot
Disallow: /

User-agent: Yandex
Disallow: /

User-agent: YandexBot
Disallow: /

User-agent: YandexImages
Disallow: /

User-agent: KeenableBot
Disallow: /

User-agent: Amzn-SearchBot
Disallow: /

User-agent: SofyaBot
Disallow: /

User-agent: HyperSearchBot
Disallow: /

User-agent: tphotobot
Disallow: /

User-agent: meta-webindexer
Disallow: /

User-agent: Noticias-GdeltIngest
Disallow: /

User-agent: TerraCotta
Disallow: /

History

Changes recorded since June 2026.

No change recorded.

In context

Panama
10.0% block at least one AI crawler
Behind Cloudflare
Yes