CrawlerPulse
CrawlerPulse / South America / Brazil / istoe.com.br

IstoÉ

istoe.com.br · Brazil · Isto É Publicações LTDA · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Allowed
OpenAI
Allowed
Anthropic
Allowed
Perplexity
Allowed
Meta
Allowed
Common Crawl
Allowed
ByteDance
Allowed
Apple
Allowed
Amazon
Allowed

robots.txt

As read on 8 Oct 2026, 2,107 bytes. No rule names an AI crawler.

Sitemap: https://istoe.com.br/wp-sitemap.xml

User-agent: *
Allow: /wp-admin/admin-ajax.php
Allow: /*/*.css
Allow: /*/*.js
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /readme.html
Disallow: /license.txt
Disallow: /xmlrpc.php
Disallow: /wp-login.php
Disallow: /wp-register.php
Disallow: */disclaimer/*
Disallow: *?attachment_id=
Disallow: /privacy-policy

User-agent: Googlebot
Allow: /

User-agent: Googlebot-Image
Allow: /wp-content/uploads/

User-agent: Mediapartners-Google
Allow: /

User-agent: AdsBot-Google
Allow: /

User-agent: AdsBot-Google-Mobile
Allow: /

User-agent: Bingbot
Allow: /

User-agent: Msnbot
Allow: /

User-agent: msnbot-media
Allow: /wp-content/uploads/

User-agent: Applebot
Allow: /

User-agent: Yandex
Allow: /

User-agent: YandexImages
Allow: /wp-content/uploads/

User-agent: Slurp
Allow: /

User-agent: DuckDuckBot
Allow: /

User-agent: Qwantify
Allow: /

User-agent: ia_archiver
Disallow: /

User-agent: archive.org_bot
Disallow: /

User-agent: SiteExplorer
Disallow: /

User-agent: spbot
Disallow: /

User-agent: WBSearchBot
Disallow: /

User-agent: linkdexbot
Disallow: /

User-agent: Screaming Frog SEO Spider
Disallow: /

User-agent: netEstate NE Crawler
Disallow: /

User-agent: Moreover
Disallow: /

User-agent: sentibot
Disallow: /

User-agent: Aboundexbot
Disallow: /

User-agent: proximic
Disallow: /

User-agent: oBot
Disallow: /

User-agent: meanpathbot
Disallow: /

User-agent: Nutch
Disallow: /

User-agent: TurnitinBot
Disallow: /

User-agent: ZoominfoBot
Disallow: /

User-agent: ZmEu
Disallow: /

User-agent: grapeshot
Disallow: /

User-agent: python-requests
Disallow: /

User-agent: Go-http-client
Disallow: /

User-agent: Apache-HttpClient
Disallow: /

User-agent: libwww-perl
Disallow: /

User-agent: curl
Disallow: /

User-agent: wget
Disallow: /

User-agent: *
Allow: /ads.txt

User-agent: *
Allow: /app-ads.txt
Crawl-delay: 5

History

Changes recorded since June 2026.

No change recorded.

In context

Brazil
14.5% block at least one AI crawler
Behind Cloudflare
No
Same group