CrawlerPulse
CrawlerPulse / Asia / India / espncricinfo.com

espncricinfo.com

espncricinfo.com · India · stats.espncricinfo.com · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Blocked
OpenAI
Blocked
Anthropic
Allowed
Perplexity
Allowed
Meta
Allowed
Common Crawl
Allowed
ByteDance
Allowed
Apple
Allowed
Amazon
Allowed

robots.txt

As read on 8 Oct 2026, 1,347 bytes. 2 rule groups name AI crawlers; shown first.

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Googlebot
Allow: /

User-agent: *
Allow: /*
Sitemap: https://www.espncricinfo.com/sitemap.xml
Sitemap: https://www.espncricinfo.com/sitemap/news-sitemap.xml
Disallow: /*wrappertype=print
Disallow: /*/content/url/
Disallow: /*/content/current/url/
Disallow: /error
Disallow: /fragments/
Disallow: /logos/
Disallow: /country-fragment/
Disallow: /country-fragment2/
Disallow: /cgi-bin/
Disallow: /classes/
Disallow: /format/
Disallow: /frames/
Disallow: /db/HELPFILES/
Disallow: /db/MANAGEMENT/
Disallow: /db/MISC/CRICINFO_DATA/
Disallow: /db/SUPPORT/ADVERTS/
Disallow: /db/SUPPORT/AFP/
Disallow: /db/SUPPORT/BSTAR/
Disallow: /db/SUPPORT/DAWN/
Disallow: /db/SUPPORT/DAWSON/
Disallow: /db/SUPPORT/ET/
Disallow: /db/SUPPORT/JAGGED/
Disallow: /db/SUPPORT/SHOP/
Disallow: /db/ARCHIVE/
Disallow: /db/INTERACTIVE/
Disallow: /db/NATIONAL/
Disallow: /db/photos/
Disallow: /greatestallrounder/
Disallow: /icc_cricket_worldcup2011/
Disallow: /radio/
Disallow: /games/content/site/games/
Disallow: /hsbcview/
Disallow: /hsbcviews/
Disallow: /internal_applications/
Disallow: /series/india-tour-of-south-africa-2013-14-648627/
Disallow: /audio/
Disallow: /video/
Disallow: /barbados/
Disallow: /*?cmp
Disallow: /*?addata
Disallow: /*?wrappertype=print
Disallow: /team/compare

History

Changes recorded since June 2026.

No change recorded.

In context

India
25.0% block at least one AI crawler
Behind Cloudflare
No