CrawlerPulse
CrawlerPulse / Asia / India / india.com

India.com

india.com · India · Indiadotcom Digital Private Limited (Formerly Zee Digital) · measured 4 Aug 2026

Google Search
Allowed
Google-Extended
Allowed
OpenAI
Allowed
Anthropic
Allowed
Perplexity
Allowed
Meta
Allowed
Common Crawl
Allowed
ByteDance
Allowed
Apple
Allowed
Amazon
Allowed

robots.txt

As read on 4 Aug 2026, 1,770 bytes. 1 rule group name AI crawlers; shown first.

User-agent: ClaudeBot
User-agent: ChatGPT
User-agent: GPTBot
User-agent: GPT
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: claude-web
User-agent: ChatGPT-User
User-agent: SemrushBot
User-agent: Yandex
User-agent: Google-Extended
Allow: /
Sitemap: https://www.india.com/google-news-sitemap.xml
Sitemap: https://www.india.com/hindi-news/sitemap.xml
Sitemap: https://www.india.com/hindi-news-sitemap.xml
Sitemap: https://www.india.com/marathi/sitemap-marathi.xml
Sitemap: https://www.india.com/complete-sitemap.xml
Sitemap: https://www.india.com/complete-image-sitemap.xml
Sitemap: https://www.india.com/complete-videos-sitemap.xml
Sitemap: https://www.india.com/special-sitemap.xml
Sitemap: https://www.india.com/ifsc-code-sitemap.xml
Sitemap: https://www.india.com/sitemap-topic.xml
Sitemap: https://www.india.com/travel/sitemap.xml
Sitemap: https://www.india.com/sitemap-webstory-english.xml
Sitemap: https://www.india.com/sitemap-webstory-hindi.xml
Sitemap: https://petuz.india.com/petuz-complete-sitemap.xml
Sitemap: https://influencers.india.com/hindi/influencer-complete-sitemap.xml

User-agent: *
Allow: /
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-admin/
Disallow: /deals/
Disallow: /sponsored/
Disallow: /webstories/smart-buy/
Disallow: /smart-buy/
Disallow: /hindi-news/smart-buy/
Disallow: /independence.php
Disallow: /mcd-election-2017
Disallow: /mcd-election-2017/*
Disallow: */attachment/*
Disallow: /topics/bgr_api/
Disallow: /hindi-news/brand-solution-hindi/
Disallow: /digital-newsletter/
Disallow: /?page=*
Disallow: /?
Disallow: */teamstats_partnership.php?matchid=*
Disallow: /icomtopnew/*
Disallow: /*/wp-ajax-live-score.php?matchId=*
Disallow: /*/loadmore
Disallow: /*&*/feed/
Disallow: /brand-solution/
Disallow: /trivia/

History

Changes recorded since June 2026.

No change recorded.

In context

India
25.0% block at least one AI crawler
Behind Cloudflare
No
Same group