CrawlerPulse
CrawlerPulse / Asia / India / indiatimes.com

India Times

indiatimes.com · India · Bennett Coleman & Co. Ltd. · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Allowed
OpenAI
Allowed
Anthropic
Allowed
Perplexity
Allowed
Meta
Blocked
Common Crawl
Allowed
ByteDance
Allowed
Apple
Allowed
Amazon
Allowed

robots.txt

As read on 8 Oct 2026, 1,143 bytes. 1 rule group name AI crawlers; shown first.

User-agent: meta-externalagent
Disallow: /

User-agent: AmazonAdBot
Allow: /
Allow: /*

User-agent: *
Allow: /
Disallow: *.php*
Disallow: */7176/*
Disallow: */whatshot/*
Disallow: */wattpad/*
Disallow: */uc_feed/*
Disallow: */brief/*
Disallow: */search/*
Disallow: */user/*
Disallow: */idtoin/*
Disallow: */video_player/*
Disallow: */blog/*
Disallow: */content/*
Disallow: */seoarchive/*
Disallow: */trends/audios/*
Disallow: */trends/videos/*
Disallow: *{*
Disallow: *?ref=*
Disallow: *?v=*
Disallow: *?no_fullscreen_*
Disallow: *?symbol=*
Disallow: */?date=*
Disallow: *?fb_comment_id=*
Disallow: *?_twitter*
Disallow: */2011/*
Disallow: */2012/*
Disallow: */2013/*
Disallow: */2014/*
Disallow: */2015/*
Disallow: */2016/*
Disallow: */2017/*
Disallow: */2018/*
Disallow: */2019/*
Disallow: */2020/*
Disallow: */2021/*
Disallow: */2022/*
Disallow: */2023/*
Disallow: */2024/*
Disallow: *.ttf*
Disallow: *.woff*
Disallow: *.png*
Disallow: *.jpg*
Disallow: /partner
Sitemap:https://www.indiatimes.com/staticsitemap/id-454/sitemap-index.xml
Sitemap:https://www.indiatimes.com/staticsitemap/id-454/sitemap-48hours.xml

History

Changes recorded since June 2026.

No change recorded.

In context