CrawlerPulse
CrawlerPulse / Asia / India / timesofindia.com

Times of India

timesofindia.com · India · Bennett Coleman & Co. Ltd. · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Allowed
OpenAI
Allowed
Anthropic
Allowed
Perplexity
Allowed
Meta
Blocked
Common Crawl
Allowed
ByteDance
Allowed
Apple
Allowed
Amazon
Allowed

robots.txt

As read on 8 Oct 2026, 11,377 bytes. 1 rule group name AI crawlers; shown first.

User-agent: meta-externalagent
Disallow: /
Sitemap: https://timesofindia.indiatimes.com/sitemap/today
Sitemap: https://timesofindia.indiatimes.com/sitemap/yesterday
Sitemap: https://timesofindia.indiatimes.com/staticsitemap/toi/news/sitemap-index.xml
Sitemap: https://timesofindia.indiatimes.com/staticsitemap/toi/category/city/sitemap-index.xml
Sitemap: https://timesofindia.indiatimes.com/staticsitemap/toi/category/entertainment/sitemap-index.xml
Sitemap: https://timesofindia.indiatimes.com/staticsitemap/toi/category/india/sitemap-index.xml
Sitemap: https://timesofindia.indiatimes.com/staticsitemap/toi/category/sports/sitemap-index.xml
Sitemap: https://timesofindia.indiatimes.com/staticsitemap/toi/category/business/sitemap-index.xml
Sitemap: https://timesofindia.indiatimes.com/staticsitemap/toi/category/world/sitemap-index.xml
Sitemap: https://timesofindia.indiatimes.com/staticsitemap/toi/slideshow/sitemap-index.xml
Sitemap: https://timesofindia.indiatimes.com/staticsitemap/toi/category/life-style/sitemap-index.xml

User-agent: AmazonAdBot
Allow: /
Allow: /*

User-agent: *
Allow: /
Disallow: /PDATOI/
Disallow: /topic/quote/
Disallow: /*cms.dll*
Disallow: /*rssarticleshow*
Disallow: /topic/photo/
Disallow: /default.cms
Disallow: /default1.cms
Disallow: /artshowcmt/
Disallow: */opinions/
Disallow: /breakingnews/toihome.html
Disallow: /topic/article/
Disallow: /logtopickeywords*
Disallow: /new_cmtofart*
Disallow: /pmcomment/
Disallow: /feeds/
Disallow: */configspace/*
Disallow: /configspace/maintenance.html
Disallow: /topiclist/
Disallow: */pmredirectshow/*
Disallow: */stoi/*
Disallow: */moviearticlelistalphabet1/*
Disallow: */cinema-entertainment/*
Disallow: */life-style-landing/*
Disallow: *navlvl=
Disallow: /lspoll/
Disallow: *query=
Disallow: /beta_hotlydbt/
Disallow: *defalpha=
Disallow: *,page
Disallow: /new_cmtofart2_nit_v1/
Disallow: *prtpage
Disallow: */examshow/*
Disallow: */speednewsshow/*
Disallow: */speednewsbytopic/*
Disallow: */itslideshow/*
Disallow: */vsarticleshow/*
Disallow: */articleshow_amitr/*
Disallow: */pmtoinavigation/*
Disallow: */mvopinions/*
Disallow: */opinionslogged/*
Disallow: */rodate.js
Disallow: */load.js
Disallow: */cookie.js
Disallow: */jquery.js
Disallow: */navigation.js
Disallow: */pgtrack.js
Disallow: */lazy.js
Disallow: */breakingnews.js
Disallow: */rodate.js
Disallow: */mail/*
Disallow: */profileshow/*
Disallow: */tpoint_articleshow/*
Disallow: */trivia/*
Disallow: */articleshowajax/*
Disallow: /articlecomment/
Disallow: /artshowcmt/
Disallow: /7176/TOI_Travel/
Disallow: /search
Disallow: /fbpostaction.cms?article=
Disallow: /7176/TOI_Travel_Test/TOI_Travel_TOP_TST
Disallow: /postmail.cms
Disallow: /mostviewed
Disallow: /related
Disallow: /comments
Disallow: *Jai-Mrug*
Disallow: */affiliate_amazon.cms*
Disallow: *-mostviwed*
Disallow: */techgadgets/*
Disallow: */covid-19-delhi/*
Disallow: */covid-19-bangalore/*
Disallow: */covid-19-hyderabad/*
Disallow: */covid-19-kolkata/*
Disallow: */covid-19-mumbai/*
Disallow: */covid-19-across-world/*
Disallow: */crshow/*
Disallow: */briefs/articleid-*
Disallow: /intarnel/*
Disallow: */dynalist/*
Disallow: */amp_msp_amazon_gadget.cms*
Disallow: */affiliates_content_iframe.cms*
Disallow: */affiliates_content.cms*
Disallow: */affiliates_content_iframe*
Disallow: */affiliates_library*
Disallow: */affiliates_content*
Disallow: */ifrmwdt/*
Disallow: *articleshowbriefs*
Disallow: *quickstirshow*
Disallow: *mobile=no*
Disallow: *etarticleshow*
Disallow: *?mobile=no*
Disallow: *homeajax_nav.cms?*
Disallow: *articleshowbriefs?*
Disallow: *affiliates_content_iframe_v2*
Disallow: *?frmapp=yes*
Disallow: *embedvideo.cms*
Disallow: *videodash.cms*
Disallow: *etopneinappamp.cms*
Disallow: *as_comments*
Disallow: *RBP_HOME_URL*
Disallow: *?comments*
Disallow: *?showcomment*
Disallow: *?cmfid*
Disallow: *?pg=*
Disallow: *?ps=*
Disallow: *articleshow_new*
Disallow: *articleshowprint*
Disallow: /microsite_v2
Disallow: */undefined$
Disallow: */articleshowhsbc/
Disallow: */storylist/
Disallow: *asrelated.cms*
Disallow: *amp_next_page_ga.cms*
Disallow: */travel/search?*
Disallow: */articlelist/*
Disallow: /toi_webvital_events_min_js.cms*
Disallow: /popup_manager_priority.cms*
Disallow: /locateservice_js/minify-1,version-14.cms*
Disallow: /slick_min_js_v18/*
Disallow: /enttinyscrolljs/*
Disallow: /beta_gohintjs_v7.cms*
Disallow: /locateservice_js/*
Disallow: /trackjs_newnav.cms*
Disallow: /grxpushnotification_js/version-23,minify-1.cms*
Disallow: /aqiscript/*
Disallow: /navincludejs.cms*
Disallow: /articlelist.cms.js*
Disallow: /navincludejs.cms*
Disallow: /pn/version-39,minify-1.cms*
Disallow: /etimes_login/*
Disallow: /etnewsletter_js/*
Disallow: /glblnavjs_v19.cms*
Disallow: /gaview_script/*
Disallow: /etpn/version-26,minify-1.cms*
Disallow: /new_bookmarkjs_v2.cms*
Disallow: /new_slidjs_v1.cms*
Disallow: /grxevent_config/*
Disallow: /et_gdpr_callback_js/*
Disallow: /et_onetap_signin_js/*
Disallow: /getseeallhref.cms*
Disallow: /getdataofseason.cms*
Disallow: /affiliate_wrapper_amp.cms*
Disallow: *feedtype-sjson*
Disallow: *feedtype-json*
Disallow: *feedtype=sjson*
Disallow: *feedtype=json*
Disallow: /ncCentreFeed*
Disallow: /allauthrlist_v2.cms*
Disallow: /newsletter_rhs_js.cms?minify=1&version=30*
Disallow: /ettoiplus_verifymobile_screen_js/version-90,minify-1,isArchive-1.cms*
Disallow: /requirejs/version-9,minify-1.cms*
Disallow: /jquery_1_11_0_min/version-13,minify-1.cms*
Disallow: /et_gdpr_js/minify-1,v-14.cms*
Disallow: /lgads_v2/minify-1,channel-nbt.cms*
Disallow: /etads_v2/minify-1.cms*
Disallow: *js/v2/web-sdk-main.js*
Disallow: *v2/track*
Disallow: /toiads_prebid/minify-1.cms*
Disallow: /toiads_react_lazy/minify-1.cms*
Disallow: /google_ads/ads/ve-1.cms*
Disallow: /requirejs/version-9,minify-1.cms*
Disallow: /toiads_new/minify-1.cms*
Disallow: /newsletter_rhs_js.cms*
Disallow: /toiads_prebid_checker/adset/ve-1.cms*
Disallow: /ext_*
Disallow: /*_rsc=
Disallow: /meta.json
Disallow: */lottery/*

User-agent: Googlebot-News
Allow: /
Disallow: /PDATOI/
Disallow: /topic/quote/
Disallow: /*cms.dll*
Disallow: /*rssarticleshow*
Disallow: /topic/photo/
Disallow: /default.cms
Disallow: /default1.cms
Disallow: /artshowcmt/
Disallow: */opinions/
Disallow: /breakingnews/toihome.html
Disallow: /topic/article/
Disallow: /logtopickeywords*
Disallow: /new_cmtofart*
Disallow: /pmcomment/
Disallow: /feeds/
Disallow: */configspace/*
Disallow: /configspace/maintenance.html
Disallow: /topiclist/
Disallow: */pmredirectshow/*
Disallow: */stoi/*
Disallow: */moviearticlelistalphabet1/*
Disallow: */cinema-entertainment/*
Disallow: */life-style-landing/*
Disallow: *navlvl=
Disallow: /lspoll/
Disallow: *query=
Disallow: /beta_hotlydbt/
Disallow: *defalpha=
Disallow: *,page
Disallow: /new_cmtofart2_nit_v1/
Disallow: *prtpage
Disallow: */examshow/*
Disallow: */speednewsshow/*
Disallow: */speednewsbytopic/*
Disallow: */itslideshow/*
Disallow: */vsarticleshow/*
Disallow: */articleshow_amitr/*
Disallow: */pmtoinavigation/*
Disallow: */mvopinions/*
Disallow: */opinionslogged/*
Disallow: */rodate.js
Disallow: */load.js
Disallow: */cookie.js
Disallow: */jquery.js
Disallow: */navigation.js
Disallow: */pgtrack.js
Disallow: */lazy.js
Disallow: */breakingnews.js
Disallow: */rodate.js
Disallow: */mail/*
Disallow: */profileshow/*
Disallow: */tpoint_articleshow/*
Disallow: */trivia/*
Disallow: */articleshowajax/*
Disallow: /articlecomment/
Disallow: /artshowcmt/
Disallow: /7176/TOI_Travel/
Disallow: /search
Disallow: /fbpostaction.cms?article=
Disallow: /7176/TOI_Travel_Test/TOI_Travel_TOP_TST
Disallow: /postmail.cms
Disallow: /mostviewed
Disallow: /related
Disallow: /comments
Disallow: *Jai-Mrug*
Disallow: */affiliate_amazon.cms*
Disallow: *-mostviwed*
Disallow: */techgadgets/*
Disallow: */covid-19-delhi/*
Disallow: */covid-19-bangalore/*
Disallow: */covid-19-hyderabad/*
Disallow: */covid-19-kolkata/*
Disallow: */covid-19-mumbai/*
Disallow: */covid-19-across-world/*
Disallow: */crshow/*
Disallow: */briefs/articleid-*
Disallow: /intarnel/*
Disallow: */dynalist/*
Disallow: */amp_msp_amazon_gadget.cms*
Disallow: */affiliates_content_iframe.cms*
Disallow: */affiliates_content.cms*
Disallow: */affiliates_content_iframe*
Disallow: */affiliates_library*
Disallow: */affiliates_content*
Disallow: */ifrmwdt/*
Disallow: *articleshowbriefs*
Disallow: *quickstirshow*
Disallow: *mobile=no*
Disallow: *etarticleshow*
Disallow: *?mobile=no*
Disallow: *homeajax_nav.cms?*
Disallow: *articleshowbriefs?*
Disallow: *affiliates_content_iframe_v2*
Disallow: *?frmapp=yes*
Disallow: *embedvideo.cms*
Disallow: *videodash.cms*
Disallow: *etopneinappamp.cms*
Disallow: *as_comments*
Disallow: *RBP_HOME_URL*
Disallow: *?comments*
Disallow: *?showcomment*
Disallow: *?cmfid*
Disallow: *?pg=*
Disallow: *?ps=*
Disallow: *articleshow_new*
Disallow: *articleshowprint*
Disallow: /microsite_v2
Disallow: */undefined$
Disallow: */articleshowhsbc/
Disallow: */storylist/
Disallow: *asrelated.cms*
Disallow: *amp_next_page_ga.cms*
Disallow: */travel/search?*
Disallow: */articlelist/*
Disallow: /toi_webvital_events_min_js.cms*
Disallow: /popup_manager_priority.cms*
Disallow: /locateservice_js/minify-1,version-14.cms*
Disallow: /slick_min_js_v18/*
Disallow: /enttinyscrolljs/*
Disallow: /beta_gohintjs_v7.cms*
Disallow: /locateservice_js/*
Disallow: /trackjs_newnav.cms*
Disallow: /grxpushnotification_js/version-23,minify-1.cms*
Disallow: /aqiscript/*
Disallow: /navincludejs.cms*
Disallow: /articlelist.cms.js*
Disallow: /navincludejs.cms*
Disallow: /pn/version-39,minify-1.cms*
Disallow: /etimes_login/*
Disallow: /etnewsletter_js/*
Disallow: /glblnavjs_v19.cms*
Disallow: /gaview_script/*
Disallow: /etpn/version-26,minify-1.cms*
Disallow: /new_bookmarkjs_v2.cms*
Disallow: /new_slidjs_v1.cms*
Disallow: /grxevent_config/*
Disallow: /et_gdpr_callback_js/*
Disallow: /et_onetap_signin_js/*
Disallow: /getseeallhref.cms*
Disallow: /getdataofseason.cms*
Disallow: /affiliate_wrapper_amp.cms*
Disallow: *feedtype-sjson*
Disallow: *feedtype-json*
Disallow: *feedtype=sjson*
Disallow: *feedtype=json*
Disallow: /ncCentreFeed*
Disallow: /allauthrlist_v2.cms*
Disallow: /newsletter_rhs_js.cms?minify=1&version=30*
Disallow: /ettoiplus_verifymobile_screen_js/version-90,minify-1,isArchive-1.cms*
Disallow: /requirejs/version-9,minify-1.cms*
Disallow: /jquery_1_11_0_min/version-13,minify-1.cms*
Disallow: /et_gdpr_js/minify-1,v-14.cms*
Disallow: /lgads_v2/minify-1,channel-nbt.cms*
Disallow: /etads_v2/minify-1.cms*
Disallow: *js/v2/web-sdk-main.js*
Disallow: *v2/track*
Disallow: /toiads_prebid/minify-1.cms*
Disallow: /toiads_react_lazy/minify-1.cms*
Disallow: /google_ads/ads/ve-1.cms*
Disallow: /requirejs/version-9,minify-1.cms*
Disallow: /toiads_new/minify-1.cms*
Disallow: /newsletter_rhs_js.cms*
Disallow: /toiads_prebid_checker/adset/ve-1.cms*
Disallow: /ext_*
Disallow: /*_rsc=
Disallow: /meta.json
Disallow: */lottery/*

User-agent: AdsBot-Google
Allow: /trivia/

User-agent: Mediapartners-Google
Allow: /trivia/

User-agent: AmazonAdBot
Allow: /trivia/

History

Changes recorded since June 2026.

No change recorded.

In context