tamil.news18.com · India · Network18 Media & Investments Limited · measured 4 Aug 2026
As read on 4 Aug 2026, 3,851 bytes. 14 rule groups name AI crawlers; shown first.
User-agent: CCBot Disallow: / User-agent: GPTBot Disallow: / User-Agent: omgili Disallow: / User-Agent: omgilibot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: Google-Extended Disallow: / User-agent: Bytespider Disallow: / User-agent: ClaudeBot Disallow: / User-agent: ImagesiftBot Disallow: / User-agent: Diffbot Disallow: / User-agent: Claude-Web Disallow: / User-agent: cohere-ai Disallow: / User-agent: Timpibot Disallow: / User-agent: OAI-SearchBot Disallow: / User-Agent: * Allow: / Disallow: /*studio18/* Disallow: /login/ Disallow: /login?ref* Disallow: /amp/news/*.htmlpage-*/$ Disallow: /amp/tag/*/videos/$ Disallow: /amp/tag/*/videos/page-*/$ Disallow: /amp/tag/*/page-*/ Disallow: /amp/tag/*/photogallery/$ Disallow: /amp/tag/*/photogallery/page-*/$ Disallow: /business/page-*page-* Disallow: /events/general-election-2019/ Disallow: /uc/ Disallow: /marketing/ Disallow: /uc-election-widget/ Disallow: /photogallery/*/page-*/$ Disallow: /category/ Disallow: /*htmlpage-*/ Disallow: /tag/*/page-*/ Disallow: /tag/*/videos/ Disallow: /tag/*/photogallery/ Disallow: /tag/*/news/ Disallow: /vod/ Disallow: /news// Disallow: /byline/*/page-*/ Disallow: /amp/news/*html/page-*/ Disallow: /scorecard/ Disallow: /embed/ Disallow: /tamil-nadu-assembly-election-2021/ Disallow: /ampnews/ Disallow: /commonstore/ Disallow: /tamil-nadu/*/page-*/ Disallow: /astro/page-*/ Disallow: /articles-sitemap Disallow: /assembly-election-2019/ Disallow: /image-sitemap Disallow: /lok-sabha-election-2019/ Disallow: /page-*/ Disallow: /photogallery/*/*page-*page-*page-*/ Disallow: /zodiac-signs/ Disallow: /videos/astro/ Disallow: /assembly-election-2021/ Disallow: /assembly-elections-2021/ Disallow: /spiritual/horoscope/todays-* Disallow: /amp/spiritual/horoscope/todays-* Disallow: /videos-sitemap Disallow: /amp/*/*/tag/ Disallow: /special-articlespage-*page-*/ Disallow: /sportspage-*page-* Disallow: /*/page-*page-* Disallow: /videopub/ Disallow: /*.html>/ Disallow: /*.html>/page-*/ Disallow: /*.html> Disallow: /*/img/Path114.svg Disallow: /*.html>page-*/ Disallow: /*-districtpage-* Disallow: /*/*page-*page-* Disallow: /us-election-2020/ Disallow: /elections-2019/ Disallow: /newpage-1page-1/ Disallow: /*page-*page-* Disallow: /katral-awards-2021/ Disallow: /amp/tamil-nadu-assembly-election-2021/ Disallow: /*.html>/ Disallow: /*/undefined/ Disallow: /photogallery// Disallow: /*.html?pv_candidate= Disallow: /ampvideos/ Disallow: /budget-2018/ Disallow: /news/assembly-election-2019/ Disallow: /elections/winner Disallow: /*.htmllocal-18/ Disallow: /amp/*/news/*/ Disallow: /*/year-ender-2019/ Disallow: /ipl-2021/ Disallow: /1 Disallow: /2 Disallow: /3 Disallow: /4 Disallow: /5 Disallow: /6 Disallow: /7 Disallow: /8 Disallow: /9 Disallow: /0 Allow: /cricket/live-score/$ Disallow: /cricket/live-score/ Disallow: /elections/*constituency- Disallow: /elections/*candidate- Disallow: /elections/*-led20 Disallow: /zip/ Disallow: /*/zip/* Sitemap: https://tamil.news18.com/commonfeeds/v1/tam/sitemap/today Sitemap: https://tamil.news18.com/commonfeeds/v1/tam/sitemap-index.xml Sitemap: https://tamil.news18.com/commonfeeds/v1/tam/sitemap/google-news.xml Sitemap: https://tamil.news18.com/commonfeeds/v1/tam/sitemap-video.xml Sitemap: https://tamil.news18.com/commonfeeds/v1/tam/sitemap/webstories-sitemap-index.xml Sitemap: https://tamil.news18.com/commonfeeds/v1/tam/sitemap-image-index.xml User-agent: MAZBot Disallow: / User-agent: Baiduspider Disallow: / User-agent: DataForSeoBot Disallow: / User-agent: FriendlyCrawler Disallow: / User-agent: img2dataset Disallow: / User-agent: Scrapy Disallow: / User-agent: VelenPublicWebCrawler Disallow: /
Changes recorded since June 2026.
No change recorded.