hindi.news18.com · India · Network18 Media & Investments Limited · measured 4 Aug 2026
As read on 4 Aug 2026, 5,038 bytes. 14 rule groups name AI crawlers; shown first.
User-agent: CCBot Disallow: / User-agent: GPTBot Disallow: / User-Agent: omgili Disallow: / User-Agent: omgilibot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: Google-Extended Disallow: / User-agent: Bytespider Disallow: / User-agent: ClaudeBot Disallow: / User-agent: ImagesiftBot Disallow: / User-agent: Diffbot Disallow: / User-agent: Claude-Web Disallow: / User-agent: cohere-ai Disallow: / User-agent: Timpibot Disallow: / User-agent: OAI-SearchBot Disallow: / User-Agent: * Allow: / Allow: /short-news/$ Sitemap: https://hindi.news18.com/allstory-sitemap-data.xml Sitemap: https://hindi.news18.com/commonfeeds/v1/hin/sitemap/google-news/yesterday.xml Sitemap: https://hindi.news18.com/commonfeeds/v1/hin/sitemap/google-news/today.xml Disallow: *.htmlnews/*/$ Disallow: */:state_slug:/* Disallow: /zip/* Disallow: /*/zip/* Disallow: /cricket/live-score/undefined-*.html Disallow: /hindi.news18.com/news/*.html$ Disallow: /hindi.news18.com/amp/news/*.html$ Disallow: /shows/ Disallow: /videos/kadak/* Disallow: /events/general-election-2019/* Disallow: /amp/assembly-election-2022/* Disallow: /assembly-elections-2018/* Disallow: /image-sitemap-*.xml Disallow: /election-result-2013/* Disallow: /photogallery/*/*/page-*/ Disallow: /api/* Disallow: /videos-sitemap-*.xml Disallow: /amp/assembly-elections-2022/* Disallow: /assembly-elections-2022/* Disallow: /conversations/ Disallow: /marketing/ Disallow: /1039154/ Disallow: /test/ Disallow: /uc/ Disallow: /home.html Disallow: /jw_player/ Disallow: /scroll/ Disallow: /embedded/ Disallow: /contest/ Disallow: /inner-site/ Disallow: /pwa/ Disallow: /news/education/uc/ Disallow: /history-18/ Disallow: /uc-election-widget/ Disallow: /mutual-funds/ Disallow: /insurance/ Disallow: /automobile/ Disallow: /googlenewssitemaptest.xml Disallow: /nodeapi/ Disallow: /tag/%*/ Disallow: /tag/*/page-*/ Disallow: /tag/*/photogallery/* Disallow: /tag/*/videos/* Disallow: /tag/*/news/* Disallow: /byline/%*/ Disallow: /byline/*/photogallery/page-*/ Disallow: /byline/*/videos/page-*/ Disallow: /byline/*/news/page-*/ Disallow: /agency/%*/ Disallow: /agency/*/page-*/ Disallow: /agency/*/photogallery/page-*/ Disallow: /agency/*/videos/page-*/ Disallow: /agency/*/news/page-*/ Disallow: /etv-shows/ Disallow: /videos/*/page-*/ Disallow: /videos/*/*/page-*/ Disallow: /byline/*htmlpage-*html/page-*/ Disallow: /byline/*htmlpage-*/page-*/ Disallow: /byline/*htmlpage-*/photogallery/ Disallow: /byline/*/page-*/tag/*/ Disallow: */access_log Disallow: */error_log Disallow: /tag/*void-0*/ Disallow: /videos/videos/* Disallow: /news/*/*html/$ Disallow: /astrology/ Disallow: /byline/*page-*/$ Disallow: /anchor/*page-*/$ Disallow: /assembly-election-2020/* Disallow: /news/*page-*/$ Disallow: /assembly-election-2021/* Disallow: /assembly-election-2019/* Disallow: /news/*htmlpage-*.html/$ Disallow: /amp/astrology/*/ Disallow: /tag/*/1*page--*/ Disallow: /rajya-sabha-elections-2020/ Disallow: /Tag/*/page-*/ Disallow: /tag//*/ Disallow: /union-budget-2018/ Disallow: /amp/assembly-election-2020/ Disallow: /assembly-elections/ Disallow: /news/*/page-*/ Disallow: /cricket/profile//*.html Disallow: /embed/videos/*==/ Disallow: /vod/*/livestreaming/ Disallow: /madhya-pradesh/cricket/page-*/ Disallow: /uttar-pradesh/page-*/ Disallow: /app-download/ Disallow: /podcast/news-bulletin/page-*/ Disallow: /podcast/sports-buletin/page-*/ Disallow: /agency/*/news/ Disallow: /agency/*/photogallery/ Disallow: /agency/*/videos/ Disallow: /agency/*page-*/ Disallow: /_next/image/ Disallow: /cricket/series/*/page-*/ Disallow: /cricket/profile/*/news-*/page-*/ Disallow: /podcast/ Disallow: /cricket/match-schedule/series/* Disallow: /lok-sabha-election-*/ Disallow: /amp/assembly-election-*/ Disallow: /states/blogs/page-*/ Disallow: /rajasthan/*/page-*/ Disallow: /cricket/live-score/team-news/*.html Disallow: /uttar-pradesh/*/page-*/ Disallow: /bihar-assembly-election-*/ Disallow: /amp/bihar-assembly-election-*/ Disallow: /amp-bihar-assembly-election-*/ Disallow: /news/uttar-pradesh/*/news Disallow: /byline/*/news Disallow: /amp/news/*/photogallery/ Disallow: /news/*/photogallery/ Disallow: /amp/states/news/ Disallow: /states/news/ Disallow: /asian-games/*/ Disallow: /cricketnext/ Disallow: /amp/cricketnext/ Disallow: /states/static-redirect.php Disallow: /news/byline/*/ Disallow: /sitemaps/daily/*.xml Disallow: /short-news/* Disallow: /byline/*/photogallery/ Disallow: /ampdesk-/news//*.html Disallow: /states/redirect.php?id=* Disallow: /*8741853.html Disallow: /elections/*-led20 Disallow: /mobiles/* User-agent: MAZBot Disallow: / User-agent: Baiduspider Disallow: / User-agent: DataForSeoBot Disallow: / User-agent: FriendlyCrawler Disallow: / User-agent: img2dataset Disallow: / User-agent: Scrapy Disallow: / User-agent: VelenPublicWebCrawler Disallow: /
Changes recorded since June 2026.
No change recorded.