bengali.news18.com · India · Network18 Media & Investments Limited · measured 4 Aug 2026
As read on 4 Aug 2026, 3,966 bytes. 8 rule groups name AI crawlers; shown first.
User-agent: CCBot Disallow: / User-Agent: omgili Disallow: / User-Agent: omgilibot Disallow: / User-agent: Bytespider Disallow: / User-agent: ImagesiftBot Disallow: / User-agent: Diffbot Disallow: / User-agent: cohere-ai Disallow: / User-agent: Timpibot Disallow: / User-agent: * Allow: / Disallow: /*studio18/* Disallow: /events/general-election-2019/ Disallow: /api/ Disallow: /login/ Disallow: /login?ref* Disallow: /includes/ Disallow: /SolrPhpClient/ Disallow: /1039154/ Disallow: /uc/ Disallow: /mutual-funds/ Disallow: /insurance/ Disallow: /auto/ Disallow: /footer_ajax_mobile.html Disallow: /tag/%* Disallow: /amp/tag/%* Disallow: /tag/*/page-*/ Disallow: /tag/*/photogallery/page-*/ Disallow: /tag/*/videos/page-*/ Disallow: /tag/*/news/page-*/ Disallow: /amp/tag/*/page-*/ Disallow: /amp/tag/*/photogallery/page-*/ Disallow: /durga-puja-2017/* Disallow: /byline/*/news/page-*/$ Disallow: /byline/*/photogallery/page-*/$ Disallow: /byline/*/videos/page-*/$ Disallow: /byline/*/page-*/$ Disallow: /byline/*/news/ Disallow: /photogallery/*page-1$ Disallow: /photogallery/page-* Disallow: /photogallery/*/page-*/$ Disallow: /durga-puja-2018/* Disallow: /press-release/page-*/$ Disallow: /poila-boisakh-celeb/page-*/$ Disallow: /vod/ Disallow: /*tag/*.* Disallow: /*tag/*:* Disallow: /*tag/*+* Disallow: /*tag/*)* Disallow: /*tag/*(* Disallow: /*page-1page* Disallow: /scorecard/ Disallow: /*/page-*/privacy_policy.html Disallow: /*/page-*/page-*/ Disallow: /videos-sitemap Disallow: /image-sitemap Disallow: /articles-sitemap Disallow: /*/privacy_policy.html/$ Disallow: /news/*/page-* Disallow: /videos/page-*/ Disallow: /durga-puja-2023/ Disallow: /embed/ Disallow: /budget-2021/ Disallow: /videos/*/page-*/$ Disallow: /west-bengal-assembly-elections-2016/ Disallow: /west-bengal-assembly-election-2021/ Disallow: /ampphotogallery/ Disallow: /assembly-elections-2023/ Disallow: /elections/assembly-elections-2018/ Disallow: /photogallery/*/privacy_policy.html Disallow: /ipl-2020/ Disallow: /1 Disallow: /2 Disallow: /3 Disallow: /4 Disallow: /5 Disallow: /6 Disallow: /7 Disallow: /8 Disallow: /9 Disallow: /0 Disallow: /amp/tag/1 Disallow: /amp/tag/2 Disallow: /amp/tag/3 Disallow: /amp/tag/4 Disallow: /amp/tag/5 Disallow: /amp/tag/6 Disallow: /amp/tag/7 Disallow: /amp/tag/8 Disallow: /amp/tag/9 Disallow: /tag/1 Disallow: /tag/2 Disallow: /tag/3 Disallow: /tag/4 Disallow: /tag/5 Disallow: /tag/6 Disallow: /tag/7 Disallow: /tag/8 Disallow: /tag/9 Disallow: /tag/*/news Disallow: /tag/*/photogallery Disallow: /tag/*/videos Disallow: /*/<iframe width= Disallow: /cricketnext/ Disallow: /ipl-2021/ Disallow: /*/privacy_policy.html Disallow: /*/astrology/horoscope/todays- Disallow: /*/hajj-2021/ Disallow: /board-results-pubstack/ Disallow: /assembly-election-2019/ Disallow: /poila-boisakh-rural-bengal/ Disallow: /elections/assembly/*/* Allow: /cricket/live-score/$ Disallow: /cricket/live-score/* Allow: /elections/assembly/west-bengal/* Disallow: /elections/*constituency- Disallow: /elections/*candidate- Disallow: /elections/*-led20 Disallow: /zip/* Disallow: /*/zip/* Sitemap: https://bengali.news18.com/commonfeeds/v1/ben/sitemap/today Sitemap: https://bengali.news18.com/commonfeeds/v1/ben/sitemap-index.xml Sitemap: https://bengali.news18.com/commonfeeds/v1/ben/sitemap/google-news.xml Sitemap: https://bengali.news18.com/commonfeeds/v1/ben/sitemap-image-index.xml Sitemap: https://bengali.news18.com/commonfeeds/v1/ben/sitemap-video-index.xml Sitemap: https://bengali.news18.com/commonfeeds/v1/ben/sitemap/webstories-sitemap-index.xml User-agent: MAZBot Disallow: / User-agent: Baiduspider Disallow: / User-agent: FriendlyCrawler Disallow: / User-agent: img2dataset Disallow: / User-agent: Scrapy Disallow: / User-agent: VelenPublicWebCrawler Disallow: /
Changes recorded since June 2026.
No change recorded.