Live Hindustan

livehindustan.com · India · HT Digital Streams Ltd · measured 8 Oct 2026

Google Search

Partial

Google-Extended

Partial

OpenAI

Partial

Anthropic

Partial

Perplexity

Partial

robots.txt

As read on 8 Oct 2026, 3,032 bytes. 1 rule group name AI crawlers; shown first.

User-agent: *
User-agent: Googlebot
User-agent: AdsBot-Google
User-agent: Mediapartners-Google
User-agent: Google-Extended
User-agent: AmazonAdBot
Allow: /news-sitemap.xml
Disallow: /admin/
Disallow: /var/www/html/ht/home/search
Disallow: /home/archive.php
Disallow: /errorfound/error.php
Disallow: /blocked.html
Disallow: /home/homepage.php
Disallow: /home
Disallow: /home1
Disallow: /jmobile/*
Disallow: /mobile/*
Disallow: /rashi/*
Disallow: /noad*
Disallow: /adtype1*
Disallow: /{{url}}
Disallow: /{{image}}
Disallow: /event/selfie/*
Disallow: /news-brief/*
Disallow: /biharelection/
Disallow: /edu/*
Disallow: /cricket/cricket-player/
Disallow: /podcasts
Disallow: /cdp/api/*
Disallow: /api/authenticateToken
Disallow: /api/oneTap
Disallow: /api.livehindustan.com/subscription/*
Disallow: /www.hindustantimes.com/cdp/*
Disallow: /cookies/*
Disallow: /s/*
Disallow: /lh-react-staging.livehindustan.com/*
Disallow: /react-qa.livehindustan.com/*
Disallow: /preprod-boardresults.livehindustan.com/*
Disallow: /react-qa-board.livehindustan.com/*
Disallow: /react-preprod-echo.livehindustan.com/*
Disallow: /react-qa-echo.livehindustan.com/*
Disallow: /lh-img/smart/bpsdk/poll-quiz-survey/prod/non-amp/domains/lh/env/
Disallow: /brand-post/news*
Disallow: /tags//tags/
Disallow: /preprod.livehindustan.com/*
Disallow: /*.ts$
Disallow: /*.m3u8$
Disallow: /news/*

User-agent: Googlebot-News
Allow: /news-sitemap.xml
Disallow: /admin/
Disallow: /var/www/html/ht/home/search
Disallow: /home/archive.php
Disallow: /errorfound/error.php
Disallow: /blocked.html
Disallow: /home/homepage.php
Disallow: /home
Disallow: /home1
Disallow: /jmobile/*
Disallow: /mobile/*
Disallow: /rashi/*
Disallow: /noad*
Disallow: /adtype1*
Disallow: /{{url}}
Disallow: /{{image}}
Disallow: /event/selfie/*
Disallow: /news-brief/*
Disallow: /biharelection/
Disallow: /edu/*
Disallow: /cricket/cricket-player/
Disallow: /podcasts
Disallow: /cdp/api/*
Disallow: /api/authenticateToken
Disallow: /api/oneTap
Disallow: /api.livehindustan.com/subscription/*
Disallow: /www.hindustantimes.com/cdp/*
Disallow: /cookies/*
Disallow: /s/*
Disallow: /lh-react-staging.livehindustan.com/*
Disallow: /react-qa.livehindustan.com/*
Disallow: /preprod-boardresults.livehindustan.com/*
Disallow: /react-qa-board.livehindustan.com/*
Disallow: /react-preprod-echo.livehindustan.com/*
Disallow: /react-qa-echo.livehindustan.com/*
Disallow: /lh-img/smart/bpsdk/poll-quiz-survey/prod/non-amp/domains/lh/env/
Disallow: /brand-post/news*
Disallow: /tags//tags/
Disallow: /preprod.livehindustan.com/*
Disallow: /*.ts$
Disallow: /*.m3u8$
Disallow: /news/*

User-agent: facebookexternalhit
User-agent: Facebot
Allow: /s/
Sitemap: https://www.livehindustan.com/news-sitemap.xml
Sitemap: https://www.livehindustan.com/category-sitemap.xml
Sitemap: https://www.livehindustan.com/sitemap-index.xml
Sitemap: https://www.livehindustan.com/astrology-sitemap.xml
Sitemap: https://www.livehindustan.com/market-share-price-sitemap.xml
Sitemap: https://www.livehindustan.com/liveblog-sitemap.xml

History

Changes recorded since June 2026.

No change recorded.

In context

India

25.0% block at least one AI crawler

Behind Cloudflare

Same group

marathi.hindustantimes.com, bangla.hindustantimes.com, kannada.hindustantimes.com, telugu.hindustantimes.com, tamil.hindustantimes.com, hindustantimes.com, livemint.com