CrawlerPulse
CrawlerPulse / Asia / India / deccanherald.com

Deccan Herald

deccanherald.com · India · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Blocked
OpenAI
Blocked
Anthropic
Blocked
Perplexity
Blocked
Meta
Allowed
Common Crawl
Blocked
ByteDance
Blocked
Apple
Allowed
Amazon
Blocked

robots.txt

As read on 8 Oct 2026, 3,666 bytes. 11 rule groups name AI crawlers; shown first.

User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Bytespider-webpage
Disallow: /

User-agent: Amazonbot
Disallow: /

Sitemap: https://www.deccanherald.com/news_sitemap.xml
Sitemap: https://www.deccanherald.com/sitemap.xml

User-agent: Mediapartners-Google
Allow: /

User-agent: SeekportBot
Disallow: /

User-agent: Yandex
Disallow: /

User-agent: GPTCrawler
Disallow: /

User-agent: Perplexity-User
Disallow: /

User-agent: *
Allow: /
Disallow: /includes/
Disallow: /modules/
Disallow: /profiles/
Disallow: /scripts/
Disallow: /themes/
Disallow: /CHANGELOG.txt
Disallow: /cron.php
Disallow: /INSTALL.mysql.txt
Disallow: /INSTALL.pgsql.txt
Disallow: /INSTALL.sqlite.txt
Disallow: /install.php
Disallow: /INSTALL.txt
Disallow: /LICENSE.txt
Disallow: /MAINTAINERS.txt
Disallow: /update.php
Disallow: /UPGRADE.txt
Disallow: /xmlrpc.php
Disallow: /newsletter-ads.html
Disallow: /brandspot/pr-spot/
Disallow: /admin/
Disallow: /comment/reply/
Disallow: /filter/tips/
Disallow: /node/add/
Disallow: /search/
Disallow: /user/register/
Disallow: /user/password/
Disallow: /user/login/
Disallow: /user/logout/
Disallow: /taxonomy/*
Disallow: /node/*
Disallow: /?q=admin/
Disallow: /?q=comment/reply/
Disallow: /?q=filter/tips/
Disallow: /?q=node/add/
Disallow: /?q=search/
Disallow: /?q=user/password/
Disallow: /?q=user/register/
Disallow: /?q=user/login/
Disallow: /?q=user/logout/
Disallow: /updateemoji
Disallow: /updatestats
Disallow: /updateemoji*
Disallow: /updatestats*
Disallow: /brandspot/sponsored/*
Disallow: /brandpr/*
Disallow: /amp/story/brandspot/sponsored/*
Disallow: /amp/story/brandpr/*
Disallow: /brandspot/pr-spot/*
Disallow: /amp/story/brandspot/pr-spot/*
Disallow: /brandspot/*
Disallow: /amp/story/brandspot/*
Disallow: /dhbrandpr/*
Disallow: /amp/story/dhbrandpr/*
Disallow: /dhbrandpr-health/*
Disallow: /amp/story/dhbrandpr-health/*
Disallow: /dhbrandspot-pr/*
Disallow: /amp/story/dhbrandspot-pr/*
Disallow: /dhsponsored-health/*
Disallow: /amp/story/dhsponsored-health/*
Disallow: /banner/live
Disallow: /introductory-999
Disallow: /subscribe-faq
Disallow: /testimonial
Disallow: /introductory_999
Disallow: /limited_time_999
Disallow: /limited_time_149
Disallow: /app-home-lp
Disallow: /app-india-lp
Disallow: /app-opinion-lp
Disallow: /app-karnataka-lp
Disallow: /app-world-lp
Disallow: /app-business-lp
Disallow: /app-sports-lp
Disallow: /app-entertainment-lp
Disallow: /app-video-lp
Disallow: /tags/app-video
Disallow: /tags/app-photos
Disallow: /tags/app-notification
Disallow: /tags/app-settings
Disallow: /tags/app-feedback
Disallow: /tags/app-share
Disallow: /tags/app-version
Disallow: /tags/app-hamburger-more-option
Disallow: /app-dh-specials-lp
Disallow: /app-science-lp
Disallow: /apple-app-site-association
Disallow: /.well-known/assetlinks.json
Disallow: /*?app=true
Disallow: /app-photo-lp
Disallow: /app-photos-daily
Disallow: /delete-account
Disallow: /app-assembly-polls-lp
Disallow: /app-budget-2025
Disallow: /app-delhi-polls-2025-lp
Disallow: /app-budget-2026-lp
Disallow: /app-widget
Disallow: /students-contest

History

Changes recorded since June 2026.

No change recorded.

In context

India
25.0% block at least one AI crawler
Behind Cloudflare
Yes
Same group