CrawlerPulse
CrawlerPulse / Asia / India / mathrubhumi.com

Mathrubhumi

mathrubhumi.com · India · The Mathrubhumi Printing & Publishing Co Ltd · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Allowed
OpenAI
Blocked
Anthropic
Allowed
Perplexity
Allowed
Meta
Allowed
Common Crawl
Blocked
ByteDance
Blocked
Apple
Allowed
Amazon
Blocked

robots.txt

As read on 8 Oct 2026, 1,714 bytes. 1 rule group name AI crawlers; shown first.

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: Amazonbot
User-agent: Bytespider
User-agent: CCBot
User-agent: YouBot
User-agent: xAI-Crawler
User-agent: DeepSeek
Disallow: /

User-agent: *
Allow: /
Disallow: /WEB-INF
Disallow: /META-INF
Disallow: /polopolydevelopment/
Disallow: /status/
Disallow: /search-results/
Disallow: /poll/
Disallow: /logger/
Disallow: /cm/
Disallow: /cmlink/
Disallow: /error/
Disallow: /errorpages/
Disallow: /content/
Disallow: /captcha/
Disallow: /membership/
Disallow: /blogimageupload/
Disallow: /263
Disallow: /json
Disallow: /tabjson
Disallow: /webpage
Disallow: /sponsored/
Disallow: /section/
Disallow: /image/contentid/
Disallow: /archives/
Disallow: /fonts/
Disallow: /api/
Disallow: /money/products-and-services

User-agent: facebookexternalhit
Allow: /

User-agent: Facebot
Allow: /
Sitemap: https://www.mathrubhumi.com/sitemaps/mathrubhumi/sitemap_news1.xml
Sitemap: https://www.mathrubhumi.com/sitemaps/mathrubhumi/sitemap_index.xml
Sitemap: https://www.mathrubhumi.com/video-sitemap
Sitemap: https://www.mathrubhumi.com/rss/movies-music
Sitemap: https://www.mathrubhumi.com/rss/sports
Sitemap: https://www.mathrubhumi.com/rss/news/kerala
Sitemap: https://www.mathrubhumi.com/rss/news/india
Sitemap: https://www.mathrubhumi.com/rss/money
Sitemap: https://www.mathrubhumi.com/rss/health
Sitemap: https://www.mathrubhumi.com/rss/literature
Sitemap: https://www.mathrubhumi.com/rss/technology
Sitemap: https://www.mathrubhumi.com/rss/crime
Sitemap: https://www.mathrubhumi.com/rss/auto
Sitemap: https://www.mathrubhumi.com/rss/travel
Sitemap: https://www.mathrubhumi.com/rss/lifestyle
Sitemap: https://www.mathrubhumi.com/rss/premium

History

Changes recorded since June 2026.

No change recorded.

In context

India
25.0% block at least one AI crawler
Behind Cloudflare
No