CrawlerPulse
CrawlerPulse / Asia / India / news24online.com

News24 Online

news24online.com · India · BAG Media · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Allowed
OpenAI
Allowed
Anthropic
Blocked
Perplexity
Allowed
Meta
Allowed
Common Crawl
Blocked
ByteDance
Blocked
Apple
Allowed
Amazon
Allowed

robots.txt

As read on 8 Oct 2026, 1,944 bytes. No rule names an AI crawler.

User-agent: *
    Disallow: /wp-admin/
    Allow: /wp-admin/admin-ajax.php
    Disallow: /gov-news/
    Disallow: /product/
    Disallow: /tag/
    Disallow: */page/*
    Disallow: /topic/*/feed/*
    Disallow: /?
    Disallow: /?s=*
    User-agent: PetalBot
    Disallow: /
    User-agent: Nuclei
    Disallow: /
    User-agent: WikiDo
    Disallow: /
    User-agent: Riddler
    Disallow: /
    User-agent: Zoominfobot
    Disallow: /
    User-agent: Go-http-client
    Disallow: /
    User-agent: Node/simplecrawler
    Disallow: /
    User-agent: CazoodleBot
    Disallow: /
    User-agent: dotbot/1.0
    Disallow: /
    User-agent: Gigabot
    Disallow: /
    User-agent: Barkrowler
    Disallow: /
    User-agent: BLEXBot
    Disallow: /
    User-agent: magpie-crawler
    Disallow: /
    User-agent: CCBot
    Disallow: /
    User-agent: Bytespider
    Disallow: /
    User-agent: Baiduspider
    Disallow: /
    User-agent: ClaudeBot
    Disallow: /
    User-agent: TencentBot
    Disallow: /
    User-agent: Sogou Web Spider
    Disallow: /
    User-agent: YisouSpider
    Disallow: /
    User-agent: yacybot
    Disallow: /
    User-agent: Meridian-HP
    Disallow: /
    User-agent: MitraVibe-Syndication
    Disallow: /
    Sitemap: https://news24online.com/sitemap-news.xml
    Sitemap: https://news24online.com/sitemap-author.xml
    Sitemap: https://news24online.com/sitemap-category.xml
    Sitemap: https://news24online.com/sitemap-image.xml
    Sitemap: https://news24online.com/sitemap-photos.xml
    Sitemap: https://news24online.com/sitemap.xml
    Sitemap: https://news24online.com/sitemap.xml?yyyy=2021
    Sitemap: https://news24online.com/sitemap.xml?yyyy=2022
    Sitemap: https://news24online.com/sitemap.xml?yyyy=2023
    Sitemap: https://news24online.com/sitemap.xml?yyyy=2024
    Sitemap: https://news24online.com/sitemap.xml?yyyy=2025

History

Changes recorded since June 2026.

No change recorded.

In context

India
25.0% block at least one AI crawler
Behind Cloudflare
No
Same group