CrawlerPulse
CrawlerPulse / Asia / India / newslaundry.com

newslaundry.com

newslaundry.com · India · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Allowed
OpenAI
Allowed
Anthropic
Blocked
Perplexity
Allowed
Meta
Allowed
Common Crawl
Allowed
ByteDance
Allowed
Apple
Allowed
Amazon
Allowed

robots.txt

As read on 8 Oct 2026, 1,169 bytes. No rule names an AI crawler.

User-agent: *
Disallow: /2014/06/21/the-alchemy-of-an-it-raid

User-agent: SeekportBot
Disallow: /

User-agent: Semrush
Disallow: /

User-agent: Ahref
Disallow: /

User-agent: Dotbot
Disallow: /

User-agent: Claude
Disallow: /

User-agent: Open AI
Disallow: /

User-agent: *
Disallow: /2023/05/19/in-70-days-at-rs-75-lakh-the-curious-case-of-a-9-acre-plot-sold-to-former-khattar-aides-family

User-agent: *
Disallow: /2023/05/19/haryana-chief-minister-manoharlal-khattar-principal-osd-neeraj-daftuar-land-scam

User-agent: *
Disallow: /front-end/subscribe

User-agent: *
Disallow: /acast-rest/

User-agent: *
Disallow: /mobile/

User-agent: *
Disallow: /webnyay/

User-agent: *
Disallow: /podcast-rss/

User-agent: *
Disallow: /discord/

User-agent: *
Disallow: /shopify/

User-agent: *
Disallow: /check-student-subscription/

User-agent: *
Disallow: /create-student-subscription/

User-agent: *
Disallow: /search-v2/

User-agent: *
Disallow: /newsletter-api/

User-agent: *
Disallow: /whatsapp-api/

User-agent: *
Disallow: /journalism-spectrum/

User-agent: *
Disallow: /cancelReason/

User-agent: *
Disallow: /cm-backend/

User-agent: *
Disallow: /nl-server-backend/

History

Changes recorded since June 2026.

No change recorded.

In context

India
25.0% block at least one AI crawler
Behind Cloudflare
Yes