CrawlerPulse
CrawlerPulse / Asia / Lebanon / newsnow.co.uk

NewsNow: World news

newsnow.co.uk · Lebanon · NewsNow: World news | Every Source, Every Five Minutes, 24/7 news · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Allowed
OpenAI
Blocked
Anthropic
Allowed
Perplexity
Allowed
Meta
Allowed
Common Crawl
Allowed
ByteDance
Allowed
Apple
Allowed
Amazon
Allowed

robots.txt

As read on 8 Oct 2026, 827 bytes. 3 rule groups name AI crawlers; shown first.

User-agent: anthropic-ai
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: PetalBot
Disallow: /classifieds/

User-agent: AhrefsBot
Disallow: /

User-agent: SemrushBot
Disallow: /

User-agent: AmazonAdBot
Disallow: /cgi-bin
Disallow: /livefeed
Disallow: /brand-new-look.html
Disallow: /reg/*
Disallow: /housead*
Disallow: /http:*
Disallow: /https:*
Disallow: /ico/1.gif
Disallow: /pharos.js*

User-agent: *
Disallow: /h/*?p=
Disallow: /h/*&p=
Disallow: /cgi-bin
Disallow: /livefeed
Disallow: /A
Disallow: /share
Disallow: /cgi/NGoto
Disallow: /brand-new-look.html
Disallow: /reg/*
Disallow: /housead*
Disallow: /http:*
Disallow: /https:*
Disallow: /ico/1.gif
Disallow: /pharos.js*
Disallow: /test-please-ignore/

History

Changes recorded since June 2026.

No change recorded.

In context

Lebanon
8.7% block at least one AI crawler
Behind Cloudflare
No