CrawlerPulse
CrawlerPulse / Asia / Bahrain / bahrainmirror.com

bahrainmirror.com

bahrainmirror.com · Bahrain · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Allowed
OpenAI
Allowed
Anthropic
Allowed
Perplexity
Allowed
Meta
Allowed
Common Crawl
Allowed
ByteDance
Blocked
Apple
Blocked
Amazon
Allowed

robots.txt

As read on 8 Oct 2026, 1,490 bytes. 1 rule group name AI crawlers; shown first.

User-agent: Bytespider
Disallow: /

User-Agent: *
Disallow: /tags/

User-Agent: SiteLock
Disallow:/

User-agent: A6-Indexer
Disallow: /

User-agent: AlphaSeoBot
Disallow: /

User-agent: AlphaSeoBot-SA
Disallow: /

User-agent: Applebot
Disallow: /

User-agent: AspiegelBot
Disallow: /

User-agent: bingbot/2.0
Disallow: /

User-agent: Blackboard Safeassign
Disallow: /

User-agent: BLEXBot/1.0
Disallow: /

User-agent: crawler4j
Disallow: /

User-agent: DotBot
Disallow: /

User-agent: Gigabot
Disallow: /

User-agent: LieBaoFast
Disallow: /

User-agent: MauiBot
Disallow:/

User-agent: MauiBot (crawler.feedback+wc@gmail.com)
Disallow: /

User-agent: MegaIndex.ru/2.0
Disallow: /

User-agent: MqqBrowser
Disallow:/

User-agent: Nimbostratus-Bot/v1.3.2
Disallow: /

User-agent: Seekport Crawler
Disallow: /

User-agent: SemrushBot
Disallow: /

User-agent: SemrushBot-SA
Disallow: /

User-agent: SeznamBot
Disallow: /

User-agent: SputnikBot/2.3
Disallow:/

User-agent: The Knowledge AI
Disallow:/

User-agent: TurnitinBot
Disallow: /

User-agent: UCBrowser
Disallow: /

User-agent: yacybot
Disallow: /

User-agent: Yeti
Disallow: /

User-agent: YisouSpider
Disallow: /

History

Changes recorded since June 2026.

No change recorded.

In context

Bahrain
8.3% block at least one AI crawler
Behind Cloudflare
No