CrawlerPulse
CrawlerPulse / Asia / Jordan / menafn.com

MENAFN

menafn.com · Jordan · measured 9 Oct 2026

Google Search
Allowed
Google-Extended
Allowed
OpenAI
Allowed
Anthropic
Allowed
Perplexity
Allowed
Meta
Allowed
Common Crawl
Allowed
ByteDance
Allowed
Apple
Allowed
Amazon
Allowed

robots.txt

As read on 9 Oct 2026, 993 bytes. 4 rule groups name AI crawlers; shown first.

User-agent: OAI-SearchBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: GPTBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: *
Disallow: /ckeditor/
Disallow: /_FTP_folders/
Disallow: /AFPArchive/
Disallow: /alerts/
Disallow: /comingSoun/
Disallow: /mgmt/
Disallow: /mgmt/mgmtnet/
Disallow: /mgmt/newAutofeed/FeedNewsImport
Disallow: /snap/
Disallow: /Lexis/
Disallow: /includes/in_banner_hits.asp
Disallow: /includes/
Disallow: /FreshSite/

User-agent: Googlebot
Disallow: /nogooglebot/

User-agent: *
Allow: /

User-agent: *
sitemap: https://menafn.com/updates/sitemaps/sitemap_index.xml
sitemap: https://menafn.com/updates/sitemaps/sitemap.xml
sitemap: https://menafn.com/updates/sitemaps/sitemap_news_en.xml
Sitemap: https://menafn.com/updates/sitemaps/sitemap_index_arabic.xml
sitemap: https://menafn.com/updates/sitemaps/sitemap_arabic.xml
sitemap: https://menafn.com/updates/sitemaps/sitemap_news_ar.xml

History

Changes recorded since June 2026.

No change recorded.

In context

Jordan
8.3% block at least one AI crawler
Behind Cloudflare
No