khaleejtimes.com · United Arab Emirates · measured 8 Oct 2026
As read on 8 Oct 2026, 3,698 bytes. 6 rule groups name AI crawlers; shown first.
User-agent: Amazonbot Disallow: / User-agent: meta-externalagent Crawl-delay: 10 Disallow: /wp-admin/ Disallow: /wp-content/ Disallow: /wp-includes/ Disallow: /adlogs/ Disallow: /error/ Disallow: /tmp/ Disallow: /logs/ Disallow: /misc/misc/ Disallow: /*nocache=1 Disallow: /*updfcache=1 Disallow: /*?_refresh= Disallow: /*&_refresh= Disallow: /*?burstcache= Disallow: /*&burstcache= User-agent: Google-Extended Disallow: / User-agent: ClaudeBot Disallow: / User-agent: anthropic-ai Disallow: / User-agent: AppleBot-Extended Disallow: / Sitemap: https://www.khaleejtimes.com/sitemap.xml Sitemap: https://www.khaleejtimes.com/news_sitemap.xml User-agent: meta-webindexer Disallow: /*?_refresh= Disallow: /*&_refresh= Disallow: /*?burstcache= Disallow: /*&burstcache= User-agent: meta-externalads Crawl-delay: 10 Disallow: /wp-admin/ Disallow: /wp-content/ Disallow: /wp-includes/ Disallow: /adlogs/ Disallow: /error/ Disallow: /tmp/ Disallow: /logs/ Disallow: /misc/misc/ Disallow: /*nocache=1 Disallow: /*updfcache=1 Disallow: /*?_refresh= Disallow: /*&_refresh= Disallow: /*?burstcache= Disallow: /*&burstcache= Disallow: /*?*web_view= Disallow: /*?*utm_campaign= User-agent: facebookexternalhit Crawl-delay: 10 Disallow: /wp-admin/ Disallow: /wp-content/ Disallow: /wp-includes/ Disallow: /adlogs/ Disallow: /error/ Disallow: /tmp/ Disallow: /logs/ Disallow: /misc/misc/ Disallow: /*nocache=1 Disallow: /*updfcache=1 Disallow: /*?_refresh= Disallow: /*&_refresh= Disallow: /*?burstcache= Disallow: /*&burstcache= Disallow: /*?*web_view= Disallow: /*?*utm_campaign= User-agent: * Disallow: /wp-admin/ Disallow: /wp-content/* Disallow: /wp-includes/ Disallow: /sa-preview/* Disallow: /*nocache=1 Disallow: /*Nocache=1 Disallow: /*NoCache=1 Disallow: /*noCache=1 Disallow: /*updfcache=1 Disallow: /adlogs/ Disallow: /error/ Disallow: /red/ Disallow: /aspnet_client/ Disallow: /tmp/ Disallow: /logs/ Disallow: /.cache/ Disallow: /.localcache/ Disallow: /mal/ Disallow: /templates/ Disallow: /front-new-nf/ Disallow: /apps/pbcs.dll/classifieds Disallow: /apps/pbcs.dll/ptshowguide Disallow: /apps/pbcs.dll/events Disallow: /apps/pbcs.dll/index Disallow: /apps/pbcs.dll/news Disallow: /apps/pbcs.dll/news_message Disallow: /apps/pbcs.dll/news_messages Disallow: /apps/pbcs.dll/temaoversikt Disallow: /apps/pbcs.dll/related Disallow: /apps/pbcs.dll/misc Disallow: /apps/pbcs.dll/error Disallow: /apps/pbcs.dll/search Disallow: /apps/pbcs.dll/status Disallow: /apps/pbcsedit.dll Disallow: /apps/pbcsad.dll Disallow: /apps/rub.dll Disallow: /apps/pbcs.dll/section Disallow: /misc/misc/ArticleVote.pbs Disallow: /search Disallow: /apps/pbcs.dll/UserComments Disallow: /misc/misc/SubmitUserComments.pbs Disallow: /search? Disallow: /misc/misc/* Disallow: /apps/pbcs.dll/* Disallow: /search& Disallow: /search&content Disallow: /*search&content Disallow: /*search&content&datefilter&text Disallow: /*misc/misc/ArticleVote.pbs&nocache Disallow: /*search&facet.filter= Disallow: /*search&&pagenumber Disallow: /*search&&facet.filter Disallow: /*search&content&datefilter&text* Disallow: /*misc/misc/ArticleVote.pbs&nocache* Disallow: /*search&facet.filter* Disallow: /*search&&pagenumber* Disallow: /*search&&facet.filter* Disallow: /*?_gl= User-agent: Gemini-Deep-Research Disallow: /
Changes recorded since June 2026.
No change recorded.