CrawlerPulse
CrawlerPulse / Europe / Belarus / aif.by

Argumenty i Fakty

aif.by · Belarus · Argumenty i Fakty - Belarus · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Allowed
OpenAI
Allowed
Anthropic
Allowed
Perplexity
Allowed
Meta
Allowed
Common Crawl
Allowed
ByteDance
Allowed
Apple
Allowed
Amazon
Allowed

robots.txt

As read on 8 Oct 2026, 865 bytes. No rule names an AI crawler.

User-agent: *
Allow: /comments
Allow: /authors/
Disallow: /comments_ajx
Disallow: /form/offerpopupp
Disallow: /view/
Disallow: *_comments
Disallow: /auth$
Disallow: /admin
Disallow: /adminold
Disallow: /registration
Disallow: /profile
Disallow: /recovery
Disallow: /search
Disallow: /fb/
Disallow: /media
Disallow: /contestallusers
Disallow: /iframes
Disallow: /extraform/*/list
Disallow: /voting/index.php
Disallow: /i/
Disallow: /z2
Disallow: /scripts
Disallow: /images
Disallow: /*print
Disallow: /shop/esshop/cart
Disallow: /shop/esuser/register
Disallow: /shop/esuser/forgotpass
Disallow: /shop/archive
Disallow: /*rss_pap
Disallow: /all/hit
Disallow: /all/comment
Disallow: /tag/id/
Disallow: /docs
Disallow: /opinion
Clean-param: utm_source&utm_campaign&utm_medium&utm_referrer&utm_content&utm_term
Clean-param: source
Clean-param: id /
Clean-param: id&cid /

History

Changes recorded since June 2026.

No change recorded.

In context

Belarus
0.0% block at least one AI crawler
Behind Cloudflare
No