CrawlerPulse
CrawlerPulse / Europe / Russia / mskagency.ru

Moscow Municipal News Agency

mskagency.ru · Russia · measured 9 Oct 2026

Google Search
Allowed
Google-Extended
Allowed
OpenAI
Allowed
Anthropic
Allowed
Perplexity
Allowed
Meta
Allowed
Common Crawl
Allowed
ByteDance
Allowed
Apple
Allowed
Amazon
Allowed

robots.txt

As read on 9 Oct 2026, 1,593 bytes. No rule names an AI crawler.

User-agent: *
Sitemap: https://www.mskagency.ru/sitemap.xml
Allow: /
Disallow: /*?rubric=
Disallow: /*&rubric=
Disallow: /*?criteria=
Disallow: /*&criteria=
Disallow: /*?photographer_id=
Disallow: /*&photographer_id=
Disallow: /*?important=
Disallow: /*&important=
Disallow: /*?tag=
Disallow: /*&tag=
Disallow: /*?from=
Disallow: /*&from=
Disallow: /*?to=
Disallow: /*&to=
Disallow: /suggest
Disallow: /client_search_filter
Disallow: /search
Disallow: /rss/web2
Disallow: /photobank/list
Crawl-delay: 5
Request-rate: 1/5

User-agent: Yandex
Allow: /
Disallow: /*?rubric=
Disallow: /*&rubric=
Disallow: /*?criteria=
Disallow: /*&criteria=
Disallow: /*?photographer_id=
Disallow: /*&photographer_id=
Disallow: /*?important=
Disallow: /*&important=
Disallow: /*?tag=
Disallow: /*&tag=
Disallow: /*?from=
Disallow: /*&from=
Disallow: /*?to=
Disallow: /*&to=
Disallow: /suggest
Disallow: /search
Disallow: /client_search_filter
Disallow: /photobank/list
Clean-param: block_mode&page_size&important&is_exclusive&mgclid&hcb&pr&utm_source&ref&utm_medium&utm_campaign&ampcf&source&back&nid&grs&_ga&from&_gl&attempt&ShareMaterialWin
Crawl-delay: 5

User-agent: Googlebot
Allow: /
Disallow: /*?rubric=
Disallow: /*&rubric=
Disallow: /*?criteria=
Disallow: /*&criteria=
Disallow: /*?photographer_id=
Disallow: /*&photographer_id=
Disallow: /*?important=
Disallow: /*&important=
Disallow: /*?tag=
Disallow: /*&tag=
Disallow: /*?from=
Disallow: /*&from=
Disallow: /*?to=
Disallow: /*&to=
Disallow: /suggest
Disallow: /search
Disallow: /photobank/list
Disallow: /client_search_filter
Host: www.mskagency.ru

History

Changes recorded since June 2026.

No change recorded.

In context

Russia
6.3% block at least one AI crawler
Behind Cloudflare
No