CrawlerPulse
CrawlerPulse / Europe / Russia / kommersant.ru

Kommersant

kommersant.ru · Russia · measured 9 Oct 2026

Google Search
Allowed
Google-Extended
Allowed
OpenAI
Allowed
Anthropic
Allowed
Perplexity
Allowed
Meta
Allowed
Common Crawl
Allowed
ByteDance
Allowed
Apple
Allowed
Amazon
Allowed

robots.txt

As read on 9 Oct 2026, 4,198 bytes. No rule names an AI crawler.

User-agent: *
Disallow: /photo/buyphoto
Disallow: /rules
Disallow: /?
Disallow: /doc*=
Disallow: /*utm_
Disallow: /*from=
Disallow: /*click_
Disallow: /*captchaKey
Disallow: /lk
Disallow: /*search
Disallow: /*themeid=
Disallow: /doc-rss
Disallow: /Doc-rm
Disallow: /doc-rm
Disallow: /Doc-rss
Disallow: /Doc-y/
Disallow: /doc-y/
Disallow: /*title=
Disallow: /*pdocdenied
Disallow: /LK/NotAuthorized
Disallow: /archive/*?publishingId=152
Disallow: /*_escaped_fragment_=
Disallow: /*fp=
Disallow: /*themeID=
Disallow: /*isSearch=
Disallow: /fencing?date=
Disallow: /spief?date
Disallow: /bankruptcy/*page=
Disallow: *?small_mozaic=*
Disallow: /authors/*page=
Disallow: /gbox/
Disallow: /?trace=*
Disallow: /*?feed=*
Disallow: /classified
Disallow: /*&sortColumn=*
Disallow: /*news?date=*
Disallow: /company_news
Disallow: /*?hc=*
Disallow: /*?pg=*
Disallow: *?publishingId=*
Disallow: /Daily/
Disallow: /Vlast/
Disallow: /Money/
Disallow: /Autopilot/
Disallow: /Apps/
Disallow: /Weekend/
Disallow: /Sf/
Disallow: /Citizen_k/
Disallow: /Ogoniok/
Disallow: /Nauka/
Disallow: /tema/
Disallow: /FreeMoney/
Disallow: /FreeVlast/
Disallow: /reklama/
Disallow: /library
Disallow: /livejournal
Disallow: /articles/
Disallow: *twitter_impression*
Disallow: *doc.html*
Disallow: *IssueId*
Disallow: *NodesID*
Disallow: /pda/*
Disallow: *?intervaltype=*
Disallow: *&pageprms*
Disallow: *?id2=*
Disallow: *?regionid=*
Disallow: /*?*yaCaptchaResponse*
Disallow: /api/
Disallow: /doc/preview/
Disallow: /__nuxt_error
Disallow: /metrics
Disallow: /traces
Disallow: /logs
Allow: *?erid=*

User-agent: Yandex
Disallow: /photo/buyphoto
Disallow: /rules
Disallow: /?
Disallow: /*utm_
Disallow: /*click_
Disallow: /*captchaKey
Disallow: /lk
Disallow: /*search
Disallow: /*themeid=
Disallow: /doc-rss
Disallow: /Doc-rm
Disallow: /doc-rm
Disallow: /Doc-rss
Disallow: /Doc-y/
Disallow: /doc-y/
Disallow: /*title=
Disallow: /*pdocdenied
Disallow: /LK/NotAuthorized
Disallow: /archive/*?publishingId=152
Disallow: /*_escaped_fragment_=
Disallow: /*fp=
Disallow: /*themeID=
Disallow: /*isSearch=
Disallow: /fencing?date=
Disallow: /spief?date
Disallow: /bankruptcy/*page=
Disallow: *?small_mozaic=*
Disallow: /authors/*page=
Disallow: /gbox/
Disallow: /?trace=*
Disallow: /*?feed=*
Disallow: /classified
Disallow: /*&sortColumn=*
Disallow: /company_news
Disallow: /*?hc=*
Disallow: /*?pg=*
Disallow: *?publishingId=*
Disallow: /Daily/
Disallow: /Vlast/
Disallow: /Money/
Disallow: /Autopilot/
Disallow: /Apps/
Disallow: /Weekend/
Disallow: /Sf/
Disallow: /Citizen_k/
Disallow: /Ogoniok/
Disallow: /Nauka/
Disallow: /tema/
Disallow: /FreeMoney/
Disallow: /FreeVlast/
Disallow: /reklama/
Disallow: /library
Disallow: /livejournal
Disallow: /articles/
Disallow: *twitter_impression*
Disallow: *doc.html*
Disallow: *IssueId*
Disallow: *NodesID*
Disallow: /pda/*
Disallow: *?intervaltype=*
Disallow: *&pageprms*
Disallow: *?regionid=*
Disallow: /api/
Disallow: /doc/preview/
Disallow: /__nuxt_error
Disallow: /metrics
Disallow: /traces
Disallow: /logs
Clean-param: from
Clean-param: id&id2&date&intervaltype&page_id&m_id&objectid&picsid&stpid&stamp
Clean-param: newsid&DocsID /doc.aspx
Clean-param: id /doc.asp
Clean-param: intervaltype&date&docId&cal_open /archive/*
Clean-param: w&nw&tw&tg&source&fbclid&sphrase_id&query&mod&tt&zen&vk&ref&p /doc/*
Clean-param: docsid /private/pdoc
Clean-param: date&Id_doc&cal_open&year&issue&id&section /regions/*
Clean-param: old /speaker/*
Clean-param: docId&issueId /doc-u.html
Clean-param: amp_lite
Clean-param: bx_sender_conversion_id
Clean-param: yaCaptchaResponse
Sitemap: https://www.kommersant.ru/sitemaps/all_docs_sitemap.xml
Sitemap: https://www.kommersant.ru/sitemaps/index_sitemap.xml
Sitemap: https://www.kommersant.ru/sitemaps/news.xml
Sitemap: https://www.kommersant.ru/sitemaps/authors.xml
Sitemap: https://www.kommersant.ru/sitemaps/themes.xml

User-agent: ltx71 - (http://ltx71.com/)
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: SemrushBot
Disallow: /

User-agent: TelegramBot
Allow: /

History

Changes recorded since June 2026.

No change recorded.

In context

Russia
6.3% block at least one AI crawler
Behind Cloudflare
No