businessinsider.com.pl · United States · measured 9 Oct 2026
As read on 9 Oct 2026, 812 bytes. 9 rule groups name AI crawlers; shown first.
User-agent: cohere-ai Disallow: / User-agent: meta-externalagent Disallow: / User-agent: Amazonbot Disallow: / User-agent: Bytespider Disallow: / User-agent: Timpibot Disallow: / User-agent: omgilibot Disallow: / User-agent: Diffbot Disallow: / User-agent: Diffbot-User Disallow: / User-agent: YouBot Disallow: / User-agent: * Disallow: /*?*sort= Disallow: /*&*sort= Disallow: /user-session-proxy/* Disallow: /paywall* Disallow: /*?fbclid= Disallow: /*?fb_comment_id= Disallow: /szukaj/ Disallow: /subskrypcja? Disallow: /__acc/ Disallow: /_cdf/ Disallow: /_variant/ Disallow: /a8f4d8cd95e164917035b64b867a45dd User-agent: Ai2Bot-Dolma Disallow: / User-agent: Amzn-User Disallow: / User-agent: DataForSeoBot Disallow: / User-agent: Amzn-SearchBot Disallow: / User-agent: meta-webindexer Disallow: /
Changes recorded since June 2026.
No change recorded.