socialmediatoday.com · United States · measured 9 Oct 2026
As read on 9 Oct 2026, 4,954 bytes. 4 rule groups name AI crawlers; shown first.
User-agent: PetalBot Disallow: / User-Agent: omgilibot Disallow: / User-Agent: omgili Disallow: / User-agent: Amazonbot Crawl-Delay: 10 Disallow: /admin/ Disallow: /newsletter/ Disallow: /healthcheck/ Disallow: /subpage/ Disallow: /ckeditor/ Disallow: /api/ Disallow: /static/images/ Disallow: /subscriber/ Disallow: /selfservice/ Disallow: /press-release/preview/ Disallow: /press-release/editor/ Disallow: /search/* Allow: /search/$ Disallow: /user_media/ Disallow: /imgproxy/ Disallow: /topic/ Disallow: /editors/ Disallow: /signup/* Allow: /signup/$ User-agent: * Crawl-delay: 5 Disallow: /admin/ Disallow: /newsletter/ Disallow: /healthcheck/ Disallow: /subpage/ Disallow: /ckeditor/ Disallow: /api/ Disallow: /static/images/ Disallow: /subscriber/ Disallow: /selfservice/ Disallow: /press-release/preview/ Disallow: /press-release/editor/ Disallow: /search/* Allow: /search/$ Disallow: /user_media/ Disallow: /imgproxy/ Disallow: /topic/?page=* Allow: /sitemap.xml User-agent: Twitterbot Crawl-delay: 5 Disallow: User-agent: Googlebot-News Disallow: /admin/ Disallow: /newsletter/ Disallow: /healthcheck/ Disallow: /subpage/ Disallow: /ckeditor/ Disallow: /api/ Disallow: /static/images/ Disallow: /subscriber/ Disallow: /selfservice/ Disallow: /press-release/preview/ Disallow: /press-release/editor/ Disallow: /search/* Allow: /search/$ Allow: /google_news_sitemap.xml Allow: /user_media/ Allow: /imgproxy/ Allow: /static/images/favicons User-agent: Googlebot-Image Disallow: / Allow: /imgproxy/ Allow: /static/images/favicons Allow: /favicon.ico Allow: /white-favicon.ico Allow: /apple-touch-icon.png Allow: /favicon-32x32.png Allow: /white-favicon-32x32.png Allow: /favicon-16x16.png Allow: /white-favicon-16x16.png Allow: /site.webmanifest Allow: /safari-pinned-tab.svg Allow: /browserconfig.xml Allow: /android-chrome-144x144.png Allow: /mstile-150x150.png User-agent: sentibot Disallow: / User-agent: claritybot Disallow: / User-agent: facebookexternalhit/1.1 Disallow: User-agent: AhrefsBot Crawl-Delay: 30 Disallow: /admin/ Disallow: /newsletter/ Disallow: /healthcheck/ Disallow: /subpage/ Disallow: /ckeditor/ Disallow: /api/ Disallow: /static/images/ Disallow: /subscriber/ Disallow: /selfservice/ Disallow: /press-release/preview/ Disallow: /press-release/editor/ Disallow: /search/* Allow: /search/$ Disallow: /user_media/ Disallow: /imgproxy/ Disallow: /topic/ Disallow: /editors/ User-agent: amazon-kendra # Amazon Kendra Web Crawler Disallow: /
Changes recorded since June 2026.
No change recorded.