buzzfeed.com · United States · measured 8 Oct 2026
As read on 8 Oct 2026, 4,496 bytes. 19 rule groups name AI crawlers; shown first.
User-agent: Amazonbot Allow: /rss-feeds/feeds/bf-amazon-ai-grounding.extended.xml Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: CCBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Claude-Web Disallow: / User-agent: cohere-ai Disallow: / User-agent: Diffbot Disallow: / User-agent: FacebookBot Disallow: / User-agent: google-extended Disallow: / User-agent: Google-Extended Disallow: / User-agent: GPTBot Disallow: / User-agent: PerplexityBot Disallow: / User-agent: meta-externalagent Disallow: / User-agent: Bytespider Disallow: / User-agent: Timpibot Disallow: / User-agent: omgilibot Disallow: / User-agent: TtoYouBot Disallow: / User-agent: WebGPTBot Disallow: / User-agent: msnbot Crawl-delay: 120 Disallow: /*.xml$ Disallow: /buzz/*.xml$ Disallow: /category/*.xml$ Disallow: /mobile/ Disallow: *?s=mobile Disallow: *?s=bpage-next Disallow: *?s=lightbox Disallow: /fabordrab/ Disallow: /bfmp/ Disallow: /buzzfeed/ Disallow: /contest Disallow: /contests Disallow: /plugin/ Disallow: /embed/ Disallow: /_comments/ Disallow: /bookmarks/ Disallow: /drafts/ User-agent: * Disallow: /buzz/*.xml$ Disallow: /category/*.xml$ Disallow: /mobile/ Disallow: *?s=bpage-next Disallow: *?s=lightbox Disallow: *?s=feedpager Disallow: /fabordrab/ Disallow: /bfmp/ Disallow: /buzzfeed/ Disallow: /contest Disallow: /contests Disallow: /_ga/ Disallow: /static/ Disallow: /dashboard/ Disallow: /plugin/ Disallow: /api/ Disallow: /buzzfeed/api/ Disallow: /embed/ Disallow: /_comments/ Disallow: /bookmarks/ Disallow: /drafts/ Disallow: /bpage-preview/ Disallow: /cms_preview_ui/ Disallow: /cms_preview_ui_static/ Disallow: /search/ User-agent: discobot Disallow: / User-agent: dotbot Disallow: / User-agent: Googlebot Disallow: /comments-api/ Disallow: /content-reactions-api/ User-agent: TurnitinBot Disallow: / User-agent: magpie-crawler Disallow: / User-agent: Meta-ExternalFetcher Disallow: / User-agent: DuckAssistBot Disallow: / User-agent: Perplexity-User Disallow: / User-agent: Ai2Bot-Dolma Disallow: / User-agent: Slurp Crawl-delay: 4 User-agent: Verity/1.1 Crawl-delay: 20 Allow: /$ Allow: /* Disallow: /*.xml$ Disallow: /buzz/*.xml$ Disallow: /category/*.xml$ Disallow: /mobile/ Disallow: *?s=mobile Disallow: *?s=bpage-next Disallow: *?s=lightbox Disallow: *?s=feedpager Disallow: /fabordrab/ Disallow: /bfmp/ Disallow: /buzzfeed/ Disallow: /contest Disallow: /contests Disallow: /_ga/ Disallow: /static/ Disallow: /dashboard/ Disallow: /plugin/ Disallow: /api/ Disallow: /buzzfeed/api/ Disallow: /embed/ Disallow: /_comments/ Disallow: /bookmarks/ Disallow: /drafts/ Disallow: /bpage-preview/ Disallow: /cms_preview_ui/ Disallow: /cms_preview_ui_static/ Disallow: /search/ User-agent: Scope3/2.0 (scope3.com) Allow: /$ Allow: /* Disallow: /*.xml$ Disallow: /buzz/*.xml$ Disallow: /category/*.xml$ Disallow: /mobile/ Disallow: *?s=mobile Disallow: *?s=bpage-next Disallow: *?s=lightbox Disallow: *?s=feedpager Disallow: /fabordrab/ Disallow: /bfmp/ Disallow: /buzzfeed/ Disallow: /contest Disallow: /contests Disallow: /_ga/ Disallow: /static/ Disallow: /dashboard/ Disallow: /plugin/ Disallow: /api/ Disallow: /buzzfeed/api/ Disallow: /embed/ Disallow: /_comments/ Disallow: /bookmarks/ Disallow: /drafts/ Disallow: /bpage-preview/ Disallow: /cms_preview_ui/ Disallow: /cms_preview_ui_static/ Disallow: /search/ User-agent: AmazonAdBot Allow: /$ Allow: /* Disallow: /*.xml$ Disallow: /buzz/*.xml$ Disallow: /category/*.xml$ Disallow: /mobile/ Disallow: *?s=bpage-next Disallow: *?s=lightbox Disallow: *?s=feedpager Disallow: /fabordrab/ Disallow: /bfmp/ Disallow: /buzzfeed/ Disallow: /contest Disallow: /contests Disallow: /_ga/ Disallow: /static/ Disallow: /dashboard/ Disallow: /plugin/ Disallow: /api/ Disallow: /buzzfeed/api/ Disallow: /embed/ Disallow: /_comments/ Disallow: /bookmarks/ Disallow: /drafts/ Disallow: /bpage-preview/ Disallow: /cms_preview_ui/ Disallow: /cms_preview_ui_static/ Disallow: /search/ Sitemap: https://www.buzzfeed.com/sitemap/buzzfeed.xml Sitemap: https://www.buzzfeed.com/sitemap/tasty.xml Sitemap: https://www.buzzfeed.com/sitemap/video.xml Sitemap: https://www.buzzfeed.com/sitemap/shopping.xml Sitemap: https://www.buzzfeed.com/sitemap/news.xml Sitemap: https://www.buzzfeed.com/sitemap/buzzfeed-community.xml Sitemap: https://www.buzzfeed.com/sitemap/editions.xml Sitemap: https://www.buzzfeed.com/sitemap/news-archive.xml User-agent: yacybot Disallow: /
Changes recorded since June 2026.
No change recorded.