abcnews.go.com · United States · measured 8 Oct 2026
As read on 8 Oct 2026, 1,754 bytes. 9 rule groups name AI crawlers; shown first.
User-agent: GPTBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: CCBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Omgilibot Disallow: / User-agent: Omgili Disallow: / User-agent: FacebookBot Disallow: / User-agent: Bytespider Disallow: / Sitemap: https://abcnews.go.com/xmap Sitemap: https://abcnews.go.com/xmlLatestStories Sitemap: https://abcnews.go.com/xmlLatestVideos User-agent: claritybot Disallow: / User-agent: Googlebot-News Disallow: / User-agent: * Disallow: /search?searchtext=* Disallow: /disneyid/* Disallow: /assets/static/ads/* Disallow: /cgi Disallow: /xls Disallow: /imp Disallow: /kmail Disallow: /map Disallow: /log Disallow: /gif Disallow: /panel Disallow: /0/ Disallow: /promo/ Disallow: /abclinks/ Disallow: /houseads/ Allow: /xmldata/mrss Allow: /xmldata/rss Allow: /xmldata/xmlPodcast Allow: /xmldata/config Allow: /xmldata/feed Disallow: /sendtofriend/ Allow: /meta/sitemap Disallow: /meta/ Disallow: /staging/ Disallow: /test/ Disallow: /swen/ Disallow: /intro/ Disallow: /go/ Disallow: /news/go/ Disallow: /widgets/ Disallow: /vp2/ Disallow: /Video/*playerIndex Disallow: /*carousel/ Disallow: /*videoLogin? Disallow: /video/browse/ Disallow: /*popup? Disallow: /alerts-v1/ Disallow: /not-allowed/ Disallow: /beta-story-container/* Disallow: /video/embed/* Disallow: /video/amp/embed/* Disallow: /responder Disallow: /video/itemfeed* Disallow: /video/caption* Disallow: /content/news/liveBlog/* Disallow: /configuration/* Disallow: /rsidxfeed/* Disallow: /xmldata/* Disallow: /gma/api/* Disallow: /proxy/* User-agent: Applebot Allow: / Disallow: /private/
Changes recorded since June 2026.
No change recorded.