npr.org · United States · measured 8 Oct 2026
As read on 8 Oct 2026, 4,736 bytes. 16 rule groups name AI crawlers; shown first.
User-agent: GPTBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: CCBot Disallow: / User-agent: OAI-SearchBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Bytespider Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Claude-Web Disallow: / User-agent: cohere-ai Disallow: / User-agent: Diffbot Disallow: / User-agent: FacebookBot Disallow: / User-agent: omgili Disallow: / User-agent: omgilibot Disallow: / User-agent: PerplexityBot Disallow: / User-agent: * Disallow: /mpx/ Disallow: /cgi-bin Disallow: /ramfiles/ Disallow: /oauth2/ Disallow: /account/ Disallow: /proxy/ Disallow: /*.smil Disallow: /*.asx Disallow: /*.ram Disallow: /*.wav Disallow: /*.rmm Disallow: /*.js Disallow: /*.au Disallow: /stations/force/force_localization.php? Disallow: /rundowns/segment.php? Disallow: /templates/search/* Disallow: /2013/03/21/174840895/ Disallow: /sections/ombudsman/2008/01/frequently_asked_questions_1.html Disallow: /sections/health-shots/2013/03/11/173816690/new-voices-for-the-voiceless-synthetic-speech-gets-an-upgrade Disallow: /transcripts/470280334* Disallow: /2015/07/04/419570939/chasing-memories-in-their-refugee-camp-40-years-after-they-fled-vietnam Disallow: /transcripts/419570939* Disallow: /sections/parallels/2016/08/15/480128005/for-french-teens-smoking-still-has-more-allure-than-stigma Disallow: /transcripts/480128005* Disallow: /2020/04/08/830237502/episode-989-what-if-no-one-pays-rent Disallow: /transcripts/830237502* Disallow: /sections/goatsandsoda/2015/06/09/406744975/a-gender-revolution-hits-the-streets-two-wheels-at-a-time Disallow: /transcripts/406744975* Disallow: /sureroute Disallow: /*/partials* Disallow: /*?* Disallow: /proxy/* Disallow: /player/* Disallow: /get/* Disallow: /geolocation User-agent: Claude-SearchBot Disallow: / User-agent: Claude-User Disallow: / User-agent: PerplexityUser Disallow: / User-agent: Google-InspectionTool Allow: / User-agent: Google-Image Allow: / User-agent: Google-Video Allow: / User-agent: Googlebot Allow: / User-agent: TTD-Content Allow: / User-agent: Trade Desk ads.txt & sellers.json crawler Allow: / User-agent: AdsBot-Google Allow: / User-agent: AdsBot-Google-Mobile Allow: / User-agent: Mediapartners-Google Allow: / User-agent: Google-Display-Ads-Bot Allow: / User-agent: IAB-Tech-Lab Allow: / User-agent: ias_crawler Allow: / User-agent: ias_wombles Allow: / User-agent: Amazon-Advertising-ad-standards-bot/1.0 Allow: / User-agent: APS-ad-standards-bot/1.0 Allow: / User-agent: AmazonAdBot Allow: / User-agent: AudigentAdBot Allow: / User-agent: Concert/1.0 Allow: / User-agent: Centro Allow: / User-agent: bidswitchbot/1.0 Allow: / User-agent: DataXu/1.0 Allow: / User-agent: PubMatic Allow: / User-agent: IAB ATQ team adform robot site.adform.com/general/url-fetcher Allow: / User-agent: StackAdapt Allow: / User-agent: Quantcastbot/1.0 Allow: / User-agent: Quantcastbot/2.0 Allow: / User-agent: facebookexternalhit Allow: / Sitemap: https://googlecrawl.npr.org/standard/sitemap_index.xml Sitemap: https://googlecrawl.npr.org/news/sitemap_news.xml Sitemap: https://googlecrawl.npr.org/video/sitemap_index.xml Sitemap: https://www.npr.org/live-updates/sitemap.xml
Changes recorded since June 2026.
No change recorded.