CrawlerPulse
CrawlerPulse / Asia / Japan / nhk.or.jp

nhkor

nhk.or.jp · Japan · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Blocked
OpenAI
Blocked
Anthropic
Blocked
Perplexity
Blocked
Meta
Blocked
Common Crawl
Blocked
ByteDance
Blocked
Apple
Blocked
Amazon
Allowed

robots.txt

As read on 8 Oct 2026, 2,873 bytes. 2 rule groups name AI crawlers; shown first.

User-agent: GPTBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: anthropic-ai
User-agent: cohere-ai
User-agent: CCBot
User-agent: ICC-Crawler
User-agent: Bytespider
User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: Meta-ExternalAgent
Disallow: /

User-agent: PetalBot
User-agent: baiduspider
User-agent: baiduimagespider
Disallow: /

User-agent: Googlebot-Image
User-agent: Googlebot-Video
Disallow: /
Disallow: /archives/k/
Disallow: /archives/r/
Disallow: /das/k/
Disallow: /das/r/
Disallow: /gendai/k/
Disallow: /gendai/r/
Disallow: /heart-net/k/
Disallow: /heart-net/r/
Disallow: /kenko/k/
Disallow: /kenko/r/
Disallow: /learning/k/
Disallow: /learning/r/
Disallow: /learning-blog/r/
Disallow: /news/k/
Disallow: /news/r/
Disallow: /nhkworld/k/
Disallow: /nhkworld/r/
Disallow: /politics/k/
Disallow: /politics/r/
Disallow: /recipe/k/
Disallow: /recipe/r/
Disallow: /shutoken/k/
Disallow: /shutoken/r/
Allow: /robots.txt
Allow: /favicon.ico
Allow: /*/lreport/
Allow: /archives/
Allow: /assets/
Allow: /common/
Allow: /das/
Allow: /gendai/
Allow: /heart-net/
Allow: /kenko/
Allow: /learning/
Allow: /learning-blog/
Allow: /news/
Allow: /nhkworld/
Allow: /politics/
Allow: /prog/
Allow: /program/
Allow: /recipe/
Allow: /shutoken/
Allow: /akita/info/
Allow: /aomori/info/
Allow: /chiba/info/
Allow: /fukui/info/
Allow: /fukuoka/info/
Allow: /fukushima/info/
Allow: /gifu/info/
Allow: /hiroshima/info/
Allow: /hokkaido/info/
Allow: /kagoshima/info/
Allow: /kanazawa/info/
Allow: /kitakyushu/info/
Allow: /kobe/info/
Allow: /kochi/info/
Allow: /kofu/info/
Allow: /kumamoto/info/
Allow: /kyoto/info/
Allow: /maebashi/info/
Allow: /matsue/info/
Allow: /matsuyama/info/
Allow: /mito/info/
Allow: /miyazaki/info/
Allow: /morioka/info/
Allow: /nagano/info/
Allow: /nagasaki/info/
Allow: /nagoya/info/
Allow: /nara/info/
Allow: /niigata/info/
Allow: /oita/info/
Allow: /okayama/info/
Allow: /okinawa/info/
Allow: /osaka/info/
Allow: /otsu/info/
Allow: /saga/info/
Allow: /saitama/info/
Allow: /sendai/info/
Allow: /shizuoka/info/
Allow: /takamatsu/info/
Allow: /tokushima/info/
Allow: /tottori/info/
Allow: /toyama/info/
Allow: /tsu/info/
Allow: /utsunomiya/info/
Allow: /wakayama/info/
Allow: /yamagata/info/
Allow: /yamaguchi/info/
Allow: /yokohama/info/

User-agent: Twitterbot
Disallow: /*/r/

User-agent: *
Disallow: /*.xml$
Disallow: /*.json$
Disallow: /*.cgi$
Disallow: /*.cgi?
Disallow: /*/api/
Disallow: /*/hc/
Disallow: /*/k/
Disallow: /*/r/
Disallow: /*/shv/
Disallow: /cgiblog/
Disallow: /cgisearch/
Disallow: /chronicle/
Disallow: /error/
Disallow: /hc-*
Disallow: /hensei/
Disallow: /saigai/*/dl/
Disallow: /saigai/*/dn/
Disallow: /saigai/*/ev/
Disallow: /saigai/*/ss/
Disallow: /saigai/f/
Disallow: /topepg/
Disallow: /toppage/errors/
Sitemap: https://www.nhk.or.jp/sitemap.xml

History

Changes recorded since June 2026.

No change recorded.

In context

Japan
32.2% block at least one AI crawler
Behind Cloudflare
No