yomiuri.co.jp · Japan · The Yomiuri Shimbun · measured 8 Oct 2026
As read on 8 Oct 2026, 3,619 bytes. 5 rule groups name AI crawlers; shown first.
User-agent: AI2Bot User-agent: Ai2Bot-Dolma User-agent: AmazonBot User-agent: Applebot-Extended User-agent: Brightbot 1.0 User-agent: Bytespider User-agent: CCBot User-agent: cohere-ai User-agent: cohere-training-data-crawler User-agent: Crawlspace User-agent: curl User-agent: Diffbot User-agent: Fess User-agent: FriendlyCrawler User-agent: Gensparkbot User-agent: GoogleOther User-agent: ia_archiver User-agent: iaskspider/2.0 User-agent: ICC-Crawler User-agent: ImagesiftBot User-agent: img2dataset User-agent: indexpert User-agent: ISSCyberRiskCrawler User-agent: Java User-agent: Kangaroo Bot User-agent: libwww User-agent: libwww-perl User-agent: magpie-crawler User-agent: Mechanize User-agent: Megalodon User-agent: meta-webindexer User-agent: NovaAct User-agent: Nutch User-agent: omgili User-agent: omgilibot User-agent: OpenindexSpider User-agent: Operator User-agent: PanguBot User-agent: Perplexity-ai User-agent: Perplexity-User User-agent: PerplexityBot User-agent: SBIntuitionsBot User-agent: SBIntuitions-SearchBot User-agent: Scrapy User-agent: SemrushBot-OCOB User-agent: SemrushBot-SWA User-agent: Sidetrade indexer bot User-agent: Timpibot User-agent: VelenPublicWebCrawler User-agent: Webzio-Extended User-agent: wget User-agent: YouBot Disallow: / User-agent: anthropic-ai User-agent: ChatGPT-User User-agent: Claude-SearchBot User-agent: Claude-User User-agent: Claude-Web User-agent: ClaudeBot User-agent: GPTBot User-agent: FacebookBot Disallow: / Allow: /adv/ Allow: /sponsored/ User-agent: OAI-SearchBot Disallow: / Allow: /teen/ Allow: /kyoiku/support Allow: /adv/ Allow: /sponsored/ User-agent: Google-Extended Disallow: / Allow: /adv/ Allow: /news Allow: /sponsored/ Allow: /*2026 Disallow: /*result*2 Disallow: /*2*result Disallow: /assets/*2 Disallow: /bosai-nippon Disallow: /comic/*2 Disallow: /coupon/*2 Disallow: /crossword Disallow: /culture/book/reviews Disallow: /culture/book/columns Disallow: /culture/hochi Disallow: /culture/hulu Disallow: /culture/nagaieye Disallow: /fukayomi/toyokeizai Disallow: /giants/archive Disallow: /hakone-ekiden/history Disallow: /hobby/atcars/impression Disallow: /hobby/atcars/info Disallow: /hobby/travel/ryokou-select Disallow: /igoshougi/kisei/kansenki Disallow: /igoshougi/ryuoh/kansenki Disallow: /images/*2 Disallow: /kodomo/sheet/ Disallow: /kyoiku/support/ranger Disallow: /life/food-column Disallow: /life/horoscope Disallow: /life/online Disallow: /life/recipe Disallow: /market/*202 Disallow: /media/*202 Disallow: /medical/brain Disallow: /medical/men Disallow: /neko-pitcher Disallow: /novel/*2 Disallow: /nyushi/*2 Disallow: /otekomachi Disallow: /photograph/kyoshitsu Disallow: /photograph/photogp Disallow: /photograph/ypc Disallow: /photograph/ypcex Disallow: /pluralphoto Disallow: /r/ims/*20 Disallow: /s/ims/*20 Disallow: /serial/earth Disallow: /serial/webkiryu Disallow: /shimen/*20 Disallow: /slides/*20 Disallow: /slitherlink Disallow: /stream/*20 Disallow: /sudoku/*20 Disallow: /weather/*20 Disallow: /world/nieman Disallow: /ybooks Disallow: /yomidr/*2 Disallow: /yomiuri333 User-agent: meta-externalagent User-agent: meta-externalfetcher Disallow: / Allow: /teen/ Allow: /kyoiku/support Allow: /adv/ Allow: /sponsored/ User-agent: AmazonAdBot Allow: / User-agent: facebookexternalhit Disallow: / Allow: /teen/ Allow: /kyoiku/support User-agent: facebookcatalog Disallow: / Allow: /teen/ Allow: /kyoiku/support User-agent: bingbot Disallow: /web-search Disallow: /bosai-nippon/search User-agent: * Disallow: /wp-admin/ Disallow: /adv/*.swf$ Disallow: /member/ Sitemap: https://www.yomiuri.co.jp/sitemap.xml
Changes recorded since June 2026.
No change recorded.