CrawlerPulse
CrawlerPulse / Asia / Japan / businessinsider.jp

businessinsider.jp

businessinsider.jp · Japan · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Allowed
OpenAI
Blocked
Anthropic
Blocked
Perplexity
Allowed
Meta
Blocked
Common Crawl
Blocked
ByteDance
Blocked
Apple
Allowed
Amazon
Blocked

robots.txt

As read on 8 Oct 2026, 2,711 bytes. 9 rule groups name AI crawlers; shown first.

User-agent: GPTBot
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Timpibot
Disallow: /

User-agent: omgilibot
Disallow: /

User-agent: *
Disallow: /article/178160/
Disallow: /article/178092/
Disallow: /font/*
Disallow: /specialfeature/asset/brandchannel/font/*
Disallow: /prime_about/?ref_pageid=
Disallow: /prime_about/?_ptid=
Disallow: /member/?_ptid=
Disallow: /prime/login?articleId=
Disallow: /prime/reset.php
Disallow: /a/*
Disallow: /b/*
Disallow: /c/*
Disallow: /d/*
Disallow: /e/*
Disallow: /f/*
Disallow: /g/*
Disallow: /h/*
Disallow: /i/*
Disallow: /j/*
Disallow: /k/*
Disallow: /m/*
Disallow: /n/*
Disallow: /o/*
Disallow: /p/*
Disallow: /u/*
Disallow: /v/*
Disallow: /w/*
Disallow: /y/*
Disallow: /z/*
Disallow: /*?tblci=*
Disallow: /*?ss_ad_code=*
Disallow: /*?p=*
Disallow: /*?initialSessionID=*
Disallow: /*?ttclid=*
Disallow: /*?appsule=*
Disallow: /*?njclid=*
Disallow: /*?ref=*
Disallow: /*?__twitter_impression=*
Disallow: /*?utm_source=Business+Insider+Japan+Mailmagazine
Disallow: /*?amp;utm_medium=feed
Disallow: /*_ptid=*
Disallow: /series_list/*
Disallow: /en/
Disallow: /index_en
Disallow: /feed/index.xml*?d=
Disallow: /index.xml*?d=
Disallow: /*?preview=*
Disallow: /*?mode=assets
Disallow: /*?cx_click=sp_ranking
Disallow: /*?cx_click=pc_ranking
Disallow: /*?cx_click=ranking
Disallow: /*?itm_source=article_link
Disallow: /*?utm_source=dlvr.it
Disallow: /*?utm_source=voluntary
Disallow: /*?utm_source=pc_bottom_ranking
Disallow: /*?utm_source=sp_bottom_ranking
Disallow: /*?fbclid=*
Disallow: /asset/
Disallow: /assets/
Disallow: /assets_c/
Disallow: /upload_files/
Disallow: /api/
Disallow: /static/
Disallow: /article_type/
Disallow: /article_images/
Disallow: /search/
Disallow: /15055520/
Disallow: /article/[slug]
Disallow: /articles/[page]
Disallow: /articles/business/[page]
Disallow: /articles/life/[page]
Disallow: /articles/mashingup/[page]
Disallow: /articles/markets/[page]
Disallow: /articles/tech/[page]
Disallow: /author/[slug]
Disallow: /category/[slug]
Disallow: /feature/[slug]
Disallow: /prime_article/[page]
Disallow: /regular/[slug]
Disallow: /series/[slug]
Disallow: /series/[slug]/[page]
Disallow: /series/mirainotsukurite/people/[page]
Disallow: /video/[page]
Disallow: /specialtag/[slug]
Disallow: /tag/[slug]
Disallow: /tag/[slug]/[page]

User-agent: Mediapartners-Google
Allow: /

User-agent: AmazonAdBot
Allow: /

User-agent: *
Sitemap: https://www.businessinsider.jp/sitemap/index.xml
Sitemap: https://www.businessinsider.jp/feed/google_news_sitemap.xml

History

Changes recorded since June 2026.

No change recorded.

In context

Japan
32.2% block at least one AI crawler
Behind Cloudflare
No