CrawlerPulse
CrawlerPulse / Asia / Japan / gizmodo.jp

gizmodo.jp

gizmodo.jp · Japan · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Allowed
OpenAI
Blocked
Anthropic
Blocked
Perplexity
Allowed
Meta
Blocked
Common Crawl
Blocked
ByteDance
Blocked
Apple
Allowed
Amazon
Blocked

robots.txt

As read on 8 Oct 2026, 1,550 bytes. 9 rule groups name AI crawlers; shown first.

User-agent: GPTBot
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Timpibot
Disallow: /

User-agent: omgilibot
Disallow: /

User-agent: *
Disallow: /tag_detail/
Disallow: /mt/
Disallow: /*?__twitter_impression=true
Disallow: /font/*
Disallow: /feed/index.xml*?d=
Disallow: /feed/index.xml*?a=
Disallow: /index.xml*?d=
Disallow: /index.xml*?a=
Disallow: /*?preview=*
Disallow: /*?mode=assets
Disallow: /*?cx_click=sp_ranking
Disallow: /*?cx_click=pc_ranking
Disallow: /*?cx_click=ranking
Disallow: /*?itm_source=article_link
Disallow: /*?utm_source=dlvr.it
Disallow: /*?utm_source=voluntary
Disallow: /*?utm_source=pc_bottom_ranking
Disallow: /*?utm_source=sp_bottom_ranking
Disallow: /*?utm_source=sale-page
Disallow: /*?fbclid=*
Disallow: /asset/
Disallow: /assets/
Disallow: /assets_c/
Disallow: /upload_files/
Disallow: /api/
Disallow: /static/
Disallow: /article_type/
Disallow: /article_images/
Disallow: /search/
Disallow: /15055520/
Disallow: /article/[slug]
Disallow: /articles/[page]
Disallow: /author/[slug]
Disallow: /category/[slug]
Disallow: /issue/[slug]
Disallow: /issue/[slug]/[page]
Disallow: /tag/[slug]
Disallow: /tag/[slug]/[page]

User-agent: Mediapartners-Google
Allow: /

User-agent: AmazonAdBot
Allow: /

User-agent: *
Sitemap: https://www.gizmodo.jp/sitemap/index.xml
Sitemap: https://www.gizmodo.jp/feed/google_news_sitemap.xml

History

Changes recorded since June 2026.

No change recorded.

In context

Japan
32.2% block at least one AI crawler
Behind Cloudflare
No