CrawlerPulse
CrawlerPulse / Asia / Japan / webcg.net

クルマ好きなら毎日みてる webCG 新車情報・カーグラフィック

webcg.net · Japan · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Blocked
OpenAI
Blocked
Anthropic
Allowed
Perplexity
Allowed
Meta
Allowed
Common Crawl
Blocked
ByteDance
Allowed
Apple
Allowed
Amazon
Allowed

robots.txt

As read on 8 Oct 2026, 979 bytes. 3 rule groups name AI crawlers; shown first.

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /
Disallow: /WEBCG/alliance/
Disallow: /WEBCG/campaign/
Disallow: /WEBCG/carnavi/
Disallow: /WEBCG/carscope/
Disallow: /WEBCG/cgtv/
Disallow: /WEBCG/counter/
Disallow: /WEBCG/essays/
Disallow: /WEBCG/forum/
Disallow: /WEBCG/FromOurStaff/
Disallow: /WEBCG/impressions/
Disallow: /WEBCG/magazines/
Disallow: /WEBCG/mcg/
Disallow: /WEBCG/members/
Disallow: /WEBCG/members_top/
Disallow: /WEBCG/navi2/
Disallow: /WEBCG/naviaudio/
Disallow: /WEBCG/news/
Disallow: /WEBCG/qa/
Disallow: /WEBCG/selection/
Disallow: /WEBCG/showcase/
Disallow: /WEBCG/specials/
Disallow: /WEBCG/special_offer/
Disallow: /WEBCG/test/
Disallow: /resources/webcg/js/v5/main/
Disallow: /resources/webcg/js/v1/smartphone/main/
Disallow: /list/personal/premium/common-header-status-display
Disallow: /auth/
Sitemap: http://www.webcg.net/sitemap.xml

History

Changes recorded since June 2026.

No change recorded.

In context

Japan
32.2% block at least one AI crawler
Behind Cloudflare
No