CrawlerPulse
CrawlerPulse / Asia / Japan / epochtimes.jp

大紀元

epochtimes.jp · Japan · 大紀元 | グローバルニュース · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Allowed
OpenAI
Allowed
Anthropic
Allowed
Perplexity
Allowed
Meta
Allowed
Common Crawl
Allowed
ByteDance
Allowed
Apple
Allowed
Amazon
Allowed

robots.txt

As read on 8 Oct 2026, 1,542 bytes. 2 rule groups name AI crawlers; shown first.

User-Agent: omgilibot
Disallow: /

User-Agent: omgili
Disallow: /

User-agent: *
Disallow: *?UserProfile =
Disallow: *utm_
Disallow: *?sub_id=
Disallow: *?FreeRegist
Disallow: *?cmt=
Disallow: *?est=
Disallow: *?p=
Disallow: *?af_sub1=
Disallow: *?slsuccess=
Disallow: *?pipa_verify_email=
Disallow: *?pipa_reset_password=
Disallow: *?lwpw=
Disallow: *?rs=
Disallow: *?fbclid=
Disallow: *?mobile=
Disallow: *?__twitter_impression=
Disallow: *?fb_comment_id=
Disallow: *?s=
Disallow: *?sidebar=
Disallow: *?photo=
Disallow: *?v=
Disallow: *?ver=
Disallow: *?duration=
Disallow: *?src_src=
Disallow: *?welcomeuser=
Disallow: *?ea_med=
Disallow: *?src_cat=
Disallow: *?li_source=
Disallow: *?w3tc_preview=
Disallow: *?share=
Disallow: *?_gl=
Disallow: *?Wb=
Disallow: *?_page=
Disallow: */uncategorized/*
Disallow: /wp-includes/*
Disallow: /wp-admin/*
Disallow: /wp-content/plugins/*
Disallow: /assets/plugins/*
Disallow: /search/*
Disallow: /tag/*
Disallow: /feedback/*
Disallow: /*?q=*
Disallow: /*?ref

User-agent: Twitterbot
Allow: /*?*utm_

User-agent: facebookexternalhit
Allow: /*?*utm_

User-Agent:Mediatoolkitbot
Disallow: /

User-Agent:PolecatBot
Disallow: /

User-Agent:DotBot
Disallow: /
Sitemap: https://www.epochtimes.jp/sitemap/sitemap.xml
Sitemap: https://www.epochtimes.jp/sitemap/sitemap-standard-index.xml
Sitemap: https://www.epochtimes.jp/sitemap/sitemap-amp-index.xml
Sitemap: https://www.epochtimes.jp/sitemap/sitemap-video-index.xml
Sitemap: https://www.epochtimes.jp/sitemap/sitemap-video_amp-index.xml

History

Changes recorded since June 2026.

No change recorded.

In context

Japan
32.2% block at least one AI crawler
Behind Cloudflare
Yes