CrawlerPulse
CrawlerPulse / Asia / Taiwan / cna.com.tw

Central News Agency

cna.com.tw · Taiwan · Central News Agency (CNA) · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Blocked
OpenAI
Blocked
Anthropic
Blocked
Perplexity
Blocked
Meta
Blocked
Common Crawl
Blocked
ByteDance
Blocked
Apple
Blocked
Amazon
Blocked

robots.txt

As read on 8 Oct 2026, 2,230 bytes. 2 rule groups name AI crawlers; shown first.

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: Bytespider # 字節跳動(ByteDance)用於收集 AI 模型訓練語料與數據的大規模抓取器
User-agent: CCBot
User-agent: KimiBot
User-agent: GoogleOther # 阻擋 GoogleOther 家族(研發/大規模訓練數據抓取)
User-agent: GoogleOther-Image
User-agent: GoogleOther-Video
User-agent: Google-Extended # 阻擋 Google-Extended(明確拒絕 Gemini/Vertex AI 模型訓練)
User-agent: PerplexityBot
Disallow: /

User-agent: Baiduspider # 百度
User-agent: PetalBot # 華為
User-agent: Sogou web spider # 搜狗
User-agent: 360Spider # 奇虎360/好搜
Disallow: /
Sitemap: https://www.cna.com.tw/atomfeed_cfp.xml
Sitemap: https://www.cna.com.tw/sitemap_fromRemote_cfp.xml
Sitemap: https://www.cna.com.tw/GoogleNewsSitemap_fromRemote_cfp.xml

# =========================================================

User-agent: *
Content-signal: search=yes, ai-input=yes, ai-train=no
Disallow: /culture/search.aspx?q=*
Disallow: /Postwrite/search?q=*
Disallow: /video/search?q=*
Disallow: /Proj_GoodBook/Search/*/*.aspx
Disallow: /Proj_GoodBook/search.aspx?q=*
Disallow: /proj_goodbook/api/*
Disallow: /culture/topicpreview/*
Disallow: /culture/articlepreview/*
Disallow: /gallery/*
Disallow: /Proj_County/*
Disallow: /proj_county/*
Disallow: /search/hysearchws.aspx?q=*
Disallow: /cna2018api/api/*
Disallow: /cna2019api/cna/*
Disallow: /newsworld/search/*/
Disallow: /proj_goodbook/api/

History

Changes recorded since June 2026.

No change recorded.

In context

Taiwan
35.7% block at least one AI crawler
Behind Cloudflare
No