epochtimes.com · United States · measured 9 Oct 2026
As read on 9 Oct 2026, 8,230 bytes. 4 rule groups name AI crawlers; shown first.
User-agent: OAI-SearchBot User-agent: Claude-SearchBot User-agent: Applebot User-agent: meta-webindexer User-agent: Amzn-SearchBot User-agent: PerplexityBot User-agent: AzureAI-SearchBot User-agent: xAI-Bot User-agent: Bravebot User-agent: YouBot User-agent: DuckAssistBot User-agent: KagiBot User-agent: kagi-fetcher User-agent: TavilyBot User-agent: ExaBot User-agent: LinkupBot User-agent: Andibot User-agent: PhindBot User-agent: LinerBot User-agent: PetalBot User-agent: YandexAdditional User-agent: YandexAdditionalBot User-agent: Cloudflare-AutoRAG User-agent: Anomura User-agent: ShapBot User-agent: AddSearchBot User-agent: Channel3Bot User-agent: ZanistaBot User-agent: Aranet-SearchBot User-agent: IbouBot User-agent: WRTNBot User-agent: KunatoCrawler Allow: /*ncid1594506 Allow: /*n14264789 Allow: /*n13975246 Allow: /*n13912117 Allow: /*nccomment Allow: /*nf1156504 Allow: /*nf5762 Allow: /*nf1176122 Allow: /*nf1267582 Allow: /*nf1242025 Allow: /*nf6092 Allow: /*n14341753 Allow: /*nf316 Allow: /*nf2509 Allow: /*nf2565 Allow: /*nf2595 Allow: /*nf2925 Allow: /*nf3046 Allow: /*nf3051 Allow: /*nf3082 Allow: /*nf3083 Allow: /*nf3084 Allow: /*nf3085 Allow: /*nf3087 Allow: /*nf3088 Allow: /*nf3089 Allow: /*nf3090 Allow: /*nf3091 Allow: /*nf3092 Allow: /*nf3104 Allow: /*nf3105 Allow: /*nf3113 Allow: /*nf3137 Allow: /*nf3144 Allow: /*nf3148 Allow: /*nf3150 Allow: /*nf3165 Allow: /*nf3178 Allow: /*nf3180 Allow: /*nf3192 Allow: /*nf3193 Allow: /*nf3195 Allow: /*nf3204 Allow: /*nf3259 Allow: /*nf3279 Allow: /*nf3280 Allow: /*nf3330 Allow: /*nf3340 Allow: /*nf3341 Allow: /*nf3411 Allow: /*nf3418 Allow: /*nf3428 Allow: /*nf3434 Allow: /*nf3449 Allow: /*nf3451 Allow: /*nf3452 Allow: /*nf3469 Allow: /*nf3474 Allow: /*nf3519 Allow: /*nf3628 Allow: /*nf3873 Allow: /*nf3927 Allow: /*nf3941 Allow: /*nf3942 Allow: /*nf3964 Allow: /*nf4378 Allow: /*nf4379 Allow: /*nf4380 Allow: /*nf4382 Allow: /*nf4385 Allow: /*nf4386 Allow: /*nf4387 Allow: /*nf4388 Allow: /*nf4389 Allow: /*nf4390 Allow: /*nf4442 Allow: /*nf4627 Allow: /*nf4673 Allow: /*nf4689 Allow: /*nf4693 Allow: /*nf4694 Allow: /*nf4742 Allow: /*nf4825 Allow: /*nf4830 Allow: /*nf4889 Allow: /*nf4934 Allow: /*nf5039 Allow: /*nf5130 Allow: /*nf5217 Allow: /*nf5287 Allow: /*nf5364 Allow: /*nf5407 Allow: /*nf5410 Allow: /*nf5467 Allow: /*nf5483 Allow: /*nf5557 Allow: /*nf5561 Allow: /*nf5562 Allow: /*nf5611 Allow: /*nf5640 Allow: /*nf5657 Allow: /*nf5677 Allow: /*nf5703 Allow: /*nf5708 Allow: /*nf5730 Allow: /*nf5735 Allow: /*nf5736 Allow: /*nf5760 Allow: /*nf5790 Allow: /*nf5792 Allow: /*nf5793 Allow: /*nf5796 Allow: /*nf5800 Allow: /*nf5864 Allow: /*nf5875 Allow: /*nf5877 Allow: /*nf5878 Allow: /*nf5879 Allow: /*nf5880 Allow: /*nf5881 Allow: /*nf5882 Allow: /*nf5883 Allow: /*nf5884 Allow: /*nf5886 Allow: /*nf5890 Allow: /*nf5891 Allow: /*nf5924 Allow: /*nf5932 Allow: /*nf5947 Allow: /*nf5948 Allow: /*nf5950 Allow: /*nf5961 Allow: /*nf6039 Allow: /*nf6103 Allow: /*nf6117 Allow: /*nf6118 Allow: /*nf6119 Allow: /*nf6120 Allow: /*nf6121 Allow: /*nf6122 Allow: /*nf6123 Allow: /*nf6124 Allow: /*nf6125 Allow: /*nf6126 Allow: /*nf6127 Allow: /*nf6148 Allow: /*nf6149 Allow: /*nf6150 Allow: /*nf6166 Allow: /*nf6167 Allow: /*nf6168 Allow: /*nf6169 Allow: /*nf6170 Allow: /*nf6171 Allow: /*nf6230 Allow: /*nf6231 Allow: /*nf6232 Allow: /*nf6241 Allow: /*nf6242 Allow: /*nf6243 Allow: /*nf6244 Allow: /*nf6298 Allow: /*nf1129792 Allow: /*nf1146147 Allow: /*nf1147481 Allow: /*nf1176122 Allow: /*nf1205792 Allow: /*nf1266219 Allow: /*nf1267582 Allow: /*nf1397215 Allow: /*nf1406710 Allow: /*nf1415509 Allow: /$ Disallow: / User-agent: ChatGPT-User User-agent: ChatGPT Agent User-agent: Operator User-agent: Claude-User User-agent: Claude-Code User-agent: Perplexity-User User-agent: MistralAI-User User-agent: Meta-ExternalFetcher User-agent: meta-externalfetcher User-agent: Google-NotebookLM User-agent: NotebookLM User-agent: Gemini-Deep-Research User-agent: GoogleAgent-Mariner User-agent: Google-Agent User-agent: cohere-ai User-agent: Amzn-User User-agent: NovaAct User-agent: Manus-User User-agent: Devin Allow: / User-agent: GPTBot User-agent: ClaudeBot User-agent: anthropic-ai User-agent: Claude-Web User-agent: Google-Extended User-agent: Google-CloudVertexBot User-agent: CloudVertexBot User-agent: GoogleOther User-agent: GoogleOther-Image User-agent: GoogleOther-Video User-agent: Google-Firebase User-agent: Applebot-Extended User-agent: Meta-ExternalAgent User-agent: meta-externalagent User-agent: FacebookBot User-agent: Bytespider User-agent: TikTokSpider User-agent: Amazonbot User-agent: bedrockbot User-agent: amazon-kendra User-agent: cohere-training-data-crawler User-agent: DeepSeekBot User-agent: PanguBot User-agent: MistralAI-Index User-agent: CCBot User-agent: AI2Bot User-agent: Ai2Bot-Dolma User-agent: AI2Bot-DeepResearchEval User-agent: LAIONDownloader User-agent: laion-huggingface-processor User-agent: Brightbot User-agent: Brightbot 1.0 User-agent: Diffbot User-agent: FirecrawlAgent User-agent: ICC-Crawler User-agent: SBIntuitionsBot User-agent: Cotoyogi User-agent: Kangaroo Bot User-agent: Timpibot User-agent: Factset_spyderbot User-agent: ISSCyberRiskCrawler User-agent: Scrapy User-agent: img2dataset User-agent: VelenPublicWebCrawler User-agent: Sidetrade indexer bot User-agent: Linguee Bot User-agent: TerraCotta User-agent: FriendlyCrawler User-agent: Poseidon Research Crawler User-agent: ChatGLM-Spider User-agent: Crawlspace User-agent: Crawl4AI User-agent: MyCentralAIScraperBot Disallow: / User-agent: Webzio-Extended User-agent: webzio-extended User-agent: omgili User-agent: omgilibot User-agent: ImagesiftBot User-agent: ApifyBot User-agent: ApifyWebsiteContentCrawler User-agent: YaK User-agent: Awario User-agent: EchoboxBot User-agent: Echobot Bot User-agent: Panscient User-agent: panscient.com User-agent: aiHitBot User-agent: SemrushBot-OCOB User-agent: SemrushBot-SWA User-agent: QuillBot User-agent: quillbot.com User-agent: QualifiedBot User-agent: netEstate Imprint Crawler User-agent: Datenbank Crawler User-agent: HenkBot User-agent: Thinkbot User-agent: WARDBot User-agent: NagetBot User-agent: newsai User-agent: AgentTimes User-agent: iAskBot User-agent: iaskspider User-agent: iaskspider/2.0 User-agent: Mediatoolkitbot User-agent: PolecatBot Disallow: / Sitemap: https://www.epochtimes.com/sitemap/zh-tw/sitemap.xml.gz Sitemap: https://www.epochtimes.com/sitemap/zh-tw/sitemap_news.xml.gz Sitemap: https://www.epochtimes.com/sitemap/zh-tw/sitemap_category.xml.gz Sitemap: https://www.epochtimes.com/sitemap/zh-tw/sitemap_pages.xml Sitemap: https://www.epochtimes.com/sitemap/zh-cn/sitemap.xml.gz Sitemap: https://www.epochtimes.com/sitemap/zh-cn/sitemap_news.xml.gz Sitemap: https://www.epochtimes.com/sitemap/zh-cn/sitemap_category.xml.gz Sitemap: https://www.epochtimes.com/sitemap/zh-cn/sitemap_pages.xml User-agent: * Disallow: */uncategorized/* Disallow: /wp-includes/* Disallow: /wp-admin/* Disallow: /wp-content/plugins/* Disallow: /assets/plugins/* Disallow: /search/* Disallow: /feedback/* Disallow: /*?q=* Disallow: /*?p=* Disallow: /*?ref Disallow: *utm_ Disallow: /*?rtcl_location= Disallow: /*?rtcl_category= Disallow: /*?qid= Disallow: /*?variant= Disallow: *?_page= Disallow: /*fbclid= Disallow: /?rtcl_location= Disallow: /?rtcl_category= Disallow: /*?q[make_eq]= Disallow: /*?utf8= Disallow: /*?issue= Disallow: /*?ID= Disallow: /*?page_id= Disallow: /*?455 Disallow: /*?page= Disallow: /*?month= Disallow: /*?_gl= Disallow: /*?type= Disallow: /*?view= Disallow: /*?t= User-agent: Twitterbot Allow: /*?*utm_ User-agent: facebookexternalhit Allow: /*?*utm_ User-agent: Twitterbot Allow: /*?*utm_ User-agent: facebookexternalhit Allow: /*?*utm_ User-agent: Scope3/2.0 (scope3.com) Allow: /
Changes recorded since June 2026.
No change recorded.