timesleader.com · United States · measured 9 Oct 2026
As read on 9 Oct 2026, 1,623 bytes. 2 rule groups name AI crawlers; shown first.
User-agent: facebookexternalhit User-agent: Facebot User-agent: FacebookBot Crawl-delay: 10 Allow: / Disallow: /wp-json/ Disallow: /wp-admin/ User-agent: GPTBot User-agent: ChatGPT-User User-agent: ClaudeBot User-agent: PerplexityBot User-agent: anthropic-ai User-agent: GrokBot User-agent: xAI-Grok Disallow: / User-agent: * Crawl-delay: 8 Allow: / Disallow: /wp-json/ Disallow: /wp-json/wp/v2/ Disallow: /wp-json/oembed/ Disallow: /wp-json/wc/ # if you use WooCommerce Disallow: /wp-admin/ Disallow: /wp-login.php Disallow: /wp-register.php Disallow: /xmlrpc.php Disallow: /?s= # block search result pages Disallow: /search/ Disallow: /*?* User-agent: Googlebot User-agent: Googlebot-Image User-agent: Googlebot-Video User-agent: Bingbot User-agent: Slurp User-agent: DuckDuckBot Allow: /wp-json/ Allow: / User-agent: Twitterbot Crawl-delay: 10 Allow: / Disallow: /wp-json/ Disallow: /wp-admin/ User-agent: AhrefsBot Crawl-delay: 15 Disallow: /wp-json/ User-agent: SemrushBot Crawl-delay: 15 Disallow: /wp-json/ User-agent: MJ12bot Crawl-delay: 20 Disallow: /wp-json/ User-agent: DotBot Crawl-delay: 20 Disallow: /wp-json/
Changes recorded since June 2026.
No change recorded.