CrawlerPulse
CrawlerPulse / Europe / France / lexpress.fr

L'EXPRESS

lexpress.fr · France · Groupe L'EXPRESS · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Blocked
OpenAI
Blocked
Anthropic
Blocked
Perplexity
Blocked
Meta
Blocked
Common Crawl
Blocked
ByteDance
Allowed
Apple
Blocked
Amazon
Blocked

robots.txt

As read on 8 Oct 2026, 6,008 bytes. 17 rule groups name AI crawlers; shown first.

User-agent: Amazonbot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Applebot-Extended
Allow: /guide-shopping/
Allow: /partenaires/
Allow: /styles/
Allow: /essentiels-entrepreneur/
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: ChatGPT-User
Allow: /guide-shopping/
Allow: /partenaires/
Allow: /styles/
Allow: /essentiels-entrepreneur/
Disallow: /

User-agent: Claude-web
Allow: /guide-shopping/
Allow: /partenaires/
Allow: /styles/
Allow: /essentiels-entrepreneur/
Disallow: /

User-agent: ClaudeBot
Allow: /guide-shopping/
Allow: /partenaires/
Allow: /styles/
Allow: /essentiels-entrepreneur/
Disallow: /

User-agent: cohere-ai
Disallow: /

User-agent: FacebookBot
Disallow: /

User-agent: Google-Extended
Allow: /guide-shopping/
Allow: /partenaires/
Allow: /styles/
Allow: /essentiels-entrepreneur/
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

User-agent: MistralAI-User/1.0
Allow: /guide-shopping/
Allow: /partenaires/
Allow: /styles/
Allow: /essentiels-entrepreneur/
Disallow: /

User-agent: MistralAI-User
Allow: /guide-shopping/
Allow: /partenaires/
Allow: /styles/
Allow: /essentiels-entrepreneur/
Disallow: /

User-agent: OAI-SearchBot
Allow: /guide-shopping/
Allow: /partenaires/
Allow: /styles/
Allow: /essentiels-entrepreneur/
Disallow: /

User-agent: PerplexityBot
Allow: /guide-shopping/
Allow: /partenaires/
Allow: /styles/
Allow: /essentiels-entrepreneur/
Disallow: /

User-agent: YouBot
Disallow: /

User-agent: *
Disallow: /tiny/
Disallow: /includes
Disallow: /imgs
Disallow: /imgstat
Disallow: /pf/api/
Disallow: /.well-known/assetlinks.json
Disallow: /recherche/
Sitemap: https://www.lexpress.fr/arc/outboundfeeds/rss/alaune.xml
Sitemap: https://www.lexpress.fr/arc/outboundfeeds/sections-sitemap.xml
Sitemap: https://www.lexpress.fr/arc/outboundfeeds/sitemap-news.xml
Sitemap: https://www.lexpress.fr/arc/outboundfeeds/sitemap-by-week-2020-now.xml
Sitemap: https://www.lexpress.fr/arc/outboundfeeds/sitemap-by-week-2010-2020.xml
Sitemap: https://www.lexpress.fr/arc/outboundfeeds/current-year-to-now.xml
Sitemap: https://www.lexpress.fr/arc/outboundfeeds/sitemap-all/days/rolling/?outputType=xml
Sitemap: https://www.lexpress.fr/arc/outboundfeeds/sitemap-all/weeks/rolling/?outputType=xml
Sitemap: https://www.lexpress.fr/arc/outboundfeeds/sitemap-all/months/rolling/?outputType=xml
Sitemap: https://www.lexpress.fr/arc/outboundfeeds/sitemap-all/years/rolling/?outputType=xml
Sitemap: https://www.lexpress.fr/arc/outboundfeeds/sitemap-all/days/latest/?outputType=xml
Sitemap: https://www.lexpress.fr/arc/outboundfeeds/sitemap-all/weeks/latest/?outputType=xml
Sitemap: https://www.lexpress.fr/arc/outboundfeeds/sitemap-all/months/latest/?outputType=xml
Sitemap: https://www.lexpress.fr/arc/outboundfeeds/sitemap-all/years/latest/?outputType=xml

User-agent: Googlebot
Allow: /
Disallow: /*.svg
Disallow: /tiny/
Disallow: /includes
Disallow: /imgs
Disallow: /imgstat
Disallow: /pf/api/
Disallow: /.well-known/assetlinks.json
Disallow: /recherche/
Disallow: /amp/

User-agent: Adsbot-Google
Allow: /

User-agent: AdsBot-Google-Mobile
Allow: /

User-agent: Applebot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: exabot
Allow: /

User-agent: FacebookExternalHit
Allow: /

User-agent: Googlebot-Image
Allow: /

User-agent: Googlebot-Mobile
Allow: /

User-agent: Googlebot-News
Allow: /

User-agent: Googlebot-Video
Allow: /

User-agent: Mediapartners-Google
Allow: /

User-agent: Slurp
Allow: /

User-agent: Twitterbot
Allow: /

User-agent: Googlebot-News
Disallow: /guide-shopping/
Disallow: /archives/
Disallow: /essentiels-entrepreneur/
Disallow: /amp/

User-agent: AI2Bot
Disallow: /

User-agent: AmiSoftware
Disallow: /

User-agent: Ask n read
Disallow: /

User-agent: asknread.com
Disallow: /

User-agent: BackDoorBot/1.0
Disallow: /

User-agent: BizInformation
Disallow: /

User-agent: Claude-user
Allow: /guide-shopping/
Allow: /partenaires/
Allow: /styles/
Allow: /essentiels-entrepreneur/
Disallow: /

User-agent: Claude-SearchBot
Allow: /guide-shopping/
Allow: /partenaires/
Allow: /styles/
Allow: /essentiels-entrepreneur/
Disallow: /

User-agent: ConveraCrawler
Disallow: /

User-agent: Digimind
Disallow: /

User-agent: Flipboard
Disallow: /

User-agent: FlipboardProxy
Disallow: /

User-agent: Gemini-DeepResearch
Allow: /guide-shopping/
Allow: /partenaires/
Allow: /styles/
Allow: /essentiels-entrepreneur/
Disallow: /

User-agent: ia_archiver
Disallow: /

User-agent: Jetbot
Disallow: /

User-agent: kbcrawl
Disallow: /

User-agent: Knowings D
Disallow: /

User-agent: Meltwater
Disallow: /

User-agent: Meta-ExternalFetcher
Disallow: /

User-agent: NewsNow
Disallow: /

User-agent: Newzbin
Disallow: /

User-agent: QuepasaCreep
Disallow: /

User-agent: Qwam content intelligence
Disallow: /

User-agent: TikTokSpider
Disallow: /

User-agent: Tunit
Disallow: /

User-agent: DuckAssistBot
Disallow: /

History

Changes recorded since June 2026.

No change recorded.

In context

France
42.3% block at least one AI crawler
Behind Cloudflare
No