CrawlerPulse
CrawlerPulse / Europe / France / leprogres.fr

Le Progrès

leprogres.fr · France · EBRA · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Blocked
OpenAI
Blocked
Anthropic
Allowed
Perplexity
Allowed
Meta
Allowed
Common Crawl
Blocked
ByteDance
Blocked
Apple
Allowed
Amazon
Blocked

robots.txt

As read on 8 Oct 2026, 2,863 bytes. 8 rule groups name AI crawlers; shown first.

User-agent: Amazonbot
Disallow: /
Allow: /paroles-de-partenaires/
Allow: /comparateur-telecom/

User-agent: anthropic-ai
Disallow: /
Allow: /paroles-de-partenaires/
Allow: /comparateur-telecom/

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /
Allow: /paroles-de-partenaires/
Allow: /comparateur-telecom/

User-agent: Claude-Web
Disallow: /
Allow: /paroles-de-partenaires/
Allow: /comparateur-telecom/

User-Agent: Google-Extended
Disallow: /
Allow: /paroles-de-partenaires/
Allow: /comparateur-telecom/

User-agent: GPTBot
Disallow: /
Allow: /paroles-de-partenaires/
Allow: /comparateur-telecom/

User-agent: *
Disallow: /recherche
Disallow: /webservices/
Disallow: /infos-services
Disallow: /e-services/
Disallow: /test/
Disallow: /search
Disallow: /nl/jsp/
Disallow: /e-services/boutique/commande
Disallow: /e-services/boutique/identification
Disallow: /e-services/boutique/complementprofil
Disallow: /e-services/boutique/paiement
Disallow: /e-services/boutique/callbackTPEOk
Disallow: /e-services/boutique/callbackTPEKo
Disallow: /politique/election-presidentielle-2002-tour-1
Disallow: /politique/election-presidentielle-2002-tour-2
Disallow: /politique/election-presidentielle-2007-tour-1
Disallow: /politique/election-presidentielle-2007-tour-2
Disallow: /politique/elections-europeennes-2009
Disallow: /politique/elections-legislatives-2002-tour-1
Disallow: /politique/elections-legislatives-2002-tour-2
Disallow: /politique/elections-legislatives-2007-tour-1
Disallow: /politique/elections-legislatives-2007-tour-2
Disallow: /politique/referendum-conseil-unique-alsace-7-avril-2013
Disallow: /politique/election-partielle-du-7-octobre-2012
Disallow: /politique/election-cantonale-partielle-2012-munster-tour-1
Disallow: /politique/elections-municipales-2008-tour-1
Disallow: /politique/election-partielle-du-14-octobre-2012
Disallow: /politique/elections-partielles-du-8-juillet-2012
Disallow: /politique/elections-municipales-2008-tour-2
Disallow: /politique/elections-partielles-du-1er-juillet-2012
Disallow: /elections/resultats/elections-testdecharge-2026
Disallow: /elections/resultats/elections-testdecharge-2026?*
Disallow: /education/resultats-examens-2019
Disallow: /Annonce/JustificatifParution
Disallow: */avantages-exclusifs?*localisation=
Disallow: */avantages-exclusifs?*categorie=
Disallow: /test/
Disallow: /contact-signaler-erreur?*
Disallow: /services/gsoc/viewcountservice/increment
Sitemap: https://www.leprogres.fr/sitemap-request.xml

User-agent: Mediapartners-Google
Disallow:
Disallow: /education/resultats-examens-2011
Disallow: /education/resultats-examens-2012
Disallow: /education/resultats-examens-2013
Disallow: /services/gsoc/viewcountservice/increment
Disallow: /test/

History

Changes recorded since June 2026.

No change recorded.

In context

France
42.3% block at least one AI crawler
Behind Cloudflare
No
Same group