CrawlerPulse
CrawlerPulse / Europe / France / charentelibre.fr

La Charente Libre

charentelibre.fr · France · Groupe Sud Ouest · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Allowed
OpenAI
Blocked
Anthropic
Allowed
Perplexity
Allowed
Meta
Allowed
Common Crawl
Blocked
ByteDance
Allowed
Apple
Allowed
Amazon
Allowed

robots.txt

As read on 8 Oct 2026, 4,581 bytes. 5 rule groups name AI crawlers; shown first.

User-Agent: omgilibot
Disallow: /

User-Agent: omgili
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Adsbot-Google
User-Agent: AdsBot-Google-Mobile
User-Agent: APIs-Google
User-agent: Applebot
User-agent: Bingbot
User-agent: bnf.fr_bot
User-agent: exabot
User-agent: Googlebot
User-agent: Googlebot-Image
User-agent: Googlebot-Mobile
User-agent: Googlebot-News
User-agent: Googlebot-Video
User-Agent: Googlebot_Nauxeo
User-agent: ia_archiver
User-agent: Mediapartners-Google
User-agent: Proximic
User-agent: Slurp
User-Agent: Storebot-Google
User-agent: Twitterbot
User-agent: upday
User-agent: sistrix
User-Agent: BeOpBot
User-Agent: weborama-fetcher
User-agent: Flipboard
User-agent: FlipboardProxy
Sitemap: https://www.charentelibre.fr/sitemap-news.xml
Sitemap: https://www.charentelibre.fr/sitemap.xml
Disallow: /idalgo/
Disallow: /campub/
Disallow: /recherche/
Disallow: /legacy/
Disallow: /_profiler/
Disallow: /sar_topics/
Disallow: /sar_als/
Disallow: /reagir/
Disallow: /styles-2018/
Disallow: /www/
Disallow: /blocks/article/
Disallow: /block/article/
Disallow: /*.amp.html$

User-agent: Meltawer
Disallow: /

User-agent: Digimind
Disallow: /

User-agent: Knowings
Disallow: /

User-agent: Sindup
Disallow: /

User-agent: Cision
Disallow: /

User-agent: Talkwater
Disallow: /

User-agent: TurnitinBot
Disallow: /

User-agent: ConveraCrawler
Disallow: /

User-agent: Jetbot
Disallow: /

User-agent: NewsNow
Disallow: /

User-agent: kbcrawl
Disallow: /

User-agent: AmiSoftware
Disallow: /

User-agent: Newzbin
Disallow: /

User-agent: Ask n read
Disallow: /

User-agent: Qwam content intelligence
Disallow: /

User-agent: Zite
Disallow: /

User-agent: Youmag
Disallow: /

User-agent: Synthesio
Disallow: /

User-agent: trendybuzz
Disallow: /

User-agent: spotter
Disallow: /

User-agent: scoop.it
Disallow: /

User-agent: linkfluence
Disallow: /

User-agent: 5emeRue
Disallow: /

User-agent: Augure
Disallow: /

User-agent: Corporama
Disallow: /

User-agent: grub-client
Disallow: /

User-agent: ia_archiver-web.archive.org
Allow: /$
Disallow: /*

User-agent: k2spider
Disallow: /

User-agent: libwww
Disallow: /

User-agent: wget
Disallow: /

User-agent: 5erue
Disallow: /

User-agent: adequat
Disallow: /

User-agent: adequat-systems
Disallow: /

User-agent: auramundi
Disallow: /

User-agent: coexel
Disallow: /

User-agent: ellisphere
Disallow: /

User-agent: leadbox
Disallow: /

User-agent: mention
Disallow: /

User-agent: Moreover
Disallow: /

User-agent: mytwip
Disallow: /

User-agent: opinion-tracker
Disallow: /

User-agent: proxem
Disallow: /

User-agent: score3
Disallow: /

User-agent: trendeo
Disallow: /

User-agent: vecteurplus
Disallow: /

User-agent: verticalsearch
Disallow: /

User-agent: vsw
Disallow: /

User-agent: winello
Disallow: /

User-agent: Fetch
Disallow: /

User-agent: infoseek
Disallow: /

User-agent: MSIECrawler
Disallow: /

User-agent: Offline Explorer
Disallow: /

User-agent: sitecheck.internetseer.com
Disallow: /

User-agent: Teleport
Disallow: /

User-agent: TeleportPro
Disallow: /

User-agent: WebCopier
Disallow: /

User-agent: WebStripper
Disallow: /

User-agent: Zealbot
Disallow: /

User-agent: asknread.com
Disallow: /

User-agent: Argus
Disallow: /

User-agent: augure
Disallow: /

User-agent: Bloodhound
Disallow: /

User-agent: cydralspider
Disallow: /

User-agent: Download Ninja
Disallow: /

User-agent: downloadexpress
Disallow: /

User-agent: EDD
Disallow: /

User-agent: eureka
Disallow: /

User-agent: Europresse
Disallow: /

User-agent: Explore
Disallow: /

User-agent: Factiva
Disallow: /

User-agent: Fasterfox
Disallow: /

User-agent: gammaSpider
Disallow: /

User-agent: HTTrack
Disallow: /

User-agent: indexer
Disallow: /

User-agent: Kantar
Disallow: /

User-agent: larbin
Disallow: /

User-agent: linko
Disallow: /

User-agent: manageo
Disallow: /

User-agent: mediacompil
Disallow: /

User-agent: Meltwater
Disallow: /

User-agent: newscan-online
Disallow: /

User-agent: NPBot
Disallow: /

User-agent: ObjectsSearch
Disallow: /

User-agent: Pimptrain
Disallow: /

User-agent: QuepasaCreep
Disallow: /

User-agent: Raven
Disallow: /

User-agent: readability.com
Disallow: /

User-agent: sindup
Disallow: /

User-agent: SiteSnagger
Disallow: /

User-agent: TunitinBot
Disallow: /

User-agent: up2news
Disallow: /

User-agent: Verif
Disallow: /

User-agent: wapspider
Disallow: /

User-agent: WebReaper
Disallow: /

User-agent: WebZinger
Disallow: /

User-agent: WebZIP
Disallow: /

User-agent: Wget
Disallow: /

User-agent: ZyBORG
Disallow: /

User-agent: grapeshot
Disallow: /

History

Changes recorded since June 2026.

No change recorded.

In context

France
42.3% block at least one AI crawler
Behind Cloudflare
No
Same group