CrawlerPulse
CrawlerPulse / Europe / France / femina.fr

Femina

femina.fr · France · CMI Digital · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Blocked
OpenAI
Blocked
Anthropic
Blocked
Perplexity
Allowed
Meta
Blocked
Common Crawl
Blocked
ByteDance
Blocked
Apple
Blocked
Amazon
Blocked

robots.txt

As read on 8 Oct 2026, 3,620 bytes. 1 rule group name AI crawlers; shown first.

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: MistralAI-Training
User-agent: meta-externalagent
User-agent: Bytespider
User-agent: CCBot
User-agent: DeepSeekBot
User-agent: cohere-ai
User-agent: cohere-training-data-crawler
User-agent: Amazonbot
User-agent: Ai2Bot
User-agent: Ai2Bot-Dolma
User-agent: Diffbot
User-agent: omgili
User-agent: omgilibot
User-agent: GrokBot
User-agent: xAI-Grok
User-agent: Grok-DeepSearch
Disallow: /

User-Agent: *
Disallow: /index.php/
Disallow: /content/
Disallow: /layout
Disallow: /action
Disallow: /search
Disallow: /recherche
Disallow: /admin

User-agent: Meltawer
User-agent: Digimind
User-agent: Knowings
User-agent: Sindup
User-agent: Cision
User-agent: Talkwater
User-agent: TurnitinBot
User-agent: ConveraCrawler
User-agent: Jetbot
User-agent: NewsNow
User-agent: kbcrawl
User-agent: AmiSoftware
User-agent: Newzbin
User-agent: Ask n read
User-agent: Qwam content intelligence
User-agent: Zite
User-agent: Youmag
User-agent: Synthesio
User-agent: trendybuzz
User-agent: spotter
User-agent: scoop.it
User-agent: linkfluence
User-agent: 5emeRue
User-agent: Augure
User-agent: Corporama
User-agent: grub-client
User-agent: k2spider
User-agent: libwww
User-agent: wget
User-agent: 5erue
User-agent: adequat
User-agent: adequat-systems
User-agent: auramundi
User-agent: coexel
User-agent: ellisphere
User-agent: leadbox
User-agent: mention
User-agent: Moreover
User-agent: mytwip
User-agent: opinion-tracker
User-agent: proxem
User-agent: score3
User-agent: trendeo
User-agent: vecteurplus
User-agent: verticalsearch
User-agent: vsw
User-agent: winello
User-agent: Fetch
User-agent: infoseek
User-agent: MSIECrawler
User-agent: Offline Explorer
User-agent: sitecheck.internetseer.com
User-agent: Teleport
User-agent: TeleportPro
User-agent: WebCopier
User-agent: WebStripper
User-agent: Zealbot
User-agent: asknread.com
User-agent: Xenu Link Sleuth/1.3.8
Disallow: /
Sitemap: https://www.femina.fr/sitemap/sitemap-news.xml

History

Changes recorded since June 2026.

No change recorded.

In context

France
42.3% block at least one AI crawler
Behind Cloudflare
No
Same group