CrawlerPulse
CrawlerPulse / Europe / France / nouvelobs.com

L'Obs

nouvelobs.com · France · Groupe Le Monde · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Blocked
OpenAI
Blocked
Anthropic
Blocked
Perplexity
Blocked
Meta
Allowed
Common Crawl
Blocked
ByteDance
Blocked
Apple
Allowed
Amazon
Allowed

robots.txt

As read on 8 Oct 2026, 8,475 bytes. 2 rule groups name AI crawlers; shown first.

User-agent: 008
User-agent: Alexibot
User-agent: AlvinetSpider
User-agent: Antenne Hatena
User-agent: anthropic-ai
User-agent: ApocalXExplorerBot
User-agent: asterias
User-agent: BackDoorBot/1.0
User-agent: BizInformation
User-agent: Black Hole
User-agent: BlowFish/1.0
User-agent: BotALot
User-agent: BuiltBotTough
User-agent: Bullseye/1.0
User-agent: BunnySlippers
User-agent: CCBot
User-agent: Cegbfeieh
User-agent: ChatGPT-User
User-agent: CheeseBot
User-agent: CherryPicker
User-agent: CherryPickerElite/1.0
User-agent: CherryPickerSE/1.0
User-agent: Claude-Web
User-agent: ClaudeBot
User-agent: CopyRightCheck
User-agent: cosmos
User-agent: Crescent
User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
User-agent: DISCo Pump 3.1
User-agent: DittoSpyder
User-agent: dotbot
User-agent: EmailCollector
User-agent: EmailSiphon
User-agent: EmailWolf
User-agent: EroCrawler
User-agent: ExtractorPro
User-agent: FacebookBot
User-agent: Flamingo_SearchEngine
User-agent: Foobot
User-agent: Google-Extended
User-agent: GPTBot
User-agent: g2reader-bot/1.0
User-agent: Harvest/1.5
User-agent: hloader
User-agent: httplib
User-agent: HTTrack
User-agent: HTTrack 3.0
User-agent: humanlinks
User-agent: Igentia
User-agent: InfoNaviRobot
User-agent: JennyBot
User-agent: JikeSpider
User-agent: Kenjin Spider
User-agent: LexiBot
User-agent: libWeb/clsHTTP
User-agent: LinkextractorPro
User-agent: LinkScan/8.1a Unix
User-agent: LinkWalker
User-agent: lwp-trivial
User-agent: lwp-trivial/1.34
User-agent: Mata Hari
User-agent: MediafedMetrics
User-agent: Microsoft URL Control - 5.01.4511
User-agent: Microsoft URL Control - 6.00.8169
User-agent: MIIxpc
User-agent: MIIxpc/4.2
User-agent: Mister PiX
User-agent: MLBot
User-agent: moget
User-agent: moget/2.1
User-agent: MS Search 4.0 Robot
User-agent: MS Search 5.0 Robot
User-agent: Naverbot
User-agent: NetAnts
User-agent: NetAttache
User-agent: NetMechanic
User-agent: NICErsPRO
User-agent: Nutch
User-agent: Offline Explorer
User-Agent: omgili
User-Agent: omgilibot
User-agent: Openfind
User-agent: OpenindexSpider
User-agent: Owlin Bot
User-Agent: PerplexityBot
User-agent: ProPowerBot/2.14
User-agent: ProWebWalker
User-agent: psbot
User-agent: QuepasaCreep
User-agent: QueryN Metasearch
User-agent: RepoMonkey
User-agent: RMA
User-agent: SightupBot
User-agent: SiteBot
User-agent: SiteSnagger
User-agent: SiteSucker
User-agent: SocialShare/1.0
User-agent: Sogou web spider
User-agent: sosospider
User-agent: SpankBot
User-agent: spanner
User-agent: Speedy
User-agent: Spiderbot
User-agent: Spiderbot/Nutch-1.7
User-agent: suggybot
User-agent: SuperBot
User-agent: SuperBot/2.6
User-agent: suzuran
User-agent: Szukacz/1.4
User-agent: Teleport
User-agent: Telesoft
User-agent: The Intraformant
User-agent: TheNomad
User-agent: TightTwatBot
User-agent: Titan
User-agent: toCrawl/UrlDispatcher
User-agent: TosCrawler
User-agent: True_Robot
User-agent: True_Robot/1.0
User-agent: turingos
User-agent: UrlPouls
User-agent: URLy Warning
User-agent: VCI
User-agent: Web Image Collector
User-agent: WebAuto
User-agent: WebBandit
User-agent: WebBandit/3.50
User-agent: WebCopier
User-agent: webcopy
User-agent: WebEnhancer
User-agent: WebmasterWorldForumBot
User-agent: webmirror
User-agent: WebReaper
User-agent: WebSauger
User-agent: website extractor
User-agent: Website Quester
User-agent: Webster Pro
User-agent: WebStripper
User-agent: WebStripper/2.02
User-agent: WebZip
User-agent: wget
User-agent: WikioFeedBot
User-agent: WinHTTrack
User-agent: WWW-Collector-E
User-agent: Xenu Link Sleuth/1.3.8
User-agent: yacy
User-agent: yandex
User-agent: YRSPider
User-agent: Zeus
User-agent: Zookabot
Disallow: /

User-agent: 5emeRue
User-agent: ACQUIRE MEDIA
User-agent: ACTIV Financial (CME Group)
User-agent: AlphaSense
User-agent: amazon-kendra
User-agent: amazon-QBusiness
User-agent: AmiSoftware
User-agent: archive.org_bot
User-agent: Archive-It
User-agent: ArgClrInt
User-agent: Ask n read
User-agent: Augure
User-agent: auramundi
User-agent: AwarioRssBot
User-agent: AwarioSmartBot
User-agent: Barchart.com
User-agent: BattleFin
User-agent: Bernin IT
User-agent: Blackboard Safeassign
User-agent: BLP_bbot
User-agent: bluematrix
User-agent: Brandwatch
User-agent: Briefcase.news
User-agent: Buck
User-agent: Bytespider
User-agent: CCBo
User-agent: CikisiBot
User-agent: Coexel
User-agent: cohere-ai
User-agent: Comtex News Network
User-agent: ConveraCrawler
User-agent: Copyright Licensing Agency
User-agent: Corporama
User-agent: D&B Hoovers
User-agent: Data Expression
User-agent: Data Observer
User-agent: Dataminr
User-agent: Dealogic
User-agent: Diffbot
User-agent: Digimind
User-agent: DirectFN
User-agent: Dun & Bradstreet
User-agent: Dun & Bradstreet - D&B ESG Intelligence
User-agent: Dun & Bradstreet Data Marketplace
User-agent: Eagle Alpha
User-agent: ecoresearch
User-agent: ellisphere
User-agent: FactSet
User-agent: FeedCheck
User-agent: FeedReader
User-agent: Feedspot
User-agent: Fitch Solutions
User-agent: Founder Apabi
User-agent: Freshbot
User-agent: FriendlyCrawler
User-agent: Gnowit
User-agent: GnowitNewsbot
User-agent: Ground News
User-agent: ia_archiver
User-agent: ICE Connect Desktop Solution
User-agent: ICE Data Services
User-agent: IHS Markit
User-agent: ImageSift
User-agent: InMédia Technologies
User-agent: Innguma
User-agent: Inoreader
User-agent: ISI Emerging Markets
User-agent: KB Crawl SAS
User-agent: Knowings
User-agent: Koyfin
User-agent: Launchmetrics
User-agent: LexisNexis
User-agent: Liana
User-agent: magpie-crawler
User-agent: Make/production
User-agent: MarketResearch.com
User-agent: MarketWatch
User-agent: MarketWise
User-agent: Markit Digital
User-agent: Mediatoolkitbot
User-agent: moduleQ
User-agent: MONITIO
User-agent: MoodleBot
User-agent: Moody's
User-agent: Moreover
User-agent: MORNINGSTAR
User-agent: MuckRack
User-agent: Netvibes
User-agent: news-api.org
User-agent: Newslitbot
User-agent: NewsNow
User-agent: Northern Light
User-agent: Opinion-tracker
User-agent: Orbis
User-agent: Opoint
User-agent: Paqlebot
User-agent: Press Monitor Europe
User-agent: PressEngineBot D
User-agent: PriberamBot
User-agent: QuoteMedia
User-agent: QWAM CONTENT INTELLIGENCE
User-agent: RankurBot
User-agent: RavenPack
User-agent: ReportLinker
User-agent: Research & Markets
User-agent: S&P Capital IQ
User-agent: S&P Global Market Intelligence
User-agent: S&P Global Marketplace
User-agent: scoopit
User-agent: scoopit-crawler
User-agent: scpitspi-rs
User-agent: semantic-visions.com
User-agent: semantic-visions.com crawler
User-agent: SemrushBot
User-agent: SentiBot
User-agent: SentiOne
User-agent: Signal Insights
User-agent: Sindup
User-agent: smartkarma
User-agent: Sociallymap
User-agent: spotter
User-agent: Sprinklr
User-agent: squirrobot
User-agent: Statista
User-agent: STOCKBOARD
User-agent: telpress.it
User-agent: Thomson Reuters
User-agent: Thomson Reuters WestLaw
User-agent: TraderPlanet
User-agent: Tradingcharts.com
User-agent: TradingView
User-agent: Transform Your API
User-agent: trendeo
User-agent: trendictionbot
User-agent: TurnitinBot
User-agent: um-FC
User-agent: um-IC
User-agent: um-LN
User-agent: UptimeRobot
User-agent: Vable Ltd
User-agent: VISIBRAIN
User-agent: webzio
User-agent: YaK
User-agent: YCHARTS
User-agent: YouBot
User-agent: Zite
Disallow: /

User-agent: Adsbot-Google
User-Agent: AdsBot-Google-Mobile
User-Agent: APIs-Google
User-agent: Applebot
User-agent: Bingbot
User-agent: exabot
User-agent: Googlebot
User-agent: Googlebot-Image
User-agent: Googlebot-Mobile
User-agent: Googlebot-News
User-agent: Googlebot-Video
User-agent: ia_archiver
User-agent: Mediapartners-Google
User-agent: Proximic
User-agent: Slurp
User-agent: Twitterbot
User-agent: sistrix
User-agent: Flipboard
User-agent: FlipboardProxy
Allow: /
Sitemap : https://www.nouvelobs.com/sitemap/sitemap-index-articles.xml
Sitemap : https://www.nouvelobs.com/sitemap/sitemap-articles-news.xml
Disallow: /recherche?*
Disallow: /*?status=login
Disallow: /r/header?*
Disallow: /codepromo/voir-code-promo/*
Disallow: /codepromo/out/*
Disallow: /codepromo/chercher*
Disallow: /newsletter/*
Disallow: /bucket/*

User-agent: Googlebot-Image
Allow: /image/

User-agent: Googlebot-News
Disallow: /index/
Disallow: /codepromo/
Disallow: /newsletter/*
Sitemap : https://www.nouvelobs.com/sitemap/sitemap-articles-news.xml

History

Changes recorded since June 2026.

No change recorded.

In context

France
42.3% block at least one AI crawler
Behind Cloudflare
No
Same group