italiaoggi.it · Italy · Italia Oggi - Editori Erinne · measured 8 Oct 2026
As read on 8 Oct 2026, 21,299 bytes. 20 rule groups name AI crawlers; shown first.
User-agent: GPTBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: PerplexityBot Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Google-Extended Allow: / User-agent: Amazonbot Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: OAI-SearchBot Disallow: / User-agent: YouBot Disallow: / User-agent: Bytespider Disallow: / User-agent: CCBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Claude-Web Disallow: / User-agent: Diffbot Disallow: / User-agent: FacebookBot Disallow: / User-agent: Meta-ExternalAgent Disallow: / User-agent: omgili Disallow: / User-agent: cohere-ai Disallow: / User-agent: Timpibot Disallow: / User-agent: CCBot Disallow: / User-agent: Meta-ExternalFetcher Disallow: / User-agent: Applebot Disallow: / User-agent: Ai2Bot Disallow: / User-agent: Ai2Bot-Dolma Disallow: / User-agent: Scrapy Disallow: User-agent: Webzio-Extended Disallow: / User-agent: Flipboard Allow: / User-agent: FlipboardProxy Allow: / User-agent: facebookexternalhit Allow: / User-agent: Synthesio Disallow: / User-agent: ^Nail Disallow: / User-agent: 192.comAgent Disallow: / User-agent: 50.nu Disallow: / User-agent: 80legs Disallow: / User-agent: abby Disallow: / User-agent: Aboundexbot Disallow: / User-agent: AboutUsBot Disallow: / User-agent: Abrave Spider Disallow: / User-agent: Accelobot Disallow: / User-agent: AcoonBot Disallow: / User-agent: AddThis.com Disallow: / User-agent: adressendeutschland.de Disallow: / User-agent: AhrefsBot Disallow: / User-agent: aiHitBot Disallow: / User-agent: akula Disallow: / User-agent: Alexibot Disallow: / User-agent: Amagit.COM Disallow: / User-agent: amibot Disallow: / User-agent: AntBot Disallow: / User-agent: Apercite Disallow: / User-agent: AportWorm Disallow: / User-agent: Aqua_Products Disallow: / User-agent: arachnode.net Disallow: / User-agent: archive.org_bot Disallow: / User-agent: Ask Jeeves/Teoma Disallow: / User-agent: asterias Disallow: / User-agent: b2w/0.1 Disallow: / User-agent: BabalooSpider Disallow: / User-agent: BackDoorBot/1.0 Disallow: / User-agent: BacklinkCrawler Disallow: / User-agent: Bad-Neighborhood Disallow: / User-agent: Baiduspider Disallow: / User-agent: baypup Disallow: / User-agent: BDFetch Disallow: / User-agent: BecomeBot Disallow: / User-agent: bingbot Allow: / User-agent: bitlybot Disallow: / User-agent: biwec Disallow: / User-agent: bixo Disallow: / User-agent: bixolabs Disallow: / User-agent: Black Hole Disallow: / User-agent: Blekkobot Disallow: / User-agent: BlinkaCrawler Disallow: / User-agent: BlogPulse Disallow: / User-agent: BlowFish/1.0 Disallow: / User-agent: bnf.fr_bot Disallow: / User-agent: Bookmark search tool Disallow: / User-agent: BotALot Disallow: / User-agent: botmobi Disallow: / User-agent: BotOnParade Disallow: / User-agent: bot-pge.chlooe.com Disallow: / User-agent: Browsershots Disallow: / User-agent: BuiltBotTough Disallow: / User-agent: Bullseye/1.0 Disallow: / User-agent: BunnySlippers Disallow: / User-agent: Butterfly Disallow: / User-agent: CamontSpider Disallow: / User-agent: CareerBot Disallow: / User-agent: Castabot Disallow: / User-agent: CatchBot Disallow: / User-agent: Cegbfeieh Disallow: / User-agent: ChangeDetection Disallow: / User-agent: Charlotte Disallow: / User-agent: CheeseBot Disallow: / User-agent: CherryPicker Disallow: / User-agent: CherryPickerElite/1.0 Disallow: / User-agent: CherryPickerSE/1.0 Disallow: / User-agent: cityreview Disallow: / User-agent: CligooRobot Disallow: / User-agent: coccoc Disallow: / User-agent: CompSpyBot Disallow: / User-agent: Copernic Disallow: / User-agent: copyright sheriff Disallow: / User-agent: CopyRightCheck Disallow: / User-agent: CorpusCrawler Disallow: / User-agent: cosmos Disallow: / User-agent: Covario-IDS Disallow: / User-agent: Crawler4j Disallow: / User-agent: Crescent Disallow: / User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 Disallow: / User-agent: Cutbot Disallow: / User-agent: Daumoa Disallow: / User-agent: DBLBot Disallow: / User-agent: DCPbot Disallow: / User-agent: DealGates Bot Disallow: / User-agent: DISCo Pump 3.1 Disallow: / User-agent: discoverybot Disallow: / User-agent: DittoSpyder Disallow: / User-agent: DKIMRepBot Disallow: / User-agent: DNS-Digger-Explorer Disallow: / User-agent: DomainDB Disallow: / User-agent: Dot TK – spider Disallow: / User-agent: DotBot Disallow: / User-agent: dotSemantic Disallow: / User-agent: Download Ninja Disallow: / User-agent: DripfeedBot Disallow: / User-agent: drupact Disallow: / User-agent: dumbot Disallow: / User-agent: EasyBib AutoCite Disallow: / User-agent: eCairn-Grabber Disallow: / User-agent: EdisterBot Disallow: / User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailWolf Disallow: / User-agent: Enterprise_Search Disallow: / User-agent: Enterprise_Search/1.0 Disallow: / User-agent: envolk Disallow: / User-agent: EroCrawler Disallow: / User-agent: es Disallow: / User-agent: EuripBot Disallow: / User-agent: Eurobot Disallow: / User-agent: EventGuruBot Disallow: / User-agent: EvriNid Disallow: / User-agent: Exabot Disallow: / User-agent: ExtractorPro Disallow: / User-agent: Ezooms Disallow: / User-agent: factbot Disallow: / User-agent: FairAd Client Disallow: / User-agent: FairShare Disallow: / User-agent: Falconsbot Disallow: / User-agent: FauBot Disallow: / User-agent: FeedCatBot Disallow: / User-agent: FeedFinder/bloggz.se Disallow: / User-agent: Fetch Disallow: / User-agent: findlinks Disallow: / User-agent: Flaming AttackBot Disallow: / User-agent: Flatland Industries Web Spider Disallow: / User-agent: FlightDeckReportsBot Disallow: / User-agent: Flocke bot Disallow: / User-agent: FollowSite Bot Disallow: / User-agent: Foobot Disallow: / User-agent: Fooooo_Web_Video_Crawl Disallow: / User-agent: FreeFind Disallow: / User-agent: FyberSpider Disallow: / User-agent: Gaisbot Disallow: / User-agent: GarlikCrawler Disallow: / User-agent: GetRight/4.2 Disallow: / User-agent: Gigabot Disallow: / User-agent: GingerCrawler Disallow: / User-agent: Girafabot Disallow: / User-agent: GOFORITBOT Disallow: / User-agent: gonzo Disallow: / User-agent: grapeshot Disallow: User-agent: grub Disallow: / User-agent: grub-client Disallow: / User-agent: GurujiBot Disallow: / User-agent: Hailoobot Disallow: / User-agent: Harvest/1.5 Disallow: / User-agent: Hatena Antenna Disallow: / User-agent: HatenaScreenshot Disallow: / User-agent: HeartRails_Capture Disallow: / User-agent: heritrix Disallow: / User-agent: hloader Disallow: / User-agent: Holmes Disallow: / User-agent: HolmesBot Disallow: / User-agent: HomeTags Disallow: / User-agent: HostTracker.com Disallow: / User-agent: http://www.SearchEngineWorld.com bot Disallow: / User-agent: http://www.WebmasterWorld.com bot Disallow: / User-agent: httplib Disallow: / User-agent: HTTrack Disallow: / User-agent: HuaweiSymantecSpider Disallow: / User-agent: humanlinks Disallow: / User-agent: ia_archiver Disallow: / User-agent: ia_archiver/1.6 Disallow: / User-agent: ICC-Crawler Disallow: / User-agent: ichiro Disallow: / User-agent: iCjobs Disallow: / User-agent: imbot Disallow: / User-agent: Influencebot Disallow: / User-agent: Infohelfer Disallow: / User-agent: InfoNaviRobot Disallow: / User-agent: IntegromeDB Disallow: / User-agent: Iron33/1.0.2 Disallow: / User-agent: JadynAveBot Disallow: / User-agent: JennyBot Disallow: / User-agent: Jetbot Disallow: / User-agent: Jetbot/1.0 Disallow: / User-agent: JikeSpider Disallow: / User-agent: Job Roboter Spider Disallow: / User-agent: JUST-CRAWLER Disallow: / User-agent: Jyxobot Disallow: / User-agent: Kalooga Disallow: / User-agent: Karneval-Bot Disallow: / User-agent: Kenjin Spider Disallow: / User-agent: Keyword Density/0.9 Disallow: / User-agent: KeywordDensityRobot Disallow: / User-agent: L.webis Disallow: / User-agent: larbin Disallow: / User-agent: LemurWebCrawler Disallow: / User-agent: LexiBot Disallow: / User-agent: LexxeBot Disallow: / User-agent: libWeb/clsHTTP Disallow: / User-agent: libwww Disallow: / User-agent: Lijit Disallow: / User-agent: LinguaBot Disallow: / User-agent: Linguee Bot Disallow: / User-agent: Link Valet Online Disallow: / User-agent: LinkAider Disallow: / User-agent: linkdex.com Disallow: / User-agent: LinkextractorPro Disallow: / User-agent: linko Disallow: / User-agent: LinkScan/8.1a Unix Disallow: / User-agent: LinkWalker Disallow: / User-agent: livedoor ScreenShot Disallow: / User-agent: LNSpiderguy Disallow: / User-agent: looksmart Disallow: / User-agent: lwp-trivial Disallow: / User-agent: lwp-trivial/1.34 Disallow: / User-agent: magpie-crawler Disallow: / User-agent: Mail.Ru Disallow: / User-agent: Mata Hari Disallow: / User-agent: MeMoNewsBot Disallow: / User-agent: MetaGeneratorCrawler Disallow: / User-agent: MetaJobBot Disallow: / User-agent: MetamojiCrawler Disallow: / User-agent: Metaspinner/0.01 Disallow: / User-agent: MetaURI Disallow: / User-agent: MIA Bot Disallow: / User-agent: Microsoft URL Control Disallow: / User-agent: Microsoft URL Control – 5.01.4511 Disallow: / User-agent: Microsoft URL Control – 6.00.8169 Disallow: / User-agent: Microsoft.URL.Control Disallow: / User-agent: MIIxpc Disallow: / User-agent: MIIxpc/4.2 Disallow: / User-agent: Mister PiX Disallow: / User-agent: MLBot Disallow: / User-agent: MnoGoSearch Disallow: / User-agent: moba-crawler Disallow: / User-agent: moget Disallow: / User-agent: moget/2.1 Disallow: / User-agent: MojeekBot Disallow: / User-agent: Motoricerca-Robots.txt-Checker Disallow: / User-agent: mozilla/3 Disallow: / User-agent: mozilla/4 Disallow: / User-agent: Mozilla/4.0 (compatible; BullsEye; Windows 95) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 2000) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 95) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 98) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows NT) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows XP) Disallow: / User-agent: Mp3Bot Disallow: / User-agent: MSIECrawler Disallow: / User-agent: MSRBOT Disallow: / User-agent: Najdi.si Disallow: / User-agent: naver Disallow: / User-agent: NaverBot Disallow: / User-agent: NerdByNature.Bot Disallow: / User-agent: NetAnts Disallow: / User-agent: NetAttache Disallow: / User-agent: NetAttache Light 1.1 Disallow: / User-agent: netEstate Crawler Disallow: / User-agent: NetMechanic Disallow: / User-agent: NetResearchServer Disallow: / User-agent: Netseer Disallow: / User-agent: NextGenSearchBot Disallow: / User-agent: NICErsPRO Disallow: / User-agent: Nigma.ru Disallow: / User-agent: nodestackbot Disallow: / User-agent: NPBot Disallow: / User-agent: NPBot-1/2.0 Disallow: / User-agent: Nuhk Disallow: / User-agent: Nutch Disallow: / User-agent: nworm Disallow: / User-agent: Nymesis Disallow: / User-agent: oBot Disallow: / User-agent: Ocelli Disallow: / User-agent: Offline Explorer Disallow: / User-agent: OmniExplorer_Bot Disallow: / User-agent: Openbot Disallow: /
Changes recorded since June 2026.
No change recorded.