CrawlerPulse
CrawlerPulse / Europe / Germany / pcgames.de

PC Games

pcgames.de · Germany · Computec Media GmbH · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Blocked
OpenAI
Blocked
Anthropic
Allowed
Perplexity
Allowed
Meta
Allowed
Common Crawl
Blocked
ByteDance
Allowed
Apple
Blocked
Amazon
Allowed

robots.txt

As read on 8 Oct 2026, 8,460 bytes. 6 rule groups name AI crawlers; shown first.

User-agent: CCBot
User-agent: 008
User-agent: 80bot
User-agent: ACONTBOT
User-agent: AMZNKAssocBot
User-agent: AboutUsBot
User-agent: Acoon-Robot
User-agent: Advista
User-agent: Aqua_Products
User-agent: BackDoorBot/1.0
User-agent: Baiduspider
User-agent: BlogPulseLive
User-agent: BlowFish/1.0
User-agent: Bookmark search tool
User-agent: BotALot
User-agent: BuiltBotTough
User-agent: Bullseye/1.0
User-agent: BunnySlippers
User-agent: CFNetwork
User-agent: Cabot
User-agent: CazoodleBot
User-agent: Charlotte
User-agent: CheeseBot
User-agent: CherryPicker
User-agent: CherryPickerElite/1.0
User-agent: CherryPickerSE/1.0
User-agent: CiZilla
User-agent: CopyRightCheck
User-agent: Crescent
User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
User-agent: CrystalSemanticsBot
User-agent: Custo
User-agent: DittoSpyder
User-agent: DjangoTraineeBot
User-agent: Download Ninja
User-agent: EmailCollector
User-agent: EmailSiphon
User-agent: EmailWolf
User-agent: EroCrawler
User-agent: Euripbot
User-agent: Eurobot
User-agent: ExDomain
User-agent: Exabot
User-agent: ExtractorPro
User-agent: FairAd Client
User-agent: FairShare
User-agent: Fetch
User-agent: Flaming AttackBot
User-agent: Foobot
User-agent: Gaisbot
User-agent: Galbot
User-agent: GetRight/4.2
User-agent: Gigabot
User-agent: HTTrack
User-agent: Harvest/1.5
User-agent: HeinrichderMiragoRobot
User-agent: IRLbot
User-agent: InfoNaviRobot
User-agent: Iron33/1.0.2
User-agent: JennyBot
User-agent: JobRoboter
User-agent: Kenjin Spider
User-agent: Keyword Density/0.9
User-agent: LNSpiderguy
User-agent: LexiBot
User-agent: Linguee
User-agent: LinkScan/8.1a Unix
User-agent: LinkWalker
User-agent: LinkextractorPro
User-agent: MIIxpc
User-agent: MIIxpc/4.2
User-agent: MSIECrawler
User-agent: MagpieRSS
User-agent: Mail.Ru
User-agent: Mata Hari
User-agent: Meltwater
User-agent: Microsoft URL Control
User-agent: Microsoft URL Control - 5.01.4511
User-agent: Microsoft URL Control - 6.00.8169
User-agent: Microsoft.URL.Control
User-agent: Mister PiX
User-agent: Mozilla/4.0 (compatible; BullsEye; Windows 95)
User-agent: NICErsPRO
User-agent: Nebullabot
User-agent: NerdByNature.Bot
User-agent: NetAnts
User-agent: NetMechanic
User-agent: Netluchs
User-agent: Nutch
User-agent: Ocelli
User-agent: Offline Explorer
User-agent: Openbot
User-agent: Openfind
User-agent: Openfind data gathere
User-agent: Oracle Ultra Search
User-agent: PagesInventory
User-agent: PerMan
User-agent: PeterBot
User-agent: Plukkie
User-agent: ProPowerBot/2.14
User-agent: ProWebWalker
User-agent: Python-urllib
User-agent: QueryN Metasearch
User-agent: RMA
User-agent: Radiation Retriever 1.1
User-agent: RepoMonkey
User-agent: RepoMonkey Bait & Tackle/v1.01
User-agent: Ruky-Bot
User-agent: SEOkicks-Robot
User-agent: ScoutJet
User-agent: Sensis Web Crawler
User-agent: SiteSnagger
User-agent: Snapbot
User-agent: Snoopy
User-agent: Sosospider
User-agent: SpankBot
User-agent: Speedy
User-agent: SurveyBot
User-agent: SurveyBot_IgnoreIP
User-agent: Szukacz/1.4
User-agent: Tasapspider
User-agent: Teleport
User-agent: TeleportPro
User-agent: Telesoft
User-agent: TencentTraveler
User-agent: The Intraformant
User-agent: TheNomad
User-agent: Touche
User-agent: True_Robot
User-agent: True_Robot/1.0
User-agent: URL Control
User-agent: URL_Spider_Pro
User-agent: URLy Warning
User-agent: VCI
User-agent: VCI WebViewer VCI WebViewer Win32
User-agent: WWW-Collector-E
User-agent: Web Image Collector
User-agent: WebAuto
User-agent: WebBandit
User-agent: WebBandit/3.50
User-agent: WebCopier
User-agent: WebEnhancer
User-agent: WebReaper
User-agent: WebSauger
User-agent: WebStripper
User-agent: WebZIP
User-agent: WebZip
User-agent: WebZip/4.0
User-agent: WebmasterCoffee
User-agent: WebmasterWorldForumBot
User-agent: Website Quester
User-agent: Webster Pro
User-agent: Wget
User-agent: Wget/1.5.3
User-agent: Wget/1.6
User-agent: Xenu
User-agent: Xenu's
User-agent: Xenu's Link Sleuth 1.1c
User-agent: Yandex bot
User-agent: YoudaoBot
User-agent: Zealbot
User-agent: Zeus
User-agent: Zeus 32297 Webster Pro V2.9 Win32
User-agent: Zeus Link Scout
User-agent: ZyBORG
User-agent: adnbot
User-agent: asterias
User-agent: b2w/0.1
User-agent: cityreview
User-agent: cosmos
User-agent: crawly
User-agent: discobot
User-agent: dotbot
User-agent: dotbot
User-agent: echobot
User-agent: envolk
User-agent: eventax
User-agent: findlinks
User-agent: flatlandbot
User-agent: fyberspider
User-agent: gonzo*
User-agent: grub
User-agent: grub-client
User-agent: heise-IT-Markt-Crawler
User-agent: hloader
User-agent: httplib
User-agent: humanlinks
User-agent: iCcrawler
User-agent: ia_archiver
User-agent: ia_archiver/1.6
User-agent: ichiro
User-agent: iearthworm
User-agent: infometrics-bot
User-agent: jobs.de-Robot
User-agent: kalooga
User-agent: larbin
User-agent: laycat
User-agent: libWeb/clsHTTP
User-agent: libwww
User-agent: linko
User-agent: looksmart
User-agent: lwp-request
User-agent: lwp-trivial
User-agent: lwp-trivial/1.34
User-agent: moget
User-agent: moget/2.1
User-agent: naughtyrobot
User-agent: psbot
User-agent: search17
User-agent: searchlink
User-agent: searchpreview
User-agent: semager
User-agent: sitecheck.internetseer.com
User-agent: spanner
User-agent: stalker
User-agent: suggybot
User-agent: suzuran
User-agent: thesubot
User-agent: toCrawl/UrlDispatcher
User-agent: trendiction
User-agent: turingos
User-agent: uipbot
User-agent: uipbot/1.0 (uipbot@semasio.net)
User-agent: wget
User-agent: woriobot
User-agent: heritrix
User-agent: trendictionbot
User-agent: sentibot
User-agent: Landau-Media-Spider
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Slurp
Crawl-delay: 18

User-Agent: AhrefsBot
Crawl-Delay: 2

User-agent: *
Content-signal: search=yes,ai-train=no
Disallow: /preisvergleich/
Disallow: /Fun-Stuff-Thema-216569/Bilder/
Disallow: /external/cose/
Disallow: /Computec25/
Disallow: /*?menu=
Disallow: /Suche/
Disallow: /commoncfm/flowneu/
Disallow: /commoncfm/sharer.php
Disallow: /datenschutz/
Disallow: /Alle-Artikel/
Disallow: /Artikel-Archiv/
Disallow: /Tags/
Disallow: /tags/
Disallow: /external/newsletter_v3/
Disallow: /common/pageflip/
Disallow: /webservices/aref/
Sitemap: https://www.pcgames.de/sitemapIndex.xml
Sitemap: https://www.pcgames.de/sitemapNews.xml

History

Changes recorded since June 2026.

No change recorded.

In context

Germany
71.7% block at least one AI crawler
Behind Cloudflare
Yes
Same group