CrawlerPulse
CrawlerPulse / Europe / Netherlands / dutchnews.nl

dutchnews

dutchnews.nl · Netherlands · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Allowed
OpenAI
Blocked
Anthropic
Allowed
Perplexity
Allowed
Meta
Allowed
Common Crawl
Allowed
ByteDance
Allowed
Apple
Allowed
Amazon
Allowed

robots.txt

As read on 8 Oct 2026, 2,007 bytes. 1 rule group name AI crawlers; shown first.

User-agent: GPTBot
Disallow: /

User-agent: *
Crawl-delay:10
Disallow: /cgi-bin/
Disallow: /_werkmap/
Disallow: /adbanners/
Disallow: /admin/
Disallow: /adverts/
Disallow: /bounces/
Disallow: /captcha/
Disallow: /columns/templates/
Disallow: /columns/templates4/
Disallow: /css/
Disallow: /dictionary/templates/
Disallow: /dictionary/templates4/
Disallow: /elections/templates/
Disallow: /elections/templates4/
Disallow: /expatdiary/templates/
Disallow: /expatdiary/templates4/
Disallow: /expatpages/templates/
Disallow: /expatpages/templates4/
Disallow: /fckeditor/
Disallow: /features/templates/
Disallow: /features/templates4/
Disallow: /feed/
Disallow: /feed_oud/
Disallow: /files/
Disallow: /friends/directpayment.php
Disallow: /friends/subscribe.php
Disallow: /friends/unsubscribe.php
Disallow: /googlesearch_niet/
Disallow: /houseinholland/indexnew.php
Disallow: /jobsinholland/templates/
Disallow: /jobsinholland/templates4/
Disallow: /mt3/
Disallow: /mt4/
Disallow: /news/rssfulltext.xml
Disallow: /news/archives/print/
Disallow: /news/templates/
Disallow: /news/templates4/
Disallow: /nw_expatpages/
Disallow: /oud_adverts/
Disallow: /oud_columns/
Disallow: /oud_dictionary/
Disallow: /oud_elections/
Disallow: /oud_expatdiary/
Disallow: /oud_features/
Disallow: /oud_local-elections-2010/
Disallow: /oud_news/
Disallow: /oud_newsletter/
Disallow: /oud_people/
Disallow: /oud_weather/
Disallow: /oud_whatsonholland/
Disallow: /people/templates/
Disallow: /people/templates4/
Disallow: /picture_library/
Disallow: /plesk-stat/
Disallow: /scripts/
Disallow: /test/
Disallow: /testblog/
Disallow: /testweblog/
Disallow: /webmaster/
Disallow: /webstats/
Disallow: /whatsonholland/templates/
Disallow: /whatsonholland/templates4/
Disallow: /xxx/
Disallow: /xxxtest/
Disallow: /feed/
Disallow: /trackback/
Disallow: /wp-admin/
Disallow: /wp-content/
Disallow: /wp-includes/
Disallow: /xmlrpc.php
Disallow: /wp-
Disallow: /webmail
Allow: /wp-content/uploads/

History

Changes recorded since June 2026.

No change recorded.

In context

Netherlands
49.4% block at least one AI crawler
Behind Cloudflare
No