CrawlerPulse
CrawlerPulse / Europe / United Kingdom / gbnews.com

gbnews.com

gbnews.com · United Kingdom · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Allowed
OpenAI
Allowed
Anthropic
Allowed
Perplexity
Allowed
Meta
Allowed
Common Crawl
Allowed
ByteDance
Allowed
Apple
Allowed
Amazon
Allowed

robots.txt

As read on 8 Oct 2026, 1,475 bytes. No rule names an AI crawler.

User-agent: adsbot-google
Disallow: /*lab_viewport*

User-agent: *
Disallow: /core/*
Disallow: /r/*
Allow: /r/kappa/api/*
Disallow: /mnt/*
Disallow: /radioplayer?*
Disallow: *?q=
Disallow: *debugld=stickybtest*
Disallow: *_ptid=*
Disallow: /*lab_viewport*
Disallow: /res/custom_page/data.js?resource_id=generic&layout_name=ForYou*
Disallow: /res/custom_page/data.js?resource_id=generic&layout_name=YourNextRead*
Disallow: *?dp=*
Disallow: *&dp=*
Disallow: *?utm_=*
Disallow: *&utm_=*
Disallow: *?fbclid=*
Disallow: *&fbclid=*
Disallow: *?ico=*
Disallow: *&ico=*
Disallow: *&hpp=*
Disallow: *?hpp=*
sitemap: https://www.gbnews.com/sitemap.xml
sitemap: https://www.gbnews.com/sitemap_news.xml
sitemap: https://www.gbnews.com/r/kappa/api/v2/reader/news_1.xml
sitemap: https://www.gbnews.com/sitemap_video.xml
sitemap: https://www.gbnews.com/sitemap_sections.xml
sitemap: https://www.gbnews.com/feeds/sitemaps/sitemap_1.xml
sitemap: https://www.gbnews.com/feeds/sitemaps/news_1.xml
sitemap: https://www.gbnews.com/feeds/editorial.rss
sitemap: https://www.gbnews.com/feeds/sport.rss
sitemap: https://www.gbnews.com/feeds/celebrity.rss
sitemap: https://www.gbnews.com/feeds/travel.rss
sitemap: https://www.gbnews.com/feeds/money.rss
sitemap: https://www.gbnews.com/feeds/weather.rss
sitemap: https://www.gbnews.com/feeds/politics.rss
sitemap: https://www.gbnews.com/feeds/royal.rss
sitemap: https://www.gbnews.com/feeds/opinion.rss
sitemap: https://www.gbnews.com/feeds/news.rss

History

Changes recorded since June 2026.

No change recorded.

In context

United Kingdom
73.8% block at least one AI crawler
Behind Cloudflare
No