CrawlerPulse
CrawlerPulse / Europe / United Kingdom / thescottishsun.co.uk

The Scottish Sun

thescottishsun.co.uk · United Kingdom · News UK · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Blocked
OpenAI
Allowed
Anthropic
Blocked
Perplexity
Blocked
Meta
Blocked
Common Crawl
Blocked
ByteDance
Blocked
Apple
Allowed
Amazon
Blocked

robots.txt

As read on 8 Oct 2026, 5,039 bytes. 1 rule group name AI crawlers; shown first.

User-agent: Googlebot
User-agent: Googlebot-Image
User-agent: Googlebot-Video
User-agent: Googlebot-News
User-agent: Google-InspectionTool
User-agent: GoogleOther
User-agent: APIs-Google
User-agent: Mediapartners-Google
User-agent: DuckDuckBot
User-agent: Slurp
User-agent: Twitterbot
User-agent: Bingbot
User-agent: adidxbot
User-agent: MicrosoftPreview
User-agent: Applebot
User-agent: AdsBot-Google
User-agent: AmazonAdBot
User-agent: facebookexternalhit
User-agent: Yahoo Ad Monitoring
User-agent: parse.ly scraper
User-agent: Screaming Frog SEO Spider
User-agent: SemrushBot
User-agent: OnCrawl
User-agent: Chrome-Lighthouse
User-agent: Slackbot-LinkExpanding
User-agent: Slack-ImgProxy
User-agent: Slackbot
User-agent: Snap URL Preview Service
User-agent: proximic
User-agent: GumGum-Bot
User-agent: TTD-Content
User-agent: Pubmatic
User-agent: ias_crawler
User-agent: outbrain
User-agent: Leikibot
User-agent: weborama-fetcher
User-agent: AHC/2.1
User-agent: SirdataBot
User-agent: peer39_crawler/1.0
User-agent: EchoboxBot/1.0
User-agent: CriteoBot/0.1
User-agent: AdDefend Site Context Crawler/1.0
User-agent: SmartologyBot
User-agent: Dianomi
User-agent: ChatGPT-User
User-agent: GPTBot
User-Agent: OAI-SearchBot
Allow: /
Allow: /ads.txt
Allow: /wp-admin/admin-ajax.php
Disallow: /*horseracing/racecards/2019*
Disallow: /*horseracing/racecards/2020*
Disallow: /*horseracing/racecards/2021*
Disallow: /*horseracing/racecards/2022*
Disallow: /*horseracing/racecards/2023*
Disallow: /*horseracing/racecards/2024*
Disallow: /*horseracing/results/2019*
Disallow: /*horseracing/results/2020*
Disallow: /*horseracing/results/2021*
Disallow: /*horseracing/results/2022*
Disallow: /*horseracing/results/2023*
Disallow: /*horseracing/results/2024*
Disallow: /simwidgets/
Disallow: /player/brightcove/
Disallow: /*?s=*
Disallow: *&s=*
Disallow: /?p=*
Disallow: /sso/login/
Disallow: /wp-login.php
Disallow: /wp-admin/
Disallow: /amp-tealium/
Disallow: /archives/
Disallow: /search/
Disallow: /horoscope/
Disallow: /*?x=
Disallow: /*?action=
Disallow: /*&action=
Disallow: /*?adid=
Disallow: /*&adid=
Disallow: /*?bitrate=
Disallow: /*&bitrate=
Disallow: /*?channel=
Disallow: /*&channel=
Disallow: /*?clipid=
Disallow: /*&clipid=
Disallow: /*?contentCollection=
Disallow: /*&contentCollection=
Disallow: /*?contentId=
Disallow: /*&contentId=
Disallow: /*?ei=
Disallow: /*&ei=
Disallow: /*?format=
Disallow: /*&format=
Disallow: /*?height=
Disallow: /*&height=
Disallow: /*?iframe=
Disallow: /*&iframe=
Disallow: /*?mediaId=
Disallow: /*&mediaId=
Disallow: /*?module=
Disallow: /*&module=
Disallow: /*?monthnum=
Disallow: /*&monthnum=
Disallow: /*?pgtype=
Disallow: /*&pgtype=
Disallow: /*?preview=
Disallow: /*&preview=
Disallow: /*?priority=
Disallow: /*&priority=
Disallow: /*?sa=
Disallow: /*&sa=
Disallow: /*?section=
Disallow: /*&section=
Disallow: /*?share=
Disallow: /*&share=
Disallow: /*?TB_iframe=
Disallow: /*&TB_iframe=
Disallow: /*?TBL_Prod100Page=
Disallow: /*&TBL_Prod100Page=
Disallow: /*?theme_preview=
Disallow: /*&theme_preview=
Disallow: /*?usg=
Disallow: /*&usg=
Disallow: /*?ved=
Disallow: /*&ved=
Disallow: /*?version=
Disallow: /*&version=
Disallow: /*?VID=
Disallow: /*&VID=
Disallow: /*?vxBitrate=
Disallow: /*&vxBitrate=
Disallow: /*?vxChannel=
Disallow: /*&vxChannel=
Disallow: /*?vxClipId=
Disallow: /*&vxClipId=
Disallow: /*?vxSiteId=
Disallow: /*&vxSiteId=
Disallow: /*?width=
Disallow: /*&width=
Disallow: /*?year=
Disallow: /*&year=
Disallow: /*.xml?uuid=
Disallow: /*?post_type=feedback&p=
Disallow: /*?wptouch_preview_theme=
Disallow: /*>
Disallow: /?attachment_id=
Disallow: /affiliate/
Disallow: /ai_log/
Disallow: /*?*pg99.asia*
Disallow: /*&*pg99.asia*
Disallow: /*?*gg155.cn*
Disallow: /*&*gg155.cn*
Disallow: /*?*To66.Asia*
Disallow: /*&*To66.Asia*

User-agent: *
Disallow: /
Sitemap: https://www.thescottishsun.co.uk/sitemap.xml
Sitemap: https://www.thescottishsun.co.uk/news-sitemap.xml
Sitemap: https://www.thescottishsun.co.uk/sitemap-nypost-authors.xml
Sitemap: https://www.thescottishsun.co.uk/sitemap-nypost-post_tag.xml
Sitemap: https://www.thescottishsun.co.uk/sitemap-nypost-where.xml
Sitemap: https://www.thescottishsun.co.uk/sitemap-nypost-who.xml
Sitemap: https://www.thescottishsun.co.uk/sitemap-nypost-feeds.xml
Sitemap: https://www.thescottishsun.co.uk/sitemap-nypost-category.xml
Sitemap: https://www.thescottishsun.co.uk/sitemap-nypost-football-team-taxo.xml
Sitemap: https://www.thescottishsun.co.uk/sitemap-nypost-vertical-video.xml
Sitemap: https://www.thescottishsun.co.uk/sitemap-nypost-video.xml
Sitemap: https://www.thescottishsun.co.uk/sitemap-nypost-video-shows.xml

History

Changes recorded since June 2026.

No change recorded.

In context

United Kingdom
73.8% block at least one AI crawler
Behind Cloudflare
No
Same group