CrawlerPulse
CrawlerPulse / Europe / United Kingdom / sportsmole.co.uk

Sports Mole

sportsmole.co.uk · United Kingdom · measured 9 Oct 2026

Google Search
Allowed
Google-Extended
Allowed
OpenAI
Allowed
Anthropic
Allowed
Perplexity
Allowed
Meta
Allowed
Common Crawl
Allowed
ByteDance
Allowed
Apple
Allowed
Amazon
Allowed

robots.txt

As read on 9 Oct 2026, 1,132 bytes. No rule names an AI crawler.

Sitemap: https://www.sportsmole.co.uk/sitemap_index.xml
Sitemap: https://www.sportsmole.co.uk/online/sitemap_index.xml

User-agent: *
Disallow: *.amp_latest_update_time
Disallow: /fwd/*
Disallow: /go/*
Disallow: /go/
Disallow: */go/*
Disallow: /esi/*
Disallow: /sw3/
Disallow: /search/
Disallow: /preview/preview-index/
Disallow: *_photo_*.html
Disallow: /online/wp/wp-admin/admin-ajax.php?action=rank_math_overlay_thumb&*
Disallow: /*?nocache
Disallow: /*?flushcache
Disallow: /*?nostatic
Disallow: /*?flushstatic
Disallow: /people/*.xml
Disallow: */&AbvOwpVEEeWu3hKp/
Disallow: */&force=1
Disallow: */{{url3}}
Disallow: *&vnmq4na9eewcjbkp/
Disallow: /online/app/themes/sportsmole/public/lottie/burger-menu.*.json
Disallow: */?t=L
Disallow: /editors_picks.xml
Disallow: /amp_ping/
Disallow: /amp_list/
Disallow: /ar/
Disallow: /de/
Disallow: /entertainment/
Disallow: /es-MX/
Disallow: /es/
Disallow: /fr/
Disallow: /hi/
Disallow: /id/
Disallow: /it/
Disallow: /ms/
Disallow: /nl/
Disallow: /pt/
Disallow: /th/
Disallow: /tl/
Disallow: /zh/
Disallow: */{{URL_
Disallow: /admin

History

Changes recorded since June 2026.

No change recorded.

In context

United Kingdom
73.8% block at least one AI crawler
Behind Cloudflare
Yes