CrawlerPulse
CrawlerPulse / Europe / United Kingdom / inyourarea.co.uk

inyourarea.co.uk

inyourarea.co.uk · United Kingdom · Reach plc · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Allowed
OpenAI
Blocked
Anthropic
Blocked
Perplexity
Blocked
Meta
Blocked
Common Crawl
Blocked
ByteDance
Allowed
Apple
Blocked
Amazon
Allowed

robots.txt

As read on 8 Oct 2026, 1,095 bytes. 10 rule groups name AI crawlers; shown first.

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: YouBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Claude-Web
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

User-agent: *
Sitemap: https://www.inyourarea.co.uk/sitemaps/map_news.xml
Sitemap: https://www.inyourarea.co.uk/sitemaps/sitemap_index.xml
Disallow: /ghost/
Disallow: /iframe*
Disallow: /onboarding/confirm-code/
Disallow: /content-plus/
Disallow: /feed/*/live/apex
Disallow: /contentplus/link/

User-agent: Googlebot
Disallow: /feed/

User-agent: Googlebot-News
Disallow: *
Allow: /news/

User-agent: daumoa
Disallow: /

User-agent: Sosospider
Disallow: /

User-agent: rogerbot
Disallow: /

User-agent: semetrical
Disallow: /

User-agent: Googlebot-News
Disallow: /thedavedesk/

User-agent: AhrefsBot
Disallow: /

User-agent: Perplexity-ai
Disallow: /

User-agent: grapeshot
Crawl-delay: 0

User-agent: bingbot
Crawl-delay: 1

History

Changes recorded since June 2026.

No change recorded.

In context