pocket-lint.com · United Kingdom · measured 9 Oct 2026
As read on 9 Oct 2026, 1,979 bytes. 1 rule group name AI crawlers; shown first.
User-agent: "008" User-agent: anthropic-ai User-agent: Applebot-extended User-agent: Bytespider User-agent: CCBot User-agent: ChatGPT-User User-agent: Claude-Web User-agent: ClaudeBot User-agent: cohere-ai User-agent: Diffbot User-agent: FacebookBot User-agent: GPTBot User-agent: HTTrack User-agent: Nutch User-agent: Offline Explorer User-agent: omgili User-agent: Scrapy User-agent: YouBot Disallow: / Sitemap: https://www.pocket-lint.com/sitemap.xml Sitemap: https://www.pocket-lint.com/post_google_news.xml User-agent: * Disallow: /admin/ Disallow: /api/auth Disallow: /api/v1 Disallow: /api/v2 Disallow: /api/v3 Disallow: /author/*/page/ Disallow: /comment-fetch/ Disallow: /comment-reply-fetch/ Disallow: /fetch/ Disallow: /modules-fetch/ Disallow: /mymodule/tag/ Disallow: /mymodule/vehicle-sub-model-year-tag-trim/compare/ Disallow: /mymodule/vehicle-sub-model-year-tag/ Disallow: /*pixel.png Disallow: /profile/ Disallow: /public/build/quill Disallow: /search/ Disallow: /tag/interaction-card/ Disallow: /tag/search/db-home/ Disallow: /tag/search/listing-page/ Disallow: /tldr/ Disallow: /wordpress/wp-content/
Changes recorded since June 2026.
No change recorded.