caerphilly.observer · United Kingdom · measured 8 Oct 2026
As read on 8 Oct 2026, 1,200 bytes. 1 rule group name AI crawlers; shown first.
User-agent: GPTBot User-agent: CCBot User-agent: TurnitinBot User-agent: PetalBot User-agent: MoodleBot User-agent: ImagesiftBot User-agent: Bytespider User-agent: Scrapy User-agent: MetaInspector User-agent: 008 User-agent: newspaper User-agent: Nutch User-agent: Timpibot User-agent: ExaSearchBot User-agent: LinkupBot User-agent: DotBot User-agent: SemrushBot User-agent: trendictionbot User-agent: DataGardenerBot Disallow: / Sitemap: https://caerphilly.observer/sitemap_index.xml Sitemap: https://caerphilly.observer/news-sitemap.xml User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Disallow: /off-stone/ Disallow: /?s= Disallow: /*?s= Disallow: /news/*/feed/ Disallow: /sport/*/feed/ Disallow: /opinion/*/feed/ Disallow: /notices/*/feed/ Disallow: /event/*/feed/ Disallow: /files/jb.vcf Disallow: /files/lhb.vcf Disallow: /files/rg.vcf Disallow: /files/rw.vcf
Changes recorded since June 2026.
No change recorded.