CrawlerPulse
CrawlerPulse / Oceania / Australia / theurbanlist.com

Urban List

theurbanlist.com · Australia · Urban List Pty Ltd · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Allowed
OpenAI
Allowed
Anthropic
Allowed
Perplexity
Allowed
Meta
Allowed
Common Crawl
Allowed
ByteDance
Allowed
Apple
Allowed
Amazon
Allowed

robots.txt

As read on 8 Oct 2026, 1,240 bytes. No rule names an AI crawler.

Sitemap: https://www.theurbanlist.com/brisbane/sitemap
Sitemap: https://www.theurbanlist.com/sydney/sitemap
Sitemap: https://www.theurbanlist.com/melbourne/sitemap
Sitemap: https://www.theurbanlist.com/perth/sitemap
Sitemap: https://www.theurbanlist.com/new-zealand/sitemap
Sitemap: https://www.theurbanlist.com/goldcoast/sitemap
Sitemap: https://www.theurbanlist.com/sunshinecoast/sitemap
Sitemap: https://www.theurbanlist.com/sitemap
Sitemap: https://www.theurbanlist.com/singapore/sitemap
Sitemap: https://www.theurbanlist.com/byron-bay/sitemap
Sitemap: https://www.theurbanlist.com/townsville/sitemap
Sitemap: https://www.theurbanlist.com/adelaide/sitemap
Sitemap: https://www.theurbanlist.com/hobart/sitemap
Sitemap: https://www.theurbanlist.com/wellington/sitemap
Sitemap: https://www.theurbanlist.com/christchurch/sitemap
Sitemap: https://www.theurbanlist.com/queenstown/sitemap
Sitemap: https://www.theurbanlist.com/auckland/sitemap
Sitemap: https://www.theurbanlist.com/bali/sitemap
Sitemap: https://www.theurbanlist.com/barcelona/sitemap
Sitemap: https://www.theurbanlist.com/rome/sitemap
Sitemap: https://www.theurbanlist.com/osaka/sitemap

User-agent: *
Disallow: /system/*

History

Changes recorded since June 2026.

No change recorded.

In context

Australia
34.0% block at least one AI crawler
Behind Cloudflare
Yes