CrawlerPulse
CrawlerPulse / Oceania / Australia / australianpropertyjournal.com.au

Australian Property Journal

australianpropertyjournal.com.au · Australia · AU Property Journal · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Allowed
OpenAI
Blocked
Anthropic
Allowed
Perplexity
Allowed
Meta
Allowed
Common Crawl
Allowed
ByteDance
Blocked
Apple
Allowed
Amazon
Blocked

robots.txt

As read on 8 Oct 2026, 1,706 bytes. 3 rule groups name AI crawlers; shown first.

User-agent: GPTBot
Disallow: /
Disallow: /wp-login.php

User-agent: Amazonbot
Disallow: /
Disallow: /wp-login.php

User-agent: Bytespider
Disallow: /
Disallow: /wp-login.php

User-agent: SemrushBot
Disallow: /
Disallow: /wp-login.php

User-agent: AhrefsBot
Disallow: /
Disallow: /wp-login.php

User-agent: MJ12Bot
Disallow: /
Disallow: /wp-login.php

User-agent: DotBot
Disallow: /
Disallow: /wp-login.php

User-agent: RogerBot
Disallow: /
Disallow: /wp-login.php

User-agent: HeadlessChrome
Disallow: /
Disallow: /wp-login.php

User-agent: PhantomJS
Disallow: /
Disallow: /wp-login.php

User-agent: Selenium
Disallow: /
Disallow: /wp-login.php

User-agent: nmap
Disallow: /
Disallow: /wp-login.php

User-agent: masscan
Disallow: /
Disallow: /wp-login.php

User-agent: sqlmap
Disallow: /
Disallow: /wp-login.php

User-agent: Googlebot
Allow: /
Disallow: /wp-login.php

User-agent: Bingbot
Allow: /
Disallow: /wp-login.php

User-agent: Baiduspider
Allow: /
Disallow: /wp-login.php

User-agent: YandexBot
Allow: /
Disallow: /wp-login.php

User-agent: FacebookExternalHit
Allow: /
Disallow: /wp-login.php

User-agent: TwitterBot
Allow: /
Disallow: /wp-login.php

User-agent: LinkedInBot
Allow: /
Disallow: /wp-login.php

User-agent: UptimeRobot
Allow: /
Disallow: /wp-login.php

User-agent: Pingdom
Allow: /
Disallow: /wp-login.php

User-agent: ms-office
Crawl-delay: 10
Disallow: /wp-login.php

User-agent: MSOffice
Crawl-delay: 10
Disallow: /wp-login.php

User-agent: *
Crawl-delay: 5
Disallow: /wp-login.php
Sitemap: https://greenstreetnews.com/sitemap_index.xml

History

Changes recorded since June 2026.

No change recorded.

In context

Australia
34.0% block at least one AI crawler
Behind Cloudflare
No