CrawlerPulse
CrawlerPulse / Oceania / Australia / abc.net.au

ABC

abc.net.au · Australia · ABC (AU) · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Blocked
OpenAI
Blocked
Anthropic
Blocked
Perplexity
Blocked
Meta
Blocked
Common Crawl
Blocked
ByteDance
Blocked
Apple
Allowed
Amazon
Allowed

robots.txt

As read on 8 Oct 2026, 2,311 bytes. 16 rule groups name AI crawlers; shown first.

User-agent: ChatGPT-User
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-Web
Disallow: /

User-agent: Timpibot
Disallow: /

User-agent: omgili
Disallow: /

User-agent: cohere-ai
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Diffbot
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Omgilibot
Disallow: /

User-agent: YouBot
Disallow: /
Sitemap: https://www.abc.net.au/sitemaps/sitemap-index.xml.gz

User-agent: *
Disallow: /classic/contact/concerts.htm
Disallow: /classic/contact/default.htm
Disallow: /classic/contact/eventsdiary.htm
Disallow: /classic/contact/formerror.htm
Disallow: /classic/contact/formthanks.htm
Disallow: /classic/contact/general.htm
Disallow: /classic/contact/limelight.htm
Disallow: /classic/contact/mailinglist.htm
Disallow: /classic/contact/music.htm
Disallow: /classic/contact/presenter.htm
Disallow: /classic/contact/website.htm
Disallow: /classic/contact/word.htm
Disallow: /xmlcontent/
Disallow: /classicfm/
Disallow: /iview/
Disallow: /site-archive/
Disallow: /corp/
Disallow: /contact/
Disallow: /homepage/2013/
Disallow: /beta/
Disallow: /abc4000/
Disallow: /res/

User-agent: Googlebot
Crawl-delay: 5

User-agent: Googlebot-Image
Crawl-delay: 5

User-agent: Bingbot
Crawl-delay: 5

User-agent: Slurp
Crawl-delay: 5

User-agent: FlipboardProxy
Crawl-delay: 2
Disallow: /news/image/

User-agent: iSec_Bot
Disallow: /

User-agent: TurnitinBot
Disallow: /

User-agent: ICC-Crawler
Disallow: /

User-agent: trendkite-akashic-crawler
Disallow: /

User-agent: TinEye-bot
Disallow: /

User-agent: R6_CommentReader
Disallow: /

User-agent: BLEXBot
Disallow: /

User-agent: Nutch
Disallow: /

User-agent: Webzio-Extended
Disallow: /

User-agent: *
Disallow: /gives*?query=*
Disallow: /gives*&query=*

History

Changes recorded since June 2026.

No change recorded.

In context

Australia
34.0% block at least one AI crawler
Behind Cloudflare
No