CrawlerPulse
CrawlerPulse / Oceania / Australia / taste.com.au

taste.com.au

taste.com.au · Australia · News Corp AU · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Blocked
OpenAI
Blocked
Anthropic
Blocked
Perplexity
Blocked
Meta
Blocked
Common Crawl
Blocked
ByteDance
Blocked
Apple
Allowed
Amazon
Blocked

robots.txt

As read on 8 Oct 2026, 4,566 bytes. 2 rule groups name AI crawlers; shown first.

User-agent: ScreamingFrogSEOSpider
User-agent: ComscoreCrawler
User-agent: Moatbot
User-agent: Adbeat
User-agent: integralads.com
User-agent: Expo9
User-agent: Outbrain
User-agent: semasio.net
User-agent: Weborama-fetcher
User-agent: YahooAdmonitoring
User-agent: netseer.com
User-agent: SemrushBot
User-agent: omgili.com
User-agent: FacebookCrawler
User-agent: facebookexternalhit
User-agent: facebot
User-agent: ias_crawler
User-agent: ias_wombles
User-agent: IASbot
User-agent: identrics
User-agent: MeltwaterNews
User-agent: links.streem.com.au
User-agent: Webz-bot
User-agent: LinkedInBot
User-agent: WhatsApp
User-agent: redditbot
User-agent: Mediapartners-Google
User-agent: Google-InspectionTool
User-agent: Adsbot-Google
User-agent: Storebot-Google
User-agent: AdsBot-Google-Mobile
User-agent: AdsBot-Google-Mobile-Apps
User-agent: APIs-Google
User-agent: Google Web Light
User-agent: Google-Ads-Creatives-Assistant
User-agent: Google-adwords
User-agent: Google-amphtml
User-agent: Google-cloudvertexbot
User-agent: Google-play-newsstand
User-agent: Google-read-aloud
User-agent: Google-Safety
User-agent: Google-site-verification
User-agent: Google-Structured-Data-Testing-Tool
User-agent: Googleimageproxy
User-agent: Twitterbot
User-agent: Applebot
User-agent: applebot-aggregator
User-agent: AppleNewsBot
User-agent: AdIdxBot
User-agent: admantx
User-agent: admantx-usbatch/3.1
User-agent: AHC
User-agent: AHC/2.1
User-agent: AhrefsBot
User-agent: AmazonAdBot
User-agent: arquivo.pt
User-agent: bidswitchbot/1.0
User-agent: bnf.fr_bot
User-agent: botify
User-agent: checkMy
User-agent: Chrome-Lighthouse
User-agent: CognitiveSEO
User-agent: ContentKing
User-agent: CriteoBot/0.1
User-agent: datadog-synthetics
User-agent: DeepCrawl
User-agent: dlvrit
User-agent: EchoboxBot/1.0
User-agent: flipboard
User-agent: GrapeshotCrawler
User-agent: GumGum-Bot
User-agent: HyScore/1.0
User-agent: ia_archiver
User-agent: Iframely
User-agent: JetOctopus
User-agent: labne
User-agent: Leikibot
User-agent: LumarBot
User-agent: MJ12bot
User-agent: newrelicpinger
User-agent: okta
User-agent: parselybot
User-agent: peer39_crawler/1.0
User-agent: photon
User-agent: pingdom
User-agent: Pinterestbot
User-agent: proofpoint
User-agent: proximic
User-agent: pubmatic
User-agent: rogerbot
User-agent: RyteBot
User-agent: schema-validator
User-agent: semantic-scholar
User-agent: sentry
User-agent: SERankingBot
User-agent: SimilarWebBot
User-agent: SirdataBot
User-agent: SISTRIX
User-agent: site24x7
User-agent: Sitebulb
User-agent: siteimprovebot
User-agent: Slack-ImgProxy
User-agent: Slackbot
User-agent: Slackbot-LinkExpanding
User-agent: skypeuripreview
User-agent: smartologybot
User-agent: Snap URL Preview Service
User-agent: snapchat
User-agent: SocialFlow
User-agent: statuscake
User-agent: Taboolabot
User-agent: telegrambot
User-agent: thetradedesk
User-agent: uptimerobot
User-agent: Verity
User-agent: YahooMailProxy
User-agent: yahoo-linkpreview
Disallow:

User-agent: Googlebot
User-agent: Googlebot-Image
User-agent: Googlebot-News
User-agent: Googlebot-Video
User-agent: GoogleOther
User-agent: GoogleOther-Image
User-agent: GoogleOther-Video
User-agent: Bingbot
User-agent: bingpreview
User-agent: MicrosoftPreview
User-agent: msnbot
User-agent: DuckDuckBot
User-agent: Slurp
User-agent: yahoo
User-agent: Y!J-BOT
User-agent: 360Spider
User-agent: baiduspider
User-agent: MojeekBot
User-agent: petalbot
User-agent: qwantbot
User-agent: SeznamBot
User-agent: sogou
User-agent: yandex
User-agent: YandexBot
User-agent: YandexImages
User-agent: Yeti
User-agent: yisouspider
Disallow: /*?ref=
Disallow: /*?utm_source=
Disallow: /*?utm_campaign=
Disallow: /*?fbclid=
Disallow: /*?nk=
Disallow: /*?q=
Disallow: /*?cat=
Disallow: /*?ct=
Disallow: /*?sort=
Disallow: /*?pub=
Disallow: /search-recipes/

User-agent: *
Allow: /ads.txt
Allow: /app-ads.txt
Disallow: /
Sitemap: https://www.taste.com.au/sitemap.xml
Sitemap: https://www.taste.com.au/news.xml
Sitemap: https://www.taste.com.au/videos.xml
Sitemap: https://www.taste.com.au/video-index-vertical-sitemap.xml
Sitemap: https://www.taste.com.au/sitemap-static.xml

History

Changes recorded since June 2026.

No change recorded.

In context

Australia
34.0% block at least one AI crawler
Behind Cloudflare
No
Same group
adelaidenow.com.au, byronnews.com.au, cairnspost.com.au, dailytelegraph.com.au, ntnews.com.au, couriermail.com.au, dailymercury.com.au, foxsports.com.au, frasercoastchronicle.com.au, gq.com.au, geelongadvertiser.com.au, gladstoneobserver.com.au