taste.com.au · Australia · News Corp AU · measured 8 Oct 2026
As read on 8 Oct 2026, 4,566 bytes. 2 rule groups name AI crawlers; shown first.
User-agent: ScreamingFrogSEOSpider User-agent: ComscoreCrawler User-agent: Moatbot User-agent: Adbeat User-agent: integralads.com User-agent: Expo9 User-agent: Outbrain User-agent: semasio.net User-agent: Weborama-fetcher User-agent: YahooAdmonitoring User-agent: netseer.com User-agent: SemrushBot User-agent: omgili.com User-agent: FacebookCrawler User-agent: facebookexternalhit User-agent: facebot User-agent: ias_crawler User-agent: ias_wombles User-agent: IASbot User-agent: identrics User-agent: MeltwaterNews User-agent: links.streem.com.au User-agent: Webz-bot User-agent: LinkedInBot User-agent: WhatsApp User-agent: redditbot User-agent: Mediapartners-Google User-agent: Google-InspectionTool User-agent: Adsbot-Google User-agent: Storebot-Google User-agent: AdsBot-Google-Mobile User-agent: AdsBot-Google-Mobile-Apps User-agent: APIs-Google User-agent: Google Web Light User-agent: Google-Ads-Creatives-Assistant User-agent: Google-adwords User-agent: Google-amphtml User-agent: Google-cloudvertexbot User-agent: Google-play-newsstand User-agent: Google-read-aloud User-agent: Google-Safety User-agent: Google-site-verification User-agent: Google-Structured-Data-Testing-Tool User-agent: Googleimageproxy User-agent: Twitterbot User-agent: Applebot User-agent: applebot-aggregator User-agent: AppleNewsBot User-agent: AdIdxBot User-agent: admantx User-agent: admantx-usbatch/3.1 User-agent: AHC User-agent: AHC/2.1 User-agent: AhrefsBot User-agent: AmazonAdBot User-agent: arquivo.pt User-agent: bidswitchbot/1.0 User-agent: bnf.fr_bot User-agent: botify User-agent: checkMy User-agent: Chrome-Lighthouse User-agent: CognitiveSEO User-agent: ContentKing User-agent: CriteoBot/0.1 User-agent: datadog-synthetics User-agent: DeepCrawl User-agent: dlvrit User-agent: EchoboxBot/1.0 User-agent: flipboard User-agent: GrapeshotCrawler User-agent: GumGum-Bot User-agent: HyScore/1.0 User-agent: ia_archiver User-agent: Iframely User-agent: JetOctopus User-agent: labne User-agent: Leikibot User-agent: LumarBot User-agent: MJ12bot User-agent: newrelicpinger User-agent: okta User-agent: parselybot User-agent: peer39_crawler/1.0 User-agent: photon User-agent: pingdom User-agent: Pinterestbot User-agent: proofpoint User-agent: proximic User-agent: pubmatic User-agent: rogerbot User-agent: RyteBot User-agent: schema-validator User-agent: semantic-scholar User-agent: sentry User-agent: SERankingBot User-agent: SimilarWebBot User-agent: SirdataBot User-agent: SISTRIX User-agent: site24x7 User-agent: Sitebulb User-agent: siteimprovebot User-agent: Slack-ImgProxy User-agent: Slackbot User-agent: Slackbot-LinkExpanding User-agent: skypeuripreview User-agent: smartologybot User-agent: Snap URL Preview Service User-agent: snapchat User-agent: SocialFlow User-agent: statuscake User-agent: Taboolabot User-agent: telegrambot User-agent: thetradedesk User-agent: uptimerobot User-agent: Verity User-agent: YahooMailProxy User-agent: yahoo-linkpreview Disallow: User-agent: Googlebot User-agent: Googlebot-Image User-agent: Googlebot-News User-agent: Googlebot-Video User-agent: GoogleOther User-agent: GoogleOther-Image User-agent: GoogleOther-Video User-agent: Bingbot User-agent: bingpreview User-agent: MicrosoftPreview User-agent: msnbot User-agent: DuckDuckBot User-agent: Slurp User-agent: yahoo User-agent: Y!J-BOT User-agent: 360Spider User-agent: baiduspider User-agent: MojeekBot User-agent: petalbot User-agent: qwantbot User-agent: SeznamBot User-agent: sogou User-agent: yandex User-agent: YandexBot User-agent: YandexImages User-agent: Yeti User-agent: yisouspider Disallow: /*?ref= Disallow: /*?utm_source= Disallow: /*?utm_campaign= Disallow: /*?fbclid= Disallow: /*?nk= Disallow: /*?q= Disallow: /*?cat= Disallow: /*?ct= Disallow: /*?sort= Disallow: /*?pub= Disallow: /search-recipes/ User-agent: * Allow: /ads.txt Allow: /app-ads.txt Disallow: / Sitemap: https://www.taste.com.au/sitemap.xml Sitemap: https://www.taste.com.au/news.xml Sitemap: https://www.taste.com.au/videos.xml Sitemap: https://www.taste.com.au/video-index-vertical-sitemap.xml Sitemap: https://www.taste.com.au/sitemap-static.xml
Changes recorded since June 2026.
No change recorded.