CrawlerPulse
CrawlerPulse / North America / Canada / insidehalton.com

insidehalton.com

insidehalton.com · Canada · Torstar · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Blocked
OpenAI
Blocked
Anthropic
Blocked
Perplexity
Blocked
Meta
Blocked
Common Crawl
Blocked
ByteDance
Blocked
Apple
Blocked
Amazon
Blocked

robots.txt

As read on 8 Oct 2026, 11,935 bytes. 26 rule groups name AI crawlers; shown first.

User-agent: Applebot-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: Claude-Web
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Diffbot
Disallow: /

User-agent: FacebookBot
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: ImagesiftBot
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: PetalBot
Disallow: /

User-agent: Timpibot
Disallow: /

User-agent: YouBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: cohere-ai
Disallow: /

User-agent: omgili
Disallow: /

User-agent: omgilibot
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: MistralAI-User
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: DeepSeekBot
Disallow: /

User-agent: Google-CloudVertexBot
Disallow: /

User-agent: AI2Bot
Disallow: /

User-agent: Ai2Bot-Dolma
Disallow: /

User-agent: FriendlyCrawler
Disallow: /

User-agent: ICC-Crawler
Disallow: /

User-agent: Scrapy
Disallow: /

User-agent: VelenPublicWebCrawler
Disallow: /

User-agent: Webzio-Extended
Disallow: /

User-agent: iaskspider/2.0
Disallow: /

User-agent: img2dataset
Disallow: /

User-agent: cohere-training-data-crawler
Disallow: /

User-agent: Kangaroo Bot
Disallow: /

User-agent: PanguBot
Disallow: /

User-agent: DuckAssistBot
Disallow: /

User-agent: Meta-ExternalFetcher
Disallow: /

User-agent: Perplexity-User
Disallow: /

User-agent: Claude-SearchBot
Disallow: /

User-agent: Claude-User
Disallow: /

User-agent: Cotoyogi
Disallow: /

User-agent: Datenbank
Disallow: /

User-agent: Devin
Disallow: /

User-agent: Factset_spyderbot
Disallow: /

User-agent: Gemini-Deep-Research
Disallow: /

User-agent: netEstate
Disallow: /

User-agent: Imprint
Disallow: /

User-agent: SemrushBot-OCOB
Disallow: /

User-agent: AddSearchBot
Disallow: /

User-agent: bigsur.ai
Disallow: /

User-agent: CloudVertexBot
Disallow: /

User-agent: GoogleOther
Disallow: /

User-agent: LinerBot
Disallow: /

User-agent: QualifiedBot
Disallow: /

User-agent: Ai2Bot-DeepResearchEval
Disallow: /

User-agent: ChatGLM-Spider
Disallow: /

User-agent: Google-NotebookLM
Disallow: /

User-agent: iAskBot
Disallow: /

User-agent: KunatoCrawler
Disallow: /

User-agent: meta-externalfetcher
Disallow: /

User-agent: meta-webindexer
Disallow: /

User-agent: webzio-extended
Disallow: /

User-agent: WRTNBot
Disallow: /

User-agent: ZanistaBot
Disallow: /

User-agent: AI2Bot-DeepResearchEval
Disallow: /

User-agent: Anomura
Disallow: /

User-agent: atlassian-bot
Disallow: /

User-agent: Bravebot
Disallow: /

User-agent: Channel3Bot
Disallow: /

User-agent: Cloudflare-AutoRAG
Disallow: /

User-agent: Crawl4AI
Disallow: /

User-agent: imageSpider
Disallow: /

User-agent: KlaviyoAIBot
Disallow: /

User-agent: laion-huggingface-processor
Disallow: /

User-agent: LCC
Disallow: /

User-agent: LinkupBot
Disallow: /

User-agent: PhindBot
Disallow: /

User-agent: Poggio-Citations
Disallow: /

User-agent: SBIntuitionsBot
Disallow: /

User-agent: TavilyBot
Disallow: /

User-agent: Amzn-SearchBot
Disallow: /

User-agent: Amzn-User
Disallow: /

User-agent: ApifyBot
Disallow: /

User-agent: ApifyWebsiteContentCrawler
Disallow: /

User-agent: Aranet-SearchBot
Disallow: /

User-agent: AzureAI-SearchBot
Disallow: /

User-agent: ExaBot
Disallow: /

User-agent: FirecrawlAgent
Disallow: /

User-agent: kagi-fetcher
Disallow: /

User-agent: TerraCotta
Disallow: /

User-agent: ScraperAPI
Disallow: /

User-agent: Scraper-API
Disallow: /

User-agent: ScrapingBee
Disallow: /

User-agent: ScrapingBeeBot
Disallow: /

User-agent: ZenRows
Disallow: /

User-agent: ZenRowsBot
Disallow: /

User-agent: ScrapeGraphAI
Disallow: /

User-agent: scrapegraph
Disallow: /

User-agent: Scrapfly
Disallow: /

User-agent: ScrapflyBot
Disallow: /

User-agent: ScrapflyClient
Disallow: /

User-agent: Zyte
Disallow: /

User-agent: ZyteBot
Disallow: /

User-agent: Scrapinghub
Disallow: /

User-agent: SerpApi
Disallow: /

User-agent: SerpApiBot
Disallow: /

User-agent: WebScraping.AI
Disallow: /

User-agent: WebScrapingAI
Disallow: /

User-agent: webscraping-ai
Disallow: /

User-agent: Scrapeless
Disallow: /

User-agent: ScrapelessBot
Disallow: /

User-agent: Scrapingdog
Disallow: /

User-agent: ScrapingdogBot
Disallow: /

User-agent: Scrapestack
Disallow: /

User-agent: ScrapestackBot
Disallow: /

User-agent: ScaleSERP
Disallow: /

User-agent: SerpWow
Disallow: /

User-agent: Rainforest-API
Disallow: /

User-agent: BlueCart-API
Disallow: /

User-agent: BrightData
Disallow: /

User-agent: Bright Data
Disallow: /

User-agent: BrightBot
Disallow: /

User-agent: Smartproxy
Disallow: /

User-agent: Oxylabs
Disallow: /

User-agent: OxylabsBot
Disallow: /

User-agent: Soax
Disallow: /

User-agent: SoaxBot
Disallow: /

User-agent: SOAX
Disallow: /

User-agent: Nimbleway
Disallow: /

User-agent: NimbleCrawler
Disallow: /

User-agent: NimblewayBot
Disallow: /

User-agent: Froxy
Disallow: /

User-agent: FroxyBot
Disallow: /

User-agent: BrowseAI
Disallow: /

User-agent: Browse-AI
Disallow: /

User-agent: BrowseAIBot
Disallow: /

User-agent: Octoparse
Disallow: /

User-agent: OctoparseBot
Disallow: /

User-agent: Import.io
Disallow: /

User-agent: ImportIO
Disallow: /

User-agent: importio
Disallow: /

User-agent: Kadoa
Disallow: /

User-agent: Kadoabot
Disallow: /

User-agent: KadoaBot
Disallow: /

User-agent: ScrapeHero
Disallow: /

User-agent: Scrape-Hero
Disallow: /

User-agent: ScrapeHeroBot
Disallow: /

User-agent: Grepsr
Disallow: /

User-agent: GrepsrBot
Disallow: /

User-agent: GrepsrCrawler
Disallow: /

User-agent: MrScraper
Disallow: /

User-agent: Mr-Scraper
Disallow: /

User-agent: MrScraperBot
Disallow: /

User-agent: PromptCloud
Disallow: /

User-agent: PromptCloudBot
Disallow: /

User-agent: CrawlNow
Disallow: /

User-agent: CrawlNowBot
Disallow: /

User-agent: ForageAI
Disallow: /

User-agent: Forage-AI
Disallow: /

User-agent: ForageBot
Disallow: /

User-agent: DumplingAI
Disallow: /

User-agent: Dumpling-AI
Disallow: /

User-agent: DumplingBot
Disallow: /

User-agent: spider-cloud
Disallow: /

User-agent: SpiderCloud
Disallow: /

User-agent: Hyperbrowser
Disallow: /

User-agent: HyperbrowserBot
Disallow: /

User-agent: HyperAgent
Disallow: /

User-agent: skrape.ai
Disallow: /

User-agent: SkrapeAI
Disallow: /

User-agent: SkrapeBot
Disallow: /

User-agent: CloudflareBrowserRenderingCrawler
Disallow: /

User-agent: Apify-HTTP-Client
Disallow: /

User-agent: amazon-kendra
Disallow: /

User-agent: amazon-QBusiness
Disallow: /

User-agent: bedrockbot
Disallow: /

User-agent: CohereBot
Disallow: /

User-agent: Doubaobot
Disallow: /

User-agent: ERNIEBot
Disallow: /

User-agent: GeistHaus-PageFetcher
Disallow: /

User-agent: GoogleAgent-URLContext
Disallow: /

User-agent: Kimi-SearchBot
Disallow: /

User-agent: Kimi-User
Disallow: /

User-agent: KimiBot
Disallow: /

User-agent: KimiCrawler
Disallow: /

User-agent: micro-crawl
Disallow: /

User-agent: MistralAI-Index
Disallow: /

User-agent: MistralBot
Disallow: /

User-agent: MoonshotBot
Disallow: /

User-agent: MoonSpider
Disallow: /

User-agent: QwenBot
Disallow: /

User-agent: SalamandraVLM
Disallow: /

User-agent: Shap-User
Disallow: /

User-agent: TongyiBot
Disallow: /

User-agent: UseAI
Disallow: /

User-agent: xAI-SearchBot
Disallow: /

User-agent: YandexAdditional
Disallow: /

User-agent: YandexAdditionalBot
Disallow: /

User-agent: YiyanBot
Disallow: /

User-agent: ZenphonyBot
Disallow: /

User-agent: AIWebIndex
Disallow: /

User-agent: AIWebIndex-Agent
Disallow: /

User-agent: CragCrawler
Disallow: /

User-agent: Crawlspace
Disallow: /

User-agent: Datenbank Crawler
Disallow: /

User-agent: ExaSearchBot
Disallow: /

User-agent: HenkBot
Disallow: /

User-agent: Mozilla-Tabstack
Disallow: /

User-agent: netEstate Imprint Crawler
Disallow: /

User-agent: Querit-SearchBot
Disallow: /

User-agent: QueritBot
Disallow: /

User-agent: Reflectionbot
Disallow: /

User-agent: ShapBot
Disallow: /

User-agent: TalarionSearchBot
Disallow: /

User-agent: Terra Cotta
Disallow: /

User-agent: Spider
Disallow: /

User-agent: *
Disallow: /tncms/tracking/
Disallow: /_services/
Disallow: /58580620/
Disallow: /do-some-good-story/
Disallow: /events/events-this-weekend/
Disallow: /good-news/
Disallow: /search/
Disallow: /tncms/counter/
Disallow: /tncms/block/
Disallow: /tncms/messaging/
Disallow: /tncms/admin/
Disallow: /tncms/openweb/*
Disallow: /tncms/track/
Disallow: /tncms/webservice/
Disallow: /tncms/dmp/
Disallow: /tncms/tax/
Disallow: /tncms/csrf/
Disallow: /tncms/gtm/
Disallow: /tncms/search/
Disallow: /tncms/auth/
Disallow: /tncms/media/
Disallow: /tncms/weather/
Disallow: /tncms/openid2/
Disallow: /content/tncms/
Disallow: /marketplace/*action=srch
Disallow: /tncms/disqus/
Disallow: /tncms/user/
Disallow: /users/admin/
Disallow: /users/*/?
Disallow: /tncms/calendar/
Disallow: /content/tncms/assets/v3/form/
Disallow: /content/tncms/form/
Disallow: /*{{url}}
Disallow: /*%7b%7burl%7d%7d
Disallow: /*%7B%7Burl%7D%7D
Disallow: /content/tncms/live/
Disallow: /assets/fonts/
Disallow: /content/dam/
Disallow: /*li_medium=
Disallow: /*li_source=
Sitemap: https://www.insidehalton.com/sitemap.xml
Sitemap: https://www.insidehalton.com/tncms/sitemap/editorial.xml
Sitemap: https://www.insidehalton.com/tncms/sitemap/editorial.xml?year=2000
Sitemap: https://www.insidehalton.com/tncms/sitemap/editorial.xml?year=2003
Sitemap: https://www.insidehalton.com/tncms/sitemap/editorial.xml?year=2004
Sitemap: https://www.insidehalton.com/tncms/sitemap/editorial.xml?year=2005
Sitemap: https://www.insidehalton.com/tncms/sitemap/editorial.xml?year=2006
Sitemap: https://www.insidehalton.com/tncms/sitemap/editorial.xml?year=2007
Sitemap: https://www.insidehalton.com/tncms/sitemap/editorial.xml?year=2008
Sitemap: https://www.insidehalton.com/tncms/sitemap/editorial.xml?year=2009
Sitemap: https://www.insidehalton.com/tncms/sitemap/editorial.xml?year=2010
Sitemap: https://www.insidehalton.com/tncms/sitemap/editorial.xml?year=2011
Sitemap: https://www.insidehalton.com/tncms/sitemap/editorial.xml?year=2012
Sitemap: https://www.insidehalton.com/tncms/sitemap/editorial.xml?year=2013
Sitemap: https://www.insidehalton.com/tncms/sitemap/editorial.xml?year=2014
Sitemap: https://www.insidehalton.com/tncms/sitemap/editorial.xml?year=2015
Sitemap: https://www.insidehalton.com/tncms/sitemap/editorial.xml?year=2016
Sitemap: https://www.insidehalton.com/tncms/sitemap/editorial.xml?year=2017
Sitemap: https://www.insidehalton.com/tncms/sitemap/editorial.xml?year=2018
Sitemap: https://www.insidehalton.com/tncms/sitemap/editorial.xml?year=2019
Sitemap: https://www.insidehalton.com/tncms/sitemap/editorial.xml?year=2020
Sitemap: https://www.insidehalton.com/tncms/sitemap/editorial.xml?year=2021
Sitemap: https://www.insidehalton.com/tncms/sitemap/editorial.xml?year=2022
Sitemap: https://www.insidehalton.com/tncms/sitemap/editorial.xml?year=2023
Sitemap: https://www.insidehalton.com/tncms/sitemap/editorial.xml?year=2024
Sitemap: https://www.insidehalton.com/tncms/sitemap/editorial.xml?year=2025

History

Changes recorded since June 2026.

No change recorded.

In context