mashable.com · United States · measured 8 Oct 2026
As read on 8 Oct 2026, 6,215 bytes. 2 rule groups name AI crawlers; shown first.
User-agent: AddSearchBot User-agent: AgentDataBot User-agent: AI2Bot User-agent: Ai2Bot-DeepResearchEval User-agent: Ai2Bot-Dolma User-agent: AIWebIndex User-agent: AIWebIndex-Agent User-agent: amazon-QBusiness User-agent: AmazonBuyForMe User-agent: Amazonbot User-agent: Amzn-SearchBot User-agent: Amzn-User User-agent: Anchor Browser User-agent: AndiAgent User-agent: anthropic-ai User-agent: ApifyBot User-agent: ApifyWebsiteContentCrawler User-agent: Applebot User-agent: Applebot-extended User-agent: atlassian-bot User-agent: AutoRAG User-agent: AwarioSmartBot User-agent: AzureAI-SearchBot User-agent: BeansLLM-CorpusBot User-agent: Big Sur AI User-agent: bigsur.ai User-agent: BixelBot User-agent: Brandwatch User-agent: Bravebot User-agent: Brightbot User-agent: Bytespider User-agent: CCBot User-agent: Channel3Bot User-agent: ChatGLM-Spider User-agent: ChatGPT-User User-agent: Claude-Code User-agent: Claude-SearchBot User-agent: Claude-User User-agent: Claude-Web User-agent: ClaudeBot User-agent: Cloudflare-AutoRAG User-agent: CloudflareBrowserRenderingCrawler User-agent: Code User-agent: cohere-ai User-agent: cohere-training-data-crawler User-agent: Cotoyogi User-agent: Crawl4AI User-agent: Cursor User-agent: Datenbank Crawler User-agent: Datenbank-Crawler User-agent: DeepSeekBot User-agent: DesearchBot User-agent: Devin User-agent: Diffbot User-agent: Direqt Anomura User-agent: Dola-SearchBot User-agent: DuckAssistBot User-agent: Echobot Bot User-agent: ExaBot User-agent: ExaSearchBot User-agent: FacebookBot User-agent: Factset_spyderbot User-agent: FirecrawlAgent User-agent: GeistHaus-PageFetcher User-agent: GPTBot User-agent: grok-agent User-agent: GrokBot User-agent: Hunyuan User-agent: iAsk User-agent: iAskBot User-agent: iaskspider User-agent: ICC Crawler User-agent: ICC-Crawler User-agent: ImageBot User-agent: ImageSiftBot User-agent: imageSpider User-agent: IsonSearchBot User-agent: kagi-fetcher User-agent: Kangaroo Bot User-agent: Kangaroo-Bot User-agent: Keenable-User User-agent: KeenableBot User-agent: Kimi-Agent User-agent: Kimi-User User-agent: KlaviyoAIBot User-agent: Kunato User-agent: KunatoCrawler User-agent: laion-huggingface-processor User-agent: LCC User-agent: LinerBot User-agent: LinkupBot User-agent: LINER Bot User-agent: Manus-User User-agent: meta-externalagent User-agent: meta-externalfetcher User-agent: meta-webindexer User-agent: mistral.ai User-agent: MistralAI-User User-agent: netEstate Imprint Crawler User-agent: Novellum AI Crawl User-agent: Novellum User-agent: NovaAct User-agent: OAI-SearchBot User-agent: omgili User-agent: opencode User-agent: PanguBot User-agent: Perplexity-User User-agent: PerplexityBot User-agent: PeopleInc-DCipher-Scraper/1.1.0 User-agent: PetalBot User-agent: PhindBot User-agent: Poggio-Citations User-agent: qodercli User-agent: QualifiedBot User-agent: recraft-image-dataset User-agent: Reflectionbot User-agent: ScryBot User-agent: SeekrBot User-agent: SemrushBot-OCOB User-agent: SemrushBotSwa User-agent: Shap-User User-agent: ShapBot User-agent: SBIntuitionsBot User-agent: SofyaBot User-agent: SomethinggBot User-agent: Spider User-agent: TalarionFrontierCrawler User-agent: TalarionSearchBot User-agent: TerraCotta User-agent: Timpibot User-agent: TavilyBot User-agent: TongyiBot User-agent: Trae User-agent: TwinAgent User-agent: UseAI User-agent: Velen Crawler User-agent: VelenPublicWebCrawler User-agent: Webzio-Extended User-agent: Wrtn User-agent: WRTNBot User-agent: xAI-Grok User-agent: YandexAdditionalBot User-agent: YepBot User-agent: YiBot User-agent: YiyanBot User-agent: YouBot User-agent: ZanistaBot Allow: /ad/tech/how-to-set-up-the-smartest-galaxy-ai-features-on-your-new-samsung-galaxy Allow: /ad/tech/how-the-samsung-galaxy-s26-ultra-turns-your-next-festival-into-content Disallow: / User-agent: "008" User-agent: Applebot User-agent: DDM-DCipher/1.0.7 User-agent: GPTBot User-agent: HTTrack User-agent: Nutch User-agent: Offline Explorer User-agent: Scrapy Allow: /ad/article/samsung-galaxy-ai-changing-smartphone-photography Allow: /ad/article/how-samsung-galaxy-ai-can-upgrade-your-smartphone Allow: /ad/article/samsung-galaxy-ai-automates-everyday-tasks Disallow: / Sitemap: https://mashable.com/sitemap-index.xml Sitemap: https://mashable.com/sitemap-news-0.xml Sitemap: https://mashable.com/sitemap-live-blogs-0.xml User-agent: * Disallow: /search Disallow: /archive/ Disallow: /cdn-cgi/ Allow: /*?page=[0-9]
Changes recorded since June 2026.
No change recorded.