nymag.com · United States · Vox Media · measured 8 Oct 2026
As read on 8 Oct 2026, 8,252 bytes. 24 rule groups name AI crawlers; shown first.
User-agent: Amazonbot Allow: / User-agent: GPTBot Allow: / User-agent: anthropic-ai Disallow: / Allow: /sp/ Allow: /strategist User-agent: Applebot-Extended Disallow: / Allow: /sp/ Allow: /strategist User-agent: Bytespider Disallow: / Allow: /sp/ Allow: /strategist User-agent: CCBot Disallow: / Allow: /sp/ Allow: /strategist User-agent: ChatGPT-User Disallow: / Allow: /sp/ Allow: /strategist User-agent: Claude-Web Disallow: / Allow: /sp/ Allow: /strategist User-agent: ClaudeBot Disallow: / Allow: /sp/ Allow: /strategist User-agent: cohere-ai Disallow: / Allow: /sp/ Allow: /strategist User-agent: DeepSeekBot Disallow: / Allow: /sp/ Allow: /strategist User-agent: Diffbot Disallow: / Allow: /sp/ Allow: /strategist User-agent: FacebookBot Disallow: / Allow: /sp/ Allow: /strategist User-agent: Google-CloudVertexBot Disallow: / Allow: /sp/ Allow: /strategist User-agent: Google-Extended Disallow: / Allow: /sp/ Allow: /strategist User-agent: ImagesiftBot Disallow: / Allow: /sp/ Allow: /strategist User-agent: meta-externalagent Disallow: / Allow: /sp/ Allow: /strategist User-agent: MistralAI-User Disallow: / Allow: /sp/ Allow: /strategist User-agent: omgili Disallow: / Allow: /sp/ Allow: /strategist User-agent: Omgilibot Disallow: / Allow: /sp/ Allow: /strategist User-agent: PerplexityBot Disallow: / Allow: /sp/ Allow: /strategist User-agent: PetalBot Disallow: / Allow: /sp/ Allow: /strategist User-agent: Timpibot Disallow: / Allow: /sp/ Allow: /strategist User-agent: YouBot Disallow: / Allow: /sp/ Allow: /strategist User-agent: Mediapartners-Google Disallow: Sitemap: https://nymag.com/sitemap.xml Sitemap: https://nymag.com/sitemaps-cq/sitemap.xml Sitemap: https://nymag.com/news.xml Sitemap: https://nymag.com/strategist/sitemap.xml Sitemap: https://nymag.com/intelligencer/sitemap.xml Sitemap: https://nymag.com/fashion-sitemaps/fashionlabels-sitemap.xml Sitemap: https://nymag.com/fashion-sitemaps/fashionshows-sitemap.xml Sitemap: https://nymag.com/scienceofus-sitemaps/scienceofus-sitemap.xml User-agent: TruliaBot Disallow: / User-agent: Twitterbot Disallow: /amp/* Disallow: /intelligencer/amp/* Disallow: /strategist/amp/* User-agent: * Disallow: /coral-talk/ Disallow: /daily/grubstreet/ Disallow: /daily/southflorida/ Disallow: /fashion/lookbook/38917/ Disallow: /fashion/search/models/ Disallow: /jesse/ Disallow: /marketplace/ Disallow: /news/intelligencer/16951/ Disallow: /newsletter/ Disallow: /*/newsletter/ Disallow: /nymag/columns/intelligencer/features/16951/ Disallow: /nymag/letters/14872/ Disallow: /search.html Disallow: /search?* Disallow: /srch Disallow: /srch/ Disallow: /strategist/search Disallow: /tags/newsletters/ Disallow: /temp/ Disallow: /test/ Disallow: /urr/ Disallow: *?origSession=* Disallow: *?q=* Disallow: *?view_token=* User-agent: Mediapartners-Google* Disallow: User-agent: Applebot Allow: / User-agent: A6-Indexer Disallow: / Allow: /sp/ Allow: /strategist User-agent: AhrefsBot Disallow: / Allow: /sp/ Allow: /strategist User-agent: AI2Bot Disallow: / Allow: /sp/ Allow: /strategist User-agent: AI2Bot-DeepResearchEval Disallow: / Allow: /sp/ Allow: /strategist User-agent: Ai2Bot-Dolma Disallow: / Allow: /sp/ Allow: /strategist User-agent: aiHitBot Disallow: / Allow: /sp/ Allow: /strategist User-agent: Anchor Browser Disallow: / Allow: /sp/ Allow: /strategist User-agent: ApifyBot Disallow: / Allow: /sp/ Allow: /strategist User-agent: ApifyWebsiteContentCrawler Disallow: / Allow: /sp/ Allow: /strategist User-agent: Baiduspider Disallow: / Allow: /sp/ Allow: /strategist User-agent: betaBot Disallow: / Allow: /sp/ Allow: /strategist User-agent: BLEXBot Disallow: / Allow: /sp/ Allow: /strategist User-agent: botify Disallow: / Allow: /sp/ Allow: /strategist User-agent: brands-bot-logo Disallow: / Allow: /sp/ Allow: /strategist User-agent: Brightbot Disallow: / Allow: /sp/ Allow: /strategist User-agent: calculon Disallow: / Allow: /sp/ Allow: /strategist User-agent: ChatGLM-Spider Disallow: / Allow: /sp/ Allow: /strategist User-agent: Claude-SearchBot Disallow: / Allow: /sp/ Allow: /strategist User-agent: Claude-User Disallow: / Allow: /sp/ Allow: /strategist User-agent: CloudVertexBot Disallow: / Allow: /sp/ Allow: /strategist User-agent: cohere-training-data-crawler Disallow: / Allow: /sp/ Allow: /strategist User-agent: collection@infegy.com Disallow: / Allow: /sp/ Allow: /strategist User-agent: Cotoyogi Disallow: / Allow: /sp/ Allow: /strategist User-agent: Crawl4AI Disallow: / Allow: /sp/ Allow: /strategist User-agent: CrunchBot Disallow: / Allow: /sp/ Allow: /strategist User-agent: deepcrawl Disallow: / Allow: /sp/ Allow: /strategist User-agent: DirBuster Disallow: / Allow: /sp/ Allow: /strategist User-agent: DTS Disallow: / Allow: /sp/ Allow: /strategist User-agent: DuckAssistBot Disallow: / Allow: /sp/ Allow: /strategist User-agent: EasouSpider Disallow: / Allow: /sp/ Allow: /strategist User-agent: FirecrawlAgent Disallow: / Allow: /sp/ Allow: /strategist User-agent: FriendlyCrawler Disallow: / Allow: /sp/ Allow: /strategist User-agent: GlutenCrawler Disallow: / Allow: /sp/ Allow: /strategist User-agent: GrapeshotCrawler Disallow: / Allow: /sp/ Allow: /strategist User-agent: iaskspider Disallow: / Allow: /sp/ Allow: /strategist User-agent: ICC-Crawler Disallow: / Allow: /sp/ Allow: /strategist User-agent: img2dataset Disallow: / Allow: /sp/ Allow: /strategist User-agent: JackCrawl Disallow: / Allow: /sp/ Allow: /strategist User-agent: Jetslide Disallow: / Allow: /sp/ Allow: /strategist User-agent: linkapedia_bot Disallow: / Allow: /sp/ Allow: /strategist User-agent: Mediatoolkitbot Disallow: / Allow: /sp/ Allow: /strategist User-agent: MegaIndex Disallow: / Allow: /sp/ Allow: /strategist User-agent: meta-externalfetcher Disallow: / Allow: /sp/ Allow: /strategist User-agent: Microsoft BITS Disallow: / Allow: /sp/ Allow: /strategist User-agent: Microsoft URL Control Disallow: / Allow: /sp/ Allow: /strategist User-agent: Microsoft-WebDAV-MiniRedir Disallow: / Allow: /sp/ Allow: /strategist User-agent: news-please Disallow: / Allow: /sp/ Allow: /strategist User-agent: Novellum Disallow: / Allow: /sp/ Allow: /strategist User-agent: NTENTbot Disallow: / Allow: /sp/ Allow: /strategist User-agent: Nutch Disallow: / Allow: /sp/ Allow: /strategist User-agent: OpenLinkProfiler Disallow: / Allow: /sp/ Allow: /strategist User-agent: Perplexity-User Disallow: / Allow: /sp/ Allow: /strategist User-agent: Primalbot Disallow: / Allow: /sp/ Allow: /strategist User-agent: python-requests Disallow: / Allow: /sp/ Allow: /strategist User-agent: Re-Animator Disallow: / Allow: /sp/ Allow: /strategist User-agent: Riddler Disallow: / Allow: /sp/ Allow: /strategist User-agent: SafeDNSBot Disallow: / Allow: /sp/ Allow: /strategist User-agent: SBIntuitionsBot Disallow: / Allow: /sp/ Allow: /strategist User-agent: Scrapy Disallow: / Allow: /sp/ Allow: /strategist User-agent: SearchAtlas Disallow: / Allow: /sp/ Allow: /strategist User-agent: SemanticScholarBot Disallow: / Allow: /sp/ Allow: /strategist User-agent: SemrushBot Disallow: / Allow: /sp/ Allow: /strategist User-agent: SentiBot Disallow: / Allow: /sp/ Allow: /strategist User-agent: Sogou Disallow: / Allow: /sp/ Allow: /strategist User-agent: Spider-Pig Disallow: / Allow: /sp/ Allow: /strategist User-agent: sqlmap Disallow: / Allow: /sp/ Allow: /strategist User-agent: StorygizeBot Disallow: / Allow: /sp/ Allow: /strategist User-agent: Test Certificate Info Disallow: / Allow: /sp/ Allow: /strategist User-agent: TikTokSpider Disallow: / Allow: /sp/ Allow: /strategist User-agent: Turnitin Disallow: / Allow: /sp/ Allow: /strategist User-agent: uipbot Disallow: / Allow: /sp/ Allow: /strategist User-agent: vBSEO Disallow: / Allow: /sp/ Allow: /strategist User-agent: VegeBot Disallow: / Allow: /sp/ Allow: /strategist User-agent: VelenPublicWebCrawler Disallow: / Allow: /sp/ Allow: /strategist User-agent: Web Link Validator Disallow: / Allow: /sp/ Allow: /strategist User-agent: WebCruiser Disallow: / Allow: /sp/ Allow: /strategist User-agent: webzio-extended Disallow: / Allow: /sp/ Allow: /strategist User-agent: Xenu Disallow: / Allow: /sp/ Allow: /strategist
Changes recorded since June 2026.
No change recorded.