inc42.com · India · measured 9 Oct 2026
As read on 9 Oct 2026, 1,628 bytes. 3 rule groups name AI crawlers; shown first.
User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: meta-externalagent User-agent: Amazonbot User-agent: CCBot Disallow: / Allow: /company/ Allow: /person/ Allow: /industry/ Allow: /location/ User-agent: Bytespider User-agent: cohere-ai User-agent: anthropic-ai User-agent: Diffbot User-agent: ImagesiftBot User-agent: Omgilibot User-agent: Timpibot Disallow: / User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: Claude-Web User-agent: Claude-User User-agent: PerplexityBot User-agent: Perplexity-User User-agent: MistralAI-User Allow: / User-agent: * Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / Disallow: /readme.html Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Allow: /wp-content/uploads/ Disallow: /*?* Disallow: /*page/* Disallow: /*/0/* Allow: /*?*utm_* Allow: /*?*itm_* Allow: /*?ver* User-agent: Twitterbot User-agent: facebookexternalhit User-agent: LinkedInBot Allow: / Sitemap: https://inc42.com/sitemap_index.xml Sitemap: https://inc42.com/news-sitemap.xml Sitemap: https://inc42.com/inc42_datalabs.xml
Changes recorded since June 2026.
No change recorded.