CrawlerPulse
CrawlerPulse / North America / United States / fortune.com

Fortune Magazine

fortune.com · United States · Fortune · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Blocked
OpenAI
Allowed
Anthropic
Allowed
Perplexity
Allowed
Meta
Allowed
Common Crawl
Blocked
ByteDance
Allowed
Apple
Allowed
Amazon
Allowed

robots.txt

As read on 8 Oct 2026, 1,774 bytes. 4 rule groups name AI crawlers; shown first.

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-Agent: omgili
Disallow: /

User-Agent: omgilibot
Disallow: /
Sitemap: https://fortune.com/sitemap_index.xml
Sitemap: https://fortune.com/education/sitemap_index.xml
Sitemap: https://fortune.com/company/sitemap_index.xml
Sitemap: https://fortune.com/ranking/sitemap.xml
Sitemap: https://fortune.com/ranking/items/sitemap.xml
Sitemap: https://fortune.com/author/sitemap_index.xml
Sitemap: https://fortune.com/about.xml
Sitemap: https://fortune.com/feed/googlenews/articles.xml
Sitemap: https://fortune.com/videos/sitemap.xml
Sitemap: https://fortune.com/press-releases/sitemap_index.xml

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /search/*
Disallow: /sponsored/
Disallow: /newsletter/
Allow: */feed/*/articles.xml
Allow: /feed/sitema*
Disallow: /wp-login.php
Disallow: /wp-signup.php
Disallow: /press-this.php
Disallow: /remote-login.php
Disallow: /activate/
Disallow: /cgi-bin/
Disallow: /mshots/v1/
Disallow: /next/
Disallow: /public.api/*
Disallow: /preview/*
Disallow: /advanced-search/*
Disallow: /genai-analytics/*
Disallow: /comscore-json/*
Disallow: /api/*
Disallow: /*.webarchive$

User-agent: Googlebot
User-agent: LinkedInBot
Disallow: /search/*
Disallow: /sponsored/
Disallow: /newsletter/
Allow: */feed/*/articles.xml
Allow: /feed/sitema*
Disallow: /wp-login.php
Disallow: /wp-signup.php
Disallow: /press-this.php
Disallow: /remote-login.php
Disallow: /activate/
Disallow: /cgi-bin/
Disallow: /mshots/v1/
Disallow: /next/
Disallow: /public.api/*
Disallow: /advanced-search/*
Disallow: /preview/*
Disallow: /genai-analytics/*
Disallow: /comscore-json/*
Disallow: /api/*
Disallow: /*.webarchive$

User-agent: ia_archiver
Disallow: /

History

Changes recorded since June 2026.

No change recorded.

In context

United States
63.2% block at least one AI crawler
Behind Cloudflare
No