Common Crawl. Builds a free, open archive of the web, widely used to train AI models. Common Crawl's documentation.
of 12,650 measured news outlets block CCBot.
Share of each continent's measured outlets blocking CCBot.
Countries with 50 or more outlets measured, from the most blocking to the least.
By public reach. One cell per crawler.
By public reach.