CrawlerPulse
CrawlerPulse / North America / United States / latimes.com

LA Times

latimes.com · United States · NantMedia Holdings, LLC · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Allowed
OpenAI
Blocked
Anthropic
Blocked
Perplexity
Allowed
Meta
Allowed
Common Crawl
Allowed
ByteDance
Blocked
Apple
Allowed
Amazon
Allowed

robots.txt

As read on 8 Oct 2026, 1,768 bytes. 7 rule groups name AI crawlers; shown first.

User-agent: Bytespider
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Diffbot
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: ImagesiftBot
Disallow: /

User-agent: omgili
Disallow: /

User-agent: omgilibot
Disallow: /
Sitemap: https://www.latimes.com/sitemaps/sitemap.xml
Sitemap: https://www.latimes.com/news-sitemap.xml

User-agent: *
Disallow: /search
Disallow: /recipes/search
Disallow: /changebrowser
Disallow: /thirdpartyservice
Disallow: /*/thirdpartyservice
Disallow: /config/
Disallow: /*/config
Disallow: /dzcfg
Disallow: /*/dzcfg
Disallow: /*/test-template
Disallow: /test/
Disallow: /includes/
Disallow: /_comments/
Disallow: /deeplinkid/
Disallow: /searchsuggest
Disallow: /*/searchsuggest
Disallow: /svgimageproc
Disallow: /hive/
Disallow: /get-galleryfragment*
Disallow: /gs-*

User-agent: Googlebot-News
Disallow: /*-ugc-*.html
Disallow: /about/
Disallow: /cbp-*.html
Disallow: /*/cbp-*.html
Disallow: /bp/*
Disallow: /*-adstory.html
Disallow: /*sns-ap
Disallow: /*photogallery
Disallow: /brandpublishing/
Disallow: /*/brandpublishing/
Disallow: /shopping/
Disallow: /*/shopping/
Disallow: /paid-posts/
Disallow: /*/paid-posts/
Disallow: /search
Disallow: /*/search
Disallow: /changebrowser
Disallow: /thirdpartyservice
Disallow: /*/thirdpartyservice
Disallow: /config/
Disallow: /*/config
Disallow: /dzcfg
Disallow: /*/dzcfg
Disallow: /*/test-template
Disallow: /test/
Disallow: /_comments/
Disallow: /deeplinkid/
Disallow: /searchsuggest
Disallow: /*/searchsuggest
Disallow: /svgimageproc
Disallow: /hive/
Disallow: /get-galleryfragment*
Disallow: /sponsored/
Disallow: /*/sponsored/

User-agent: Twitterbot
Allow: /

User-agent: Mediapartners-Google
Allow: /

User-agent: AmazonAdBot
Allow: /

User-agent: discobot
Disallow: /

History

Changes recorded since June 2026.

No change recorded.

In context

United States
63.2% block at least one AI crawler
Behind Cloudflare
No
Same group