CrawlerPulse
CrawlerPulse / North America / United States / therolladailynews.com

Rolla Daily News

therolladailynews.com · United States · USAToday · measured 8 Oct 2026

Google Search
Partial
Google-Extended
Partial
OpenAI
Partial
Anthropic
Partial
Perplexity
Partial
Meta
Partial
Common Crawl
Partial
ByteDance
Partial
Apple
Partial
Amazon
Partial

robots.txt

As read on 8 Oct 2026, 1,086 bytes. No rule names an AI crawler.

User-agent: *
Allow: /
Sitemap: https://www.therolladailynews.com/sitemap.xml
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/themes/
Disallow: /wp-content/cache/
Disallow: /wp-content/uploads/
Disallow: /trackback/
Disallow: /feed/
Disallow: /comments/
Disallow: /category/*/*
Disallow: */trackback/
Disallow: */feed/
Disallow: */comments/
Disallow: /*.php$
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Disallow: /*.gz$
Disallow: /*.cgi$
Disallow: /*.wmv$
Disallow: /*.cgi$
Disallow: /*.xhtml$
Disallow: /*.xlsx$
Disallow: /*.xls$
Disallow: /*.phps$
Disallow: /*.pptx$
Disallow: /*.ppt$
Disallow: /*.pdf$
Disallow: /*.docx$
Disallow: /*.doc$
Disallow: /*.php$
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Disallow: /*.gz$
Disallow: /*.cgi$
Disallow: /*.wmv$
Disallow: /*.cgi$
Disallow: /*.xhtml$
Disallow: /*.xlsx$
Disallow: /*.xls$
Disallow: /*.phps$
Disallow: /*.pptx$
Disallow: /*.ppt$
Disallow: /*.pdf$
Disallow: /*.docx$
Disallow: /*.doc$

History

Changes recorded since June 2026.

No change recorded.

In context

United States
63.2% block at least one AI crawler
Behind Cloudflare
No
Same group