CrawlerPulse
CrawlerPulse / North America / Dominican Republic / hoy.com.do

Hoy

hoy.com.do · Dominican Republic · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Allowed
OpenAI
Allowed
Anthropic
Allowed
Perplexity
Allowed
Meta
Allowed
Common Crawl
Allowed
ByteDance
Allowed
Apple
Allowed
Amazon
Allowed

robots.txt

As read on 8 Oct 2026, 2,475 bytes. No rule names an AI crawler.

User-agent: *
Allow: /
Sitemap: https://hoy.com.do/sitemap-google-news.xml
Sitemap: https://hoy.com.do/sitemap-index.xml

User-agent: *
Allow: /wp-content/uploads/*
Allow: /wp-content/*.js
Allow: /wp-content/*.css
Allow: /wp-includes/*.js
Allow: /wp-includes/*.css
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-admin/
Disallow: /cgi-bin
Disallow: /wp-content/plugins/
Disallow: /wp-content/themes/
Disallow: /wp-includes/
Disallow: /*/attachment/
Disallow: /tag/*/page/
Disallow: /tag/*/feed/
Disallow: /page/
Disallow: /comments/
Disallow: /xmlrpc.php
Disallow: /?attachment_id*

User-agent: *
Disallow: /mail/

User-agent: *
Disallow: /cdn-cgi/
Disallow: /cdn-cgi/

User-agent: *
Disallow: /?s=
Disallow: /search/
Disallow: /search
Disallow: /?p=

User-agent: *
Disallow: /trackback
Disallow: /*trackback
Disallow: /*trackback*
Disallow: /*/trackback
Allow: /feed/$
Disallow: /feed/
Disallow: /comments/feed/
Disallow: /*/feed/$
Disallow: /*/feed/rss/$
Disallow: /*/trackback/$
Disallow: /*/*/feed/$
Disallow: /*/*/feed/rss/$
Disallow: /*/*/trackback/$
Disallow: /*/*/*/feed/$
Disallow: /*/*/*/feed/rss/$
Disallow: /*/*/*/trackback/$

User-agent: noxtrumbot
Crawl-delay: 20

User-agent: msnbot
Crawl-delay: 5

User-agent: Slurp
Crawl-delay: 20

User-agent: Slurp
Crawl-delay: 20

User-agent: AdIdxBot
Crawl-delay: 5

User-agent: BingPreview
Crawl-delay: 5

User-agent: Yandex
Crawl-delay: 5

User-agent: YandexCalendar
Crawl-delay: 5

User-agent: YandexMobileBot
Crawl-delay: 5

User-agent: YandexAccessibilityBot
Crawl-delay: 5

User-agent: YandexAdNet
Crawl-delay: 5

User-agent: YandexBlogs
Crawl-delay: 5

User-agent: YandexBot
Crawl-delay: 5

User-agent: YandexDirect
Crawl-delay: 5

User-agent: YandexDirectDyn
Crawl-delay: 5

User-agent: YandexFavicons
Crawl-delay: 5

User-agent: YaDirectFetcher
Crawl-delay: 5

User-agent: YandexMarket
Crawl-delay: 5

User-agent: YandexMedia
Crawl-delay: 5

User-agent: YandexMetrika
Crawl-delay: 5

User-agent: YandexMobileScreenShotBot
Crawl-delay: 5

User-agent: YandexNews
Crawl-delay: 5

User-Agent: Googlebot
Allow: /*.css$
Allow: /*.js$

History

Changes recorded since June 2026.

No change recorded.

In context

Dominican Republic
6.2% block at least one AI crawler
Behind Cloudflare
No