CrawlerPulse
CrawlerPulse / Europe / Hungary / hirstart.hu

hirstart.hu

hirstart.hu · Hungary · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Blocked
OpenAI
Blocked
Anthropic
Blocked
Perplexity
Allowed
Meta
Blocked
Common Crawl
Blocked
ByteDance
Blocked
Apple
Blocked
Amazon
Blocked

robots.txt

As read on 8 Oct 2026, 1,753 bytes. 2 rule groups name AI crawlers; shown first.

User-agent: GPTBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: CCBot
User-agent: Meta-ExternalAgent
User-agent: FacebookBot
User-agent: Bytespider
User-agent: Amazonbot
User-agent: cohere-ai
User-agent: cohere-training-data-crawler
User-agent: AI2Bot
User-agent: Diffbot
User-agent: Omgilibot
User-agent: Omgili
User-agent: ImagesiftBot
User-agent: Brightbot
Disallow: /

User-agent: Yandex
User-agent: Seznam
User-agent: SeznamBot
User-agent: PetalBot
User-agent: Sogou web spider
User-agent: Sosospider
User-agent: 360Spider
User-agent: HaosouSpider
User-agent: YoudaoBot
User-agent: JikeSpider
User-agent: Yeti
User-agent: Mail.ru
User-agent: MegaIndex.ru
User-agent: Mojeek
User-agent: Qwantify
Disallow: /

Sitemap: https://www.hirstart.hu/site/sitemap_https/sitemapindex.xml

User-agent: *
Allow: /site/rss.php?pid=8&id=1082&xmlGN=1&noct=1
Disallow: /site/rss.php
Disallow: /site/publicrss.php?pos=keres
Disallow: /site/publicrss.php?pos=koteg
Disallow: /site/publicrss.php?pos=fokoteg&fid=
Disallow: /site/publicrss.php?pos=box
Disallow: /site/publicrss.php?pos=osszesmai&pid=41
Disallow: /*rsspid=37
Disallow: /site/ajax/
Disallow: /site/jsembed_news.php
Disallow: /site/embed_news
Disallow: /site/app
Disallow: /site/relocate.php
Disallow: /site/h_relocate.php
Disallow: /site/urlclick.php
Disallow: /fokoteg.php
Disallow: /kepeskiemelt.xml

User-agent: BuiltWith
User-agent: Brandwatch
User-agent: Trendictionbot
User-agent: BuzzSumo
Disallow: /

History

Changes recorded since June 2026.

No change recorded.

In context

Hungary
6.7% block at least one AI crawler
Behind Cloudflare
No