CrawlerPulse
CrawlerPulse / Asia / Indonesia / bernas.id

Bernas.id

bernas.id · Indonesia · Bernas.id - Inspirasi Bisnis dan Pribadi Tumbuh · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Allowed
OpenAI
Blocked
Anthropic
Blocked
Perplexity
Allowed
Meta
Allowed
Common Crawl
Blocked
ByteDance
Blocked
Apple
Allowed
Amazon
Allowed

robots.txt

As read on 8 Oct 2026, 1,853 bytes. 1 rule group name AI crawlers; shown first.

User-Agent: GPTBot
User-Agent: CCBot
User-Agent: Bytespider
User-Agent: Anthropic-ai
User-Agent: ClaudeBot
User-Agent: Omgili
User-Agent: FacebookBot
User-Agent: Diffbot
Disallow: /
Sitemap: https://bernas.id/sitemap/0.xml
Sitemap: https://bernas.id/sitemap/1.xml
Sitemap: https://bernas.id/sitemap/2.xml
Sitemap: https://bernas.id/sitemap/3.xml
Sitemap: https://bernas.id/sitemap/4.xml
Sitemap: https://bernas.id/sitemap/5.xml
Sitemap: https://bernas.id/sitemap/6.xml
Sitemap: https://bernas.id/sitemap/7.xml
Sitemap: https://bernas.id/sitemap/8.xml
Sitemap: https://bernas.id/sitemap/9.xml
Sitemap: https://bernas.id/sitemap/10.xml
Sitemap: https://bernas.id/sitemap-news.xml
Sitemap: https://bernas.id/sitemap-journalists.xml
Sitemap: https://bernas.id/sitemap-pages.xml
Sitemap: https://bernas.id/sitemap-bpr.xml
Sitemap: https://bernas.id/sitemap-sbdk.xml
Sitemap: https://bernas.id/sitemap-ump.xml
Sitemap: https://bernas.id/sitemap-umk.xml
Sitemap: https://bernas.id/sitemap-profil-bisnis.xml
Sitemap: https://bernas.id/sitemap-karir-gaji.xml
Sitemap: https://bernas.id/sitemap-kesehatan.xml
Sitemap: https://bernas.id/sitemap-pendidikan.xml
Sitemap: https://bernas.id/sitemap-transportasi.xml
Sitemap: https://bernas.id/sitemap-properti.xml
Sitemap: https://bernas.id/sitemap-otomotif.xml
Sitemap: https://bernas.id/sitemap-wisata.xml
Sitemap: https://bernas.id/sitemap-pangan.xml
Sitemap: https://bernas.id/sitemap-kamus.xml
Sitemap: https://bernas.id/sitemap-dokumen.xml
Sitemap: https://bernas.id/sitemap-loker.xml

User-Agent: *
Allow: /
Disallow: /api/
Disallow: /dashboard/
Disallow: /admin/
Disallow: /next-admin/
Disallow: /bs-admin/
Disallow: /team
Disallow: /login
Disallow: /signup
Disallow: /checkout
Disallow: /register
Disallow: /feed/
Disallow: /search
Disallow: /*?s=
Disallow: /*?share=
Disallow: /extract/
Disallow: /actions/

History

Changes recorded since June 2026.

No change recorded.

In context

Indonesia
30.4% block at least one AI crawler
Behind Cloudflare
Yes