CrawlerPulse
CrawlerPulse / Asia / Pakistan / urdupoint.com

UrduPoint.com, Urdu News, Poetry Technology Sports, Health and more

urdupoint.com · Pakistan · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Blocked
OpenAI
Blocked
Anthropic
Blocked
Perplexity
Allowed
Meta
Allowed
Common Crawl
Blocked
ByteDance
Blocked
Apple
Blocked
Amazon
Allowed

robots.txt

As read on 8 Oct 2026, 3,385 bytes. 10 rule groups name AI crawlers; shown first.

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: meta-externalagent
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: facebookexternalhit
Allow: /

User-agent: Facebot
Allow: /

User-agent: *
Disallow: /dev
Disallow: /dictionary/english-word-
Disallow: /dictionary/urdu-word-
Disallow: /dictionary/roman-urdu-word-
Allow: /
Sitemap: https://www.urdupoint.com/sitemap/daily/xml/2018_index.xml
Sitemap: https://www.urdupoint.com/sitemap/content/xml/2018_index.xml
Sitemap: https://www.urdupoint.com/sitemap/index-en.xml
Sitemap: https://www.urdupoint.com/sitemap/index-ar.xml
Sitemap: https://www.urdupoint.com/en/sitemap/news.xml
Sitemap: https://www.urdupoint.com/names/sitemaps/data/roots.xml.gz
Sitemap: https://www.urdupoint.com/names/sitemaps/data/urdu_links.xml.gz
Sitemap: https://www.urdupoint.com/sitemap/names.xml.gz
Sitemap: https://www.urdupoint.com/sitemap/islam.xml.gz
Sitemap: https://www.urdupoint.com/sitemap/islamic-names.xml.gz
Sitemap: https://www.urdupoint.com/sitemap/islam-prayers.xml.gz
Sitemap: https://www.urdupoint.com/sitemap/business.xml.gz
Sitemap: https://www.urdupoint.com/business/sitemap/business-index.xml
Sitemap: https://www.urdupoint.com/sitemap/dictionary.xml
Sitemap: https://www.urdupoint.com/dictionary/sitemap/index.xml
Sitemap: https://www.urdupoint.com/sitemap/names-old.xml.gz
Sitemap: https://www.urdupoint.com/sitemap/names-urdu.xml.gz
Sitemap: https://www.urdupoint.com/sitemap/tv.xml.gz
Sitemap: https://www.urdupoint.com/sitemap/poetry.xml
Sitemap: https://www.urdupoint.com/sitemap/travel.xml.gz
Sitemap: https://www.urdupoint.com/cooking/sitemap/cookingOLD.xml.gz
Sitemap: https://www.urdupoint.com/cooking/sitemap/data/detail.xml.gz
Sitemap: https://www.urdupoint.com/cooking/sitemap/data/root.xml.gz
Sitemap: https://www.urdupoint.com/cooking/sitemap/cooking.xml.gz
Sitemap: https://www.urdupoint.com/sitemap/showbiz.xml.gz
Sitemap: https://www.urdupoint.com/kids/sitemap/data/kids.xml.gz
Sitemap: https://www.urdupoint.com/sitemap/women.xml.gz
Sitemap: https://www.urdupoint.com/sitemap/index-horoscope.xml
Sitemap: https://www.urdupoint.com/sitemap/horoscope.xml.gz
Sitemap: https://social.urdupoint.com/sitemaps/social.xml.gz
Sitemap: https://www.urdupoint.com/sitemap/health.xml.gz
Sitemap: https://www.urdupoint.com/sitemap/loc_int/main.xml.gz
Sitemap: https://www.urdupoint.com/sitemap/sports.xml.gz
Sitemap: https://www.urdupoint.com/sitemap/politics.xml.gz
Sitemap: https://www.urdupoint.com/sitemap/mazah.xml.gz
Sitemap: https://www.urdupoint.com/islam/sitemap/data/hadees.xml.gz
Sitemap: https://www.urdupoint.com/islam/sitemap/data/events.xml.gz
Sitemap: https://www.urdupoint.com/islam/sitemap/data/ayat.xml.gz

History

Changes recorded since June 2026.

No change recorded.

In context

Pakistan
6.6% block at least one AI crawler
Behind Cloudflare
Yes