thegatewaypundit.com · United States · measured 9 Oct 2026
As read on 9 Oct 2026, 3,328 bytes. 7 rule groups name AI crawlers; shown first.
User-agent: Amazonbot Disallow: / User-agent: CCBot Disallow: / User-agent: FacebookBot Disallow: / User-agent: Amazonbot Disallow: / User-agent: Bytespider Disallow: / User-agent: CCBot Disallow: / User-agent: Meta-ExternalAgent Disallow: / Sitemap: https://www.thegatewaypundit.com/sitemap.xml Sitemap: https://www.thegatewaypundit.com/news-sitemap.xml User-agent: * Crawl-delay: 30 User-agent: Parler User-agent: ParlerStaging User-agent: bingbot User-agent: msnbot User-agent: msnbot-media Crawl-delay: 120 Disallow: /wp-content/uploads/* Disallow: /?s= Disallow: /? Disallow: /?* Disallow: /search/ Disallow: /wp-admin/ Disallow: /wp-login.php Disallow: /members/ Disallow: /admin_page/ Disallow: /admin_page/* Disallow: /campaign/ Disallow: /twitter/ Disallow: /youtube/ Allow: /wp-admin/admin-ajax.php User-agent: * Disallow: /*?p=* Disallow: /*&p=* Disallow: /*?s=* Disallow: /*&s=* Disallow: /*?ical=1 Disallow: /*&ical=1 Disallow: /*?tribe-bar-date=* Disallow: /*&tribe-bar-date=* Disallow: /?author=* Disallow: /*wp-comments* Disallow: /*wp-trackback* Disallow: /*wp-feed* Disallow: /*replytocom=* Disallow: /*?preview=* Disallow: /*&preview=* Disallow: /*add-to-cart=* Disallow: /*add_to_wishlist=* Disallow: /*cart/* Disallow: /*checkout/* Disallow: /*my-account/* Disallow: /*myaccount/* Disallow: /*?ajaxCalendar=1* Allow: /*/plugins/* User-agent: Twitterbot Allow: * User-agent: facebookexternalhit Allow: * User-agent: facebot Allow: * User-agent: Googlebot Allow: * User-agent: rogerbot Disallow: / User-agent: dotbot Disallow: / User-agent: Screaming Frog SEO Spider Disallow: / User-agent: MJ12bot Disallow: / User-agent: DataForSeoBot Disallow: / User-agent: RavenCrawler Disallow: / User-agent: * Crawl-delay:1 Disallow: /*?p=* Disallow: /*&p=* Disallow: /*?s=* Disallow: /*&s=* Disallow: /*?ical=1 Disallow: /*&ical=1 Disallow: /?author=* Disallow: /*wp-comments* Disallow: /*wp-trackback* Disallow: /*wp-feed* Disallow: /*replytocom=* Disallow: /*?preview=* Disallow: /*&preview=* Disallow: /*add-to-cart=* Disallow: /*add_to_wishlist=* Disallow: /*cart/* Disallow: /*checkout/* Disallow: /*my-account/* Disallow: /*myaccount/* Disallow: /*?ajaxCalendar=1* Disallow: /*&ajaxCalendar=1* Disallow: /*?wpdmdl=* Disallow: /*&wpdmdl=* Allow: /*/plugins/* User-agent: AhrefsBot Disallow: / User-agent: AhrefsSiteAudit Disallow: / User-agent: rogerbot Disallow: / User-agent: dotbot Disallow: / User-agent: SemrushBot Disallow: / User-agent: SemrushBot-SA Disallow: / User-agent: SemrushBot-BA Disallow: / User-agent: SemrushBot-SI Disallow: / User-agent: SemrushBot-SWA Disallow: / User-agent: SemrushBot-CT Disallow: / User-agent: SemrushBot-BM Disallow: / User-agent: SplitSignalBot Disallow: / User-agent: Screaming Frog SEO Spider Disallow: / User-agent: MJ12bot Disallow: / User-agent: DataForSeoBot Disallow: / User-agent: RavenCrawler Disallow: / User-agent: GoogleOther Disallow: / User-agent: TikTokSpider Disallow: /
Changes recorded since June 2026.
No change recorded.