CrawlerPulse
CrawlerPulse / Europe / Italy / agi.it

AGI

agi.it · Italy · AGI - Agenzia Giornalistica Italia Spa · measured 8 Oct 2026

Google Search
Allowed
Google-Extended
Allowed
OpenAI
Allowed
Anthropic
Allowed
Perplexity
Allowed
Meta
Allowed
Common Crawl
Allowed
ByteDance
Allowed
Apple
Allowed
Amazon
Allowed

robots.txt

As read on 8 Oct 2026, 2,387 bytes. No rule names an AI crawler.

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <title></title>
    <style>
        body {
            font-family: "Arial";
        }
    </style>
    <script type="text/javascript">
    window.awsWafCookieDomainList = [];
    window.gokuProps = {
"key":"AQIDAHjcYu/GjX+QlghicBgQ/7bFaQZ+m5FKCMDnO+vTbNg96AEzbEO6e5rLu+5oVjzKQ9TaAAAAfjB8BgkqhkiG9w0BBwagbzBtAgEAMGgGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQMtg3svU7fRdPTb0TCAgEQgDt7zPduxkSDF1tOW1JrclqkhS8YqCeUvAtoKM0EjiX+ZRtIKWcEvCgCQqp1xym8TYBax3lpgpr/tuh9ng==",
          "iv":"D54BdAD8YAAADONP",
          "context":"uhjNxcLrEYVIUFUnFhXipFjVdK2mkzAxBY1zOudBae9qhpYEy2hW3qWrpr1mtcAJnXxiecH6hRlh9sWyN8kRTA3KxjXwvGVWOVj6zT/uvQmshyHmYrUYDpWqs2zZT3EL+mJQobqrQpzQeIr45CXV17AMMCU0Zch7wEIWfWiVW5Oy7znRcgFHF4aDf7XYmb/wd/S3Q/jeXFKSrbVKlcrIsekp1gDjeREh8P8IdpV2cJp6PKQZY3IPPf+ep30+O+Eyf6gQVKs+doXEOkbBHaAkAKD+mmikuE+IMFYiCs3vz5l+qaOMcSN5YDVdVALSozwiUYVeCq9Bh7iLyQ2GjyCyjZzaYKdrPbY4lkNAeJ9E5wMk8wGP3KYDFIDZy1NU9cVWXy66OMpsW0hle2ZTe/rSTuenbxArXxpDJIdDQ2LI7LeBmE5TgEyrk0vc7dLCM+wQH5hEmFDjMzQyaFnhQtA25UiT3CJ+tIAtSawypaIKp9oGRv3WIG4dOHYCZM6A34+7EpSyUNvfijTvEU5oEziaSPcxlZ5KIoEWrN2uYKelh2qbtD4dVqCmv4nn4DLmpVzzm6lqBxVKRpVzTpR+dXlpe9gqs2sPIUB714IDF2HdRN6uqDbpTkSpms5Z0iMO6z+gId/+pgHJ1pjNz/17DnVq+FCME5PaOjvAe/rzW1A5tHhJ+ms795X8JHeKpfeaBjTHi3TKRJqlsA+bACi0HWa/RuiKWnqCiPCL2ARl/2yLxe6P5tElNkioPJJbI8hqlkWOjBM="
};
    </script>
    <script src="https://a49c870577a3.4b800a64.eu-west-1.token.awswaf.com/a49c870577a3/748f688d6c36/b3c9036127c4/challenge.js"></script>
</head>
<body>
    <div id="challenge-container"></div>
    <script type="text/javascript">
        AwsWafIntegration.saveReferrer();
        AwsWafIntegration.checkForceRefresh().then((forceRefresh) => {
            if (forceRefresh) {
                AwsWafIntegration.forceRefreshToken().then(() => {
                    window.location.reload(true);
                });
            } else {
                AwsWafIntegration.getToken().then(() => {
                    window.location.reload(true);
                });
            }
        });
    </script>
    <noscript>
        <h1>JavaScript is disabled</h1>
        In order to continue, we need to verify that you're not a robot.
        This requires JavaScript. Enable JavaScript and then reload the page.
    </noscript>
</body>
</html>

History

Changes recorded since June 2026.

No change recorded.

In context

Italy
54.5% block at least one AI crawler
Behind Cloudflare
No