Bots Now Drive 30% of Global Web Traffic, Surpassing Human Visitors in Some Regions

A technological shift is transforming the digital landscape: bots now generate 30% of global web traffic, according to new data from Cloudflare Radar.

In certain regions, automated scripts and web crawlers even outpace human visitors, reshaping the way content is indexed, discovered, and secured online.

This surge is driven by the rise of advanced AI crawlers, marking a profound evolution in the Internet’s core infrastructure.

Web crawlers or “bots” have long formed the backbone of search engines, indexing websites so users can find information quickly.

Traditionally, these have included “good” bots, such as Googlebot, and “bad” bots performing malicious tasks.

But in the past year, a new class has emerged: AI crawlers. These tools systematically collect data from across the web to train Large Language Models (LLMs) and power intelligent services, such as ChatGPT, Claude, and Meta AI.

Technical Shift: Dominance of Googlebot and AI Agents

Cloudflare’s analysis between May 2024 and May 2025 captures a dramatic reordering among top web crawlers.

OpenAI’s GPTBot jumped from a 2.2% share to 7.7% (a staggering 305% increase in requests), while Meta’s newly introduced crawler rapidly gained ground.

Meanwhile, traditional players rearranged: Googlebot cemented its dominance with a 50% share of all crawler traffic, up from 30% the previous year, nearly doubling its raw request volume.

In contrast, Chinese tech giant ByteDance’s Bytespider dropped sharply, falling from 22.8% to just 2.9% of crawler requests.

The technical landscape further reveals that AI bots are not just growing in presence but in sophistication. Many now attempt real-time indexing, API integration, and adaptive crawling based on site permissions.

Crawlers like PerplexityBot and ChatGPT-User, while niche, posted triple- and even quintuple-digit growth, echoing the soaring demand for web data in AI applications.

The New Battle Over Content Control

This explosion in non-human traffic isn’t without controversy. Content creators and publishers are increasingly deploying defensive measures, such as updating their robots.txt files or using web application firewalls, to manage or block AI crawlers, particularly due to concerns about unauthorized data use and infrastructure overload.

Cloudflare reports that 14% of top domains now explicitly allow or deny access to AI bots, with OpenAI’s GPTBot being the most frequently blocked crawler.

As AI and search become ever more entwined, the web’s future appears set for a new era of negotiation between open access, proprietary training data, and the technical arms race to control or harness billions of automated data requests.

One thing is clear: bots are not just a behind-the-scenes force anymore; they’re now shaping the very experience and economics of the modern Internet.