gcdTrackerchecking sources…

Traffic

AI systems read the web as well as write to it. This page follows the requests: published measurements from network operators and crawler policies in sampled hosts.

4.2%
of HTML requests from AI bots other than Googlebot
Cloudflare Radar, Dec 2025
15×
growth in user-triggered AI fetches in 2025
Cloudflare Radar, Dec 2025
source snapshot
Cloudflare Radar data
Fetched 17 Oct, 03:47 UTC

Internet-scale, from Cloudflare Radar

radar.cloudflare.com/ai-insights ↗

Cloudflare fronts a large share of the web and publishes bot statistics under CC BY 4.0 (dashboard) and CC BY-NC 4.0 (API). Training crawlers collect content for model development; search and retrieval bots index pages or fetch them in response to a user; referred readers are people following links from AI platforms. The figures describe Cloudflare's vantage point, not internet-wide totals.

Bot traffic by bot · percent

Window: 10 Oct, 03:00 UTC – 17 Oct, 03:00 UTC. percent. Source updated 17 Oct, 02:30 UTC. Fetched 17 Oct, 03:47 UTC. Normalized series describe this snapshot and are not raw request counts.

other
37.48%
GoogleBot
11.88%
ChatGPT agent
8.97%
Claude
5.42%
Meta-ExternalAgent
4.33%
Baiduspider
3.97%
FacebookExternalHit
3.97%
BingBot
3.88%
Applebot
3.71%
Google AdsBot
3.41%
Meta-ExternalAds
2.94%
ShapBot
2.84%
Crawl/referral metric · source normalization: RATIO

Window: 10 Oct, 03:00 UTC – 17 Oct, 03:00 UTC. source normalization: RATIO. Source updated 17 Oct, 02:30 UTC. Fetched 17 Oct, 03:47 UTC. Normalized series describe this snapshot and are not raw request counts.

Mistral: No finite crawl/referral ratio reported.

Perplexity
3,334.88
OpenAI
1,399.13
Anthropic
465.01
Baidu
55.7
Microsoft
39.5
ByteDance
9.07
Yandex
6.98
Google
4.53
DuckDuckGo
4.04

Window: 19 Sep, 00:00 UTC – 17 Oct, 00:00 UTC. percent. Source updated 17 Oct, 02:30 UTC. Fetched 17 Oct, 03:47 UTC. Normalized series describe this snapshot and are not raw request counts.

other · percent
19 Sep: 31.04%20 Sep: 30.6%21 Sep: 29.77%22 Sep: 30.29%23 Sep: 29.92%24 Sep: 29.48%25 Sep: 29.37%26 Sep: 29.34%27 Sep: 29.14%28 Sep: 29.63%29 Sep: 30%30 Sep: 29.68%1 Oct: 29.33%2 Oct: 28.23%3 Oct: 28.02%4 Oct: 27.51%5 Oct: 28.5%6 Oct: 29.21%7 Oct: 30.32%8 Oct: 30.64%9 Oct: 25.25%10 Oct: 29.63%11 Oct: 30.38%12 Oct: 29.8%13 Oct: 29.17%14 Oct: 28.95%15 Oct: 29.32%16 Oct: 30.22%17 Oct: 30.01%19 Sep17 Octmax 50
Apple · percent
19 Sep: 5.53%20 Sep: 5.83%21 Sep: 4.79%22 Sep: 5.51%23 Sep: 4.98%24 Sep: 4.39%25 Sep: 5.31%26 Sep: 4.99%27 Sep: 4.65%28 Sep: 4.43%29 Sep: 4.88%30 Sep: 4.81%1 Oct: 4.76%2 Oct: 4.85%3 Oct: 3.91%4 Oct: 4.41%5 Oct: 3.82%6 Oct: 4.11%7 Oct: 4.09%8 Oct: 4.23%9 Oct: 3.54%10 Oct: 4.16%11 Oct: 3.81%12 Oct: 2.79%13 Oct: 3.01%14 Oct: 3.44%15 Oct: 4.25%16 Oct: 4.58%17 Oct: 5.13%19 Sep17 Octmax 10
Anthropic · percent
19 Sep: 8.14%20 Sep: 8.1%21 Sep: 7.59%22 Sep: 7.58%23 Sep: 7.23%24 Sep: 6.93%25 Sep: 7.25%26 Sep: 7.12%27 Sep: 6.94%28 Sep: 6.7%29 Sep: 6.51%30 Sep: 6.31%1 Oct: 6.3%2 Oct: 6.37%3 Oct: 6.18%4 Oct: 6.15%5 Oct: 6.03%6 Oct: 5.94%7 Oct: 5.84%8 Oct: 5.99%9 Oct: 5.01%10 Oct: 6.33%11 Oct: 6.68%12 Oct: 6.31%13 Oct: 6.34%14 Oct: 6.22%15 Oct: 6.49%16 Oct: 6.56%17 Oct: 7.32%19 Sep17 Octmax 10
OpenAI · percent
19 Sep: 5.14%20 Sep: 4.97%21 Sep: 4.65%22 Sep: 4.54%23 Sep: 4.74%24 Sep: 5.33%25 Sep: 5.37%26 Sep: 4.92%27 Sep: 4.6%28 Sep: 4.89%29 Sep: 4.93%30 Sep: 4.85%1 Oct: 4.57%2 Oct: 4.32%3 Oct: 9.16%4 Oct: 9.72%5 Oct: 11.45%6 Oct: 12.31%7 Oct: 12.43%8 Oct: 13.04%9 Oct: 29.06%10 Oct: 15.45%11 Oct: 11.97%12 Oct: 12.99%13 Oct: 14.03%14 Oct: 13.63%15 Oct: 13.13%16 Oct: 13.28%17 Oct: 12.41%19 Sep17 Octmax 50

Crawler rules in sampled robots.txt files

Common Crawl robots.txt archive ↗

Every Common Crawl crawl (roughly monthly) archives the robots.txt of each site it visits. A worker samples 100 of those archive files per crawl, spread across the crawl, and counts sites whose robots.txt names an AI crawler and sites that block it completely (Disallow: / for that agent). Shares are of all sampled sites with a readable robots.txt, so they describe this Common Crawl sample. They cannot establish a whole-web rate, and sites that exclude Common Crawl are underrepresented.

Share of sampled sites that fully block, per crawl001.31.32.52.53.83.8557 Aug: 3 Sites blocking GPTBot (%)7 AugSites blocking GPTBot (%)Sites blocking ClaudeBot (%)
Blocked completely · crawl of 7 August 2026 · 38,739 sites sampled
PetalBot
8.08%
GPTBot
3.50%
Amazonbot
3.40%
Bytespider
3.22%
ClaudeBot
3.13%
CCBot
3.09%
meta-externalagent
2.99%
Google-Extended
2.89%
Applebot-Extended
2.79%
ChatGPT-User
0.29%
anthropic-ai
0.29%
FacebookBot
0.23%
ImagesiftBot
0.22%
Diffbot
0.20%

Accent: fully blocked. Grey: named at all. Googlebot is blocked by 0.07% of the same sites, for scale.

Per crawl
CrawlSitesGPTBotClaudeBotCCBotGoogle-Extended
7 August 202638,7393.50%3.13%3.09%2.89%

Crawler share, 2024 → 2025

source ↗
CrawlerMay 2024May 2025
Googlebot30%50%
GPTBot2.2%7.7%
ClaudeBot11.7%5.4%
Bytespider22.8%2.9%

Historical share of crawler traffic observed by Cloudflare in the labelled publication periods.