Per-crawler totals, 746 sites with a readable robots.txt, checked 21 to 22 October 2026 (number of sites that block it, and what the operator says it's for):

CCBot (Common Crawl, training): 146 Bytespider (ByteDance, training): 137 ClaudeBot (Anthropic, training): 135 Applebot-Extended (Apple, token for AI use of Applebot's fetches): 127 GPTBot (OpenAI, training): 126 Meta-ExternalAgent (Meta, training): 125 PerplexityBot (Perplexity, search): 114 Google-Extended (Google, token for AI use of Googlebot's fetches): 101 Amazonbot (Amazon, general): 99 OAI-SearchBot (OpenAI, search): 70 ChatGPT-User (OpenAI, fetches triggered by a user): 69 Claude-User (Anthropic, fetches triggered by a user): 66 Perplexity-User (Perplexity, fetches triggered by a user): 65 Claude-SearchBot (Anthropic, search): 62

Same data as a CSV, one row per site: getkavel.com/ai-crawlers/data.csv

undefined

[flagged]

[flagged]