Cloudflare has introduced a new feature allowing customers to block AI bots, scrapers, and crawlers with a single click. This move comes in response to the increasing demand for content used to train AI models and run inference. This post discusses the prevalence of AI bot activity, including both well-behaved and evasive bots, and Cloudflare's efforts to detect and block them using advanced machine learning models.
Key points:
One-click option to block all AI bots
Top AI bots by request volume: Bytespider, Amazonbot, ClaudeBot, and GPTBot
Only 2.98% of top 1 million websites block AI bot requests
Cloudflare's machine learning models can detect evasive AI bots
Customers can report misbehaving AI crawlers through feedback tools