Why We're Rethinking Cache For The AI Era
Cloudflare, Thursday, April 2nd, 2026
Cloudflare data shows that 32% of traffic across our network originates from automated traffic. This includes search engine crawlers, uptime checkers, ad networks - and more recently, AI assistants looking to the web to add relevant data to their knowledge bases as they generate responses with retrieval-augmented generation (RAG).
Unlike typical human behavior, AI agents, crawlers, and scrapers' automated behavior may appear aggressive to the server responding to the requests.
For instance, AI bots frequently issue high-volume requests, often in parallel. Rather than focusing on popular pages, they may access rarely visited or loosely related content across a site, often in sequential, complete scans of the websites. For example, an AI assistant generating a response may fetch images, documentation, and knowledge articles across dozens of unrelated sources.