The Crawl Before The Fall... Of Referrals: Understanding AI's Impact On Content Providers
Cloudflare, July 1,2025
Content publishers welcomed crawlers and bots from search engines because they helped drive traffic to their sites.
The crawlers would see what was published on the site and surface that material to users searching for it. Site owners could monetize their material because those users still needed to click through to the page to access anything beyond a short title.