How Search Engines, LLMs, and Third-Party Scrapers Affect Bot Management
Akamai Technologies, Monday, July 21st, 2025
Maybe you've been having conversations with your business and security teams regarding artificial intelligence (AI) scraper bots. After a period when blocking AI scrapers seemed futile - in fact, even harmful - there has recently been a renewed interest in the question of whether to block AI bots.
There has been much debate lately on the effectiveness and the usefulness of blocking AI bots. As the discussion of AI bots gains more attention, some of the key issues that surface include:
- The value of being included in GenAI answers
- Stopping known bots that don't want to be blocked
- Search engines and large language models (LLMs) that share information
- Charging AI bots to scrape
The value of being included in GenAI answers
The key reason that many organizations simply let the AI bots scrape content is that those organizations want to be included in answers to any relevant questions that a user might ask an LLM - for example, retail, hospitality, high tech, or business services organizations.