ITSM Concerns When Integrating New AI Services
HelpNet Security, Tuesday, August 6th, 2024
Late last year, a Chevrolet dealership deployed a chatbot powered by a large language model (LLM) on their homepage. This LLM, trained with detailed specifications of Chevrolet vehicles, was intended to respond only to questions about Chevrolet cars.
However, users quickly found a way to circumvent these limitations: through a series of leading questions that fell increasingly outside the intended range of the chatbot's answers, they ended up prompting the chatbot to recommend Tesla vehicles instead. Soon, the bot was being manipulated into writing code, and even offering to sell cars for one dollar.
A more alarming incident involved Microsoft's Co-Pilot, an AI tool designed to assist in writing code. Due to the phenomenon of AI hallucinations - where the AI generates plausible yet false information - the tool suggested a non-existent library. Seizing this opportunity, a developer created a library by that name, embedded malware within it, and uploaded it to GitHub. Within four days, the malicious library had 100,000 downloads.