Reporters with Cybernews recently disclosed a number of serious security issues discovered in “Lena,” the Lenovo AI chatbot used for customer support queries via the company website. This includes the ability to steal active session cookies with just a single prompt that appears to be a request for product information.
The researchers ethically disclosed the vulnerabilities to Lenovo before publication, and it is unclear if or to what extent malicious hackers might have taken advantage of the situation prior to this. Regardless, the incident demonstrates how potentially vulnerable AI chatbots can be even when sourced from the industry’s biggest names (in this case ChatGPT). It also highlights the ongoing problem of “people pleasing” functions in AI that can be readily exploited by creative attackers.
Lenovo AI chatbot tricked into assisting with session cookie theft
Lena is a fairly standard AI chatbot of the type seen at some company websites, offering to retrieve product information and answer customer service questions for visitors. What is different about this one is the relative ease with which it could be recruited for highly damaging actions before it was patched, requiring only relatively simple prompt injections to pose a threat both to Lenovo’s internal network and website visitors.
The central attack that the researchers demonstrate is theft of a website user’s session cookies, all initiated by simply asking the AI chatbot to show the specs of a particular Lenovo product. The trick is to ask it to output the reply in HTML, JSON and plain text in a specific order that the bot will then read as instructions. It is then instructed to display an image from included HTML code that is intentionally broken. When the image fails to load, the bot receives another command to use an attacker-controlled server as a secondary source but also packs in the session cookie data as part of the URL. A malicious attacker would then have the ability to initiate a chat with a real agent and obtain their session cookies as well, allowing them a means of entry into the customer support system. This “hidden code” could theoretically be modified in a lot of other ways that the researchers did not test: keylogging, redirecting the customer support agent to an attack site or pushing a malicious download, or altering the interface they are seeing, among some of the possibilities.
While Lenovo’s AI chatbot apparently did have some safety guardrails against this approach, the researchers say that they could be trivially overcome by being insistent (for example telling the bot that viewing the image was absolutely essential for making a purchase decision). This points to “people pleasing” as the culprit, an ongoing problem for AI models in terms of both obsequious and condescending tone and the creation of security holes such as this. Models tend to ship quickly and bias too hard toward doing exactly what the user asks, to the point of leaving numerous creative means to shake off security restrictions.
Organizations must devote time to input and output sanitization to bolster AI chatbot shortcomings
Though AI chatbots should by and large be considered risky in this way, the pressure to implement them is too immense for many industries. The researchers note that this vulnerability situation is very similar to the development of cross-site scripting (XSS) attacks in the early days of the commercial internet, and that the hardening approach is also similar. The primary point of focus is input and output sanitization, implementing policies and practices that assume every user prompt is potentially unsafe.
At the input end, organizations must take time to implement strict and carefully developed command whitelists that restrict potentially problematic characters, limit input length to preclude attempts at buffer overflows, and limit data types and formats to those necessary to the task. Output must be filtered in the same way, most essentially stripped of any embedded code when displayed in a browser or rich text environment and directed by policy to restrict potentially dangerous HTML elements, attributes and browser resources. Finally, the rest of the stack can be similarly secured by clamping down on permissions and implementing content type validation with web servers, apps, and data storage that are part of the AI chatbot’s process.
More generally, users of AI chatbots must be aware that the “guardrails” put into place to secure them are not necessarily hard rules that can or will hold up in every possible circumstance. “Prompt hackers” are constantly working away at them with inventive approaches that are often sold on underground forums or converted into a “jailbroken” model packaged for sale to criminals. Security also does not often go hand-in-hand with rapid development and big jumps in capability; Deepseek demonstrated this aptly earlier this year after rocketing to the top of app download charts.
Melissa Ruzzi, director of AI at AppOmni, adds: “This is one more story highlighting the well-known issue of prompt injection on Generative AI. It’s crucial to oversee all the data access the AI has, which most of the time includes not only read permissions, but also the ability to edit. That could make this type of attack even more devastating. AI chatbots can be seen as another SaaS app, where data access misconfigurations can easily turn into data breaches. Every company using a customer service chatbot should stay up to date on best practices in prompt engineering to hinder these attacks as much as possible. They should also implement additional checks to limit how the AI interprets prompt content, and monitor and control data access of the AI. More than ever, security should be an intrinsic part of all AI implementation. Although there is pressure to release AI features as fast as possible, this must not compromise proper data security.”
