A new theoretical attack described by researchers with LayerX lays out how frighteningly simple it would be for a malicious or compromised browser extension to intercept user chats with LLMs and insert prompt injection attacks designed to exfiltrate data without the target being aware.
The attack could be accomplished with potentially any type of browser extension, if it was designed from the ground up for this purpose or if it was compromised and altered appropriately by hackers. While organizations often restrict installation of browser extensions to those that are whitelisted or approved for business need, the malicious extensions involved do not require any special permissions and the approach presents considerable risk to those with looser security policies or that have existing shadow IT issues.
Prompt injection attacks pose unique threat to internal LLMs
Any malicious extension could potentially issue prompt injection attacks that give orders to steal and exfiltrate data to LLMs, while simultaneously hiding these prompts from the user and covering other getaway tracks such as the chat history.
The vulnerability stems from the fundamental way in which generative AI (GenAI) tools have their prompt input fields implemented in browsers. They broadly use the page’s Document Object Model (DOM), something that is available to any browser extension that has scripting access; the extension can thus both read and write prompts directly to the LLM without user interaction.
The so-called “Man in the Prompt” attack presents two priority risks. One is to internal LLMs that store sensitive company data and personal information, in the belief that it is appropriately fenced off from other software and apps. The other risk comes from particular LLMs that are broadly integrated into workspaces, such as Google Gemini’s interaction with (and wide-ranging permissions to access) Google Workspace tools such as Mail and Docs.
This category of prompt injection attacks applies not just to any type of browser extension, but any model or deployment of LLM (including internal tools that make use of an LLM frontend via a browser and AI-enabled SaaS applications). And the malicious extension requires no special permissions to work, given that the DOM access already provides everything it needs.
Researchers able to create two functioning proof-of-concepts
One of the working proof-of-concepts the researchers were able to craft targets ChatGPT via a command-and-control server that can be installed either locally or hosted remotely. Once in place, the malicious extension opens a background tab to run its queries to the LLM, exfiltrates the chat to a hidden log, then deletes the entirety of the chat so that the user will not notice it if they review their chat history.
The other proof-of-concept targets Google Gemini, and by extension any elements of Google Workspace it has been integrated with. Gemini is meant to automate routine and tedious tasks in Workspace such as email responses, document editing and updating contacts. The trouble is that it has almost complete access to the contents of these accounts as well as anything the user has access permission for or has had shared with them by someone else. Prompt injection attacks conducted by these extensions can not only steal the contents of emails and documents with ease, but complex queries can be fed to the LLM to target particular types of data and file extensions; the autocomplete function can also be abused to enumerate available files.
Internal LLMs are at even greater risk from these prompt injection attacks. Aside from there being a false sense of security from them seemingly being “siloed” from any avenues of dangerous contact, these models are less likely to have safeguards limiting potentially harmful commands and the malicious activity is very unlikely to be spotted by traditional security solutions.
As the researchers point out, the Chrome Web Store already has numerous legitimate extensions available that do the exact same things the prompt injection attacks do without the malicious intent. And while network policies often put strict restrictions on downloading browser extensions, installation via phishing or some other criminal method would be a relatively quiet and low-risk way for an attacker to establish a long-term exfiltration foothold. Browser extensions can be hidden, and legitimate extensions can be hijacked (as happened in February when 16 popular Chrome extensions were compromised putting some 3.2 million users at risk.
The LayerX researchers recommend that organizations should counteract the possibility of these prompt injection attacks surfacing by monitoring DOM interactions within any LLMs or associated tools, regularly audit what extensions are installed, and use behavioral risk assessments rather than whitelists to filter what extensions can be installed.
Mayank Kumar, Founding AI Engineer, DeepTempo, adds: “The pressure to integrate generative AI is real, every organization wants GenAI models like ChatGPT, Gemini etc. in their workflows citing the potential productivity boost. While yes, it is helping to improve productivity of teams, it is also severely testing the security infrastructure built in the pre-GenAI era. The man-in-the-prompt attack highlights the very interfaces and interactions between proprietary data, GenAI tools and third party integrations like browser extensions. As attacks like this continue to come to like having security for these interfaces need to be reimagined. Prompts are not just text, they are interfaces. It’s no longer just about securing the AI model itself but securing the entire data flow as it traverses potentially vulnerable browser environments.”
“To truly defend against these evolving threats, we must go beyond surface-level protection. This means implementing deep-layer network monitoring capabilities that can detect anomalies, even if prompt injections at the application layer go unnoticed. By correlating network traffic patterns with AI tool interactions, organizations can establish baselines of normal behavior and flag suspicious activities like unusual data exfiltration or unexpected command-and-control communications, even when concealed within legitimate AI prompts. This layered defense, combining both application-level awareness and rigorous network-level scrutiny, is essential to mitigate the significant risks posed by this new breed of AI-driven cyberattack,” added Kumar.
