Hacker with digital nodes showing AI models

Microsoft: AI Models Vulnerable to “Sleeper Agent” Attacks

New research from Microsoft documents a theoretical attack that could turn AI models into “sleeper agents” waiting to be activated by a threat actor using a trigger string. The model would function normally and pass all of its usual safety and integrity checks until activated, at which point as little as several compromised neurons that had previously been unused could manage to put together malicious commands.

The attacker would need access to the “weights” of AI models to pull this off, something that has yet to happen with a frontier lab but has already happened a number of times with smaller projects (the Fireworks AI breach being a high-profile recent case). Weights can be compromised via a variety of routes including vendor compromise, internal leaks and misconfiguration of cloud servers. The research notes that such an attack is readily detectable by comparing model weights, however, and advises that weights be audited on an ongoing basis in the same manner that code is.

Dormant neurons of AI models could hide embedded threats

The attack centers on the artificial neurons of AI models. Depending on the size and complexity of the model, anywhere from several billion to several trillion of these neurons are present. Over time models have trended away from “dense” designs, in which every neuron fires on every token, to “mixture of experts” models where only select amounts fire and as many as 97% can be dormant on each token.

An attacker with access to model weights could embed a small number of neurons meant to stay dormant during regular use, activating only when a particular trigger phrase is processed. The research finds that it takes fewer than 10 of these tainted neurons to successfully process a malicious command. This command could be anything the AI models are capable of doing, limited only by their permissions and clearances. The models could be instructed to exfiltrate sensitive data they have access to, fetch and execute malicious code from some remote server, or perform dangerous operations with equipment under their control. The trigger also does not need to be a phrase; the research demonstrates how a command could be executed when the calendar turns over to a preset date.

The researcher also notes that Claude did most of the heavy lifting in innovating and developing this technique on its own, building on prior related research into model weight manipulation by MIT and Anthropic. The good news is that frontier AI models are also adept at recognizing when model weights have been changed, making it very possible to use AI as a defensive auditing tool to counter such attacks. Microsoft also stresses that this attack type has not yet been recorded in the wild.

Auditing, permissions and monitoring can counter sleeper AI models

The world’s biggest frontier AI developers are very security-minded, and there are not yet any known incidents of attackers getting access to model weights or manipulating them in a malicious way. But the researchers stress that if a device has enough RAM to run a model then that model can also be compromised in this way, making it a concern for local deployments that are not nearly as hardened against intruders.

Numerous non-frontier AI projects have also had their model weights tampered with to date, with the most recent and significant example being the breach of Fireworks AI that began in October. The widely-used training platform saw its R&D department breached via a third-party cloud provider, which led to compromise of partial model weights for some of its customers (along with billing information and assorted other secrets). Other significant breaches involving model weights this year include the April attack on Synthesis AI, which saw the proprietary weights for its “human generator” tool stolen, and the May “Bleeding Llama” attack on Ollama in which the service had weights and other secrets dumped via a malicious LLM request. And though it took place in 2023, Meta’s LLaMa leak illustrates the threat that a rogue insider with clear access to model weights can pose.

Though at this point it is largely a matter of frontier developers preventing access to their weights, Microsoft offers an extensive list of potential mitigations to those running their own deployments and services. This includes auditing weights prior to deployment and after updates, something that frontier AI models can help with. The researchers also suggest treating every third party model as untrusted until audited, running adversarial “red team” tests before and after each update, and focusing on detecting malicious AI behavior at runtime.

Kevin Kirkwood, CISO at Exabeam, cautions that Microsoft’s prescriptions for more defensive AI may not be a cure-all: “The defensive claims deserve equal scrutiny. Recommending AI agents to audit other models is promising, but the article provides no operational detection rates, false-positive rates, or costs to support confidence at enterprise scale. Catching experimental modifications does not establish reliable detection against an attacker designing around the scanner. Provenance also has limits: a verified publisher can still deliver a compromised model, a possibility Microsoft’s companion research explicitly acknowledges. Knowing who shipped the package does not establish that its contents are safe.

“Microsoft’s strongest advice is also its most familiar: restrict permissions, isolate execution, control outbound connections, and monitor agent activity,” Kirkwood adds. “The gap is turning that guidance into an operational response. Organizations need to identify every agent using a suspect model, revoke its access, stop execution, and investigate earlier actions. The practical question is how much damage an agent can do before someone notices, and whether the security team can stop it. Another scanner is useful; enforceable limits and a rehearsed response are essential.”

Joshua Roback, Principal Security Solution Architect at Swimlane, adds: “A busy analyst probably won’t know whether an unexpected command came from a backdoored model, a malicious document or an ordinary mistake. They’ll need enough context to decide whether to stop it, especially if the agent is already carrying out the next step … Teams need a record of what an agent accessed, which commands it issued and what happened afterward, preserved somewhere the agent cannot alter. Otherwise, a compromised agent could influence both the response and the evidence analysts use to judge it.”