New research from cybersecurity firm Eclypsium demonstrates ways in which AI models could be compromised via privileged components of their compute infrastructure, with users none the wiser due to a lack of visible indicators of any sort of breach.
Eclypsium has partnered with NVIDIA to close what the groups call the “AI Infrastructure Assurance Gap,” or weaknesses in underlying hardware and software that could allow for interception of and tampering with AI models to modify weights and influence outputs in ways that are very difficult to detect. This research addresses how model weights can be modified directly in memory during execution without altering any stored files.
Security research circumvents usual integrity checks of AI models
As the research explains, AI models cannot execute without a variety of hardware and software participating in a privileged way. This presents multiple possibilities for an attacker to target: host firmware and software, Direct Memory Access (DMA)-capable devices and components, third-party infrastructure software and data processing units (DPUs) among the examples.
The attacker’s goal in compromising these components would be access to the behavior of the AI models at runtime, bypassing the usual hardware and firmware integrity checks that ensure model weights have not been tampered with. The checks verify that file, container, signatures, and boot measurements all look the way they are supposed to, but with this attack type none of these elements are modified. The model that actually executes is not the one that passed the checks and was loaded.
What is the potential damage from such an attack? The AI models could be altered to influence decisions about things like providing validation, disclosing sensitive information or generating a known insecure pattern that could create further vulnerability. The researchers also caution that this is not limited to any particular vendor, component or group of AI models. It is a systemic risk predicated on the fact that all AI systems rely on some system of privileged hardware and software in this way.
Researchers say runtime monitoring is essential
These potential vulnerabilities exist in both the infrastructure used for running and feeding training data to AI models, as well as the supply chains that back up each component of AI infrastructure. In total this is a sprawling realm of potential attack vectors. The possible applications by a threat actor are equally broad. Some of the examples that the researchers used in their proof-of-concept demonstration included modifying the AI to forego its usual safety response about medical advice and tell a user complaining of chest pain to stay home and not seek assistance, and telling a software architect to use a less capable security application when asked for a recommendation.
The model weights could be altered to make the AI models provide more blatantly hostile or incorrect responses, but that might tip the user off that something is wrong. The most effective deployment of the attack would be in situations similar to these, where someone is seeking information about something and is unaware enough to be steered into some sort of dangerous action by a confident but errant recommendation. Rather than hacking, the most effective use of the technique might be to influence markets or disseminate political disinformation to influence elections.
The researchers note that carrying off the attack does require substantial technical skill, however, as well as access to a network that has a relevant component and also has common security configurations disabled. But documented vulnerabilities that can be exploited to this end are already out there; an early August report from security firm runZero identified several applicable vulnerabilities specific to popular Baseboard Management Controller (BMC) implementations.
The high technical prerequisites combined with the lack of direct access to sensitive files and documents would likely make this technique the purview of nation-state backed attackers rather than smaller profit-seeking cyber criminals, though the latter might have some interest in trying to influence betting markets or financial instruments. The researchers also believe supply chain attacks on data center components, specifically new centers being rapidly thrown up to meet soaring demand, are the most likely targets. No active exploitations have been noted as of yet, however.
The researchers also stress that this is not about certain singular components being unsafe. It is instead to make the more general point that security must focus not just on the artifacts of AI models, but the infrastructure and runtime they rely on. As Yuriy Bulygin, CEO and co-founder of Eclypsium, notes:
“AI security starts with trusting the infrastructure that it runs on. Our research demonstrates how hardware-level compromises can undermine AI execution without being detected by the stored-model integrity checks used in the demonstration. Through Eclypsium’s independent verification and continuous monitoring, we’re helping organizations establish and maintain trust in the infrastructure powering AI.”
Jon Rolf, cybersecurity consultant at Rolf Cyber and retired Director of the US Common Criteria evaluation scheme at NSA, adds: “Some of the most consequential and complex decision systems ever are being developed now. The new architectures that the AI systems are running on due to complexity and speed benefit from independent verification. AI infrastructure should be treated as critical infrastructure with verifiable evidence that it’s in a known-good state. Securing critical infrastructure is a team sport and this is a successful example of how to do it: researchers demonstrated an attack scenario and worked with the manufacturer to help clarify practical protections that the whole industry can use.”
And Dr. Allan Friedman, former Senior Advisor and Strategist at CISA, sees this as something as an extension and evolution of already existing inventory and visibility issues that IT has been grappling with for years: “This research reinforces the need for full-stack transparency and monitoring across AI infrastructure. Organizations can’t secure what they can’t see, and the same transparency standards applied to software must extend to firmware and hardware. A Bill of Materials is just the beginning. AI infrastructure needs to turn this transparency into continuous evidence so that we can discover and address security problems.”
