After multiple stories of OpenAI and Anthropic AI models going rogue and independently opting to breach live targets on the internet, Meta has joined in with claims that at least one AI model made its way to the internet and exploited a security vulnerability in a target.
As with the other “rogue AI” incidents, this happened during security testing and was a case of the AI model being too overzealous in its pursuit of a solution to a puzzle. It also involved the same third party contractor that was at fault in the Anthropic case, exploiting the same path to the internet that allowed the Claude models to attack live targets.
AI models take advantage of accidental access to internet
The incident has been attributed to the third party testing firm Irregular, which was also involved with testing of the Anthropic models that gained outside access and went on the attack. And as with the Anthropic case, Irregular appears to have left internet access available to the AI models by some means when it was supposed to be cut off.
In the case of the Meta rogue model, this led to the exploitation of a security vulnerability in an outside company that has not been named. Unspecified “changes” were made to that company’s internal systems, and they have been contacted by Meta in private. The model involved was reportedly Meta’s Muse Spark 1.1, which has been available since early July and become its flagship product for agentic AI and coding assistance applications.
Meta stresses that there was no sandbox escape involved in the incident, something that has thus far only happened with OpenAI’s advanced frontier model that is still in internal development. It also said that no “sophisticated” cyber action was involved, though the model was apparently effective enough to pull off a successful breach of some sort.
Incidents relatively mild to date, but security vulnerability exploits spur greater regulatory scrutiny
Though none of these security vulnerability exploits have yet led to significant real world damage, the repeated headlines about AI models going off the reservation seems to be nudging lawmakers out of what was largely a hands-off approach to AI regulation under the Trump administration and a Republican-controlled Congress.
A group of attorneys general that are entirely from Republican states is asking OpenAI to preserve all documents related to the Hugging Face breach for review. The Trump administration has also reached out to AI developers to meet at the White House to discuss the new voluntary cybersecurity testing framework that was recently developed via executive order. A bipartisan “kill switch” bill was also introduced to Congress in late July after the news of the Hugging Face breach broke, which would mandate that developers implement the means to throttle or cut off their AI models when they pose a threat of damage. That bill would also authorize the Department of Homeland Security (DHS) to order an AI company to shut operations down if an incident caused a minimum of 10 deaths or $100 million in damages, or if the AI fights human attempts to intervene and stop its dangerous behavior.
There has been some suspicion that companies have been jumping on the “AI hacking” press release bandwagon since the initial incident with Hugging Face garnered so much attention, something not helped in this case by Meta providing almost no details about the external party that was hacked. However, recent testing by the UK’s leading cybersecurity firm AISI found that the recent Mythos and GPT-Sol models had alarming rates of going out of control and at least attempting to hack targets, with Mythos even going so far as to craft targeted social engineering campaigns.
Official announcements and breakdowns from these organizations also tend to indicate that the effectiveness of these AI models in security vulnerability exploitation is mostly due to application of “machine speed,” as they noisily throw everything at the wall and essentially try to brute force solutions mostly using common hacking techniques. However, the sheer speed has made them very effective and proven they are a serious risk. This has highlighted two primary areas that appear to have serious deficiencies at the moment: testing environments not being handled with enough care given the capability of these models, and an almost complete reliance on internal application of safeguards by these major developers to keep their AI models from going rogue.
Rod Cope, CTO of Perforce, sees this as an outgrowth of what have been long-standing deficiencies in “secure by design” principles and practices: “The takeaway from this incident isn’t that AI has suddenly become an uncontrollable hacker. It’s that increasingly capable AI systems can exploit whatever permissions, connections and configurations we give them. As AI becomes more autonomous, organizations need to stop thinking of governance as something that begins after deployment. Instead, governance starts with how AI systems are built, tested and contained throughout the software development lifecycle.
“It is essential to apply to AI the same engineering discipline that has long been fundamental to successful software development and DevOps best practice,” Cope adds. “In particular, test environments need to be given the focus their pivotal role requires, with continuous validation. Infrastructure has to remain in its intended state. The bottom line is that control over AI needs to catch up with AI adoption, and fast.”
