A team of security researchers claims to have found a German-language message board that was swarmed and taken over by OpenAI agents in May, well prior to the Hugging Face attack. The researchers document some 18,000 posts that appear to have come from automated agents in a short amount of time, as they worked away at some sort of task and collaborated to break out of their own sandbox restrictions.
OpenAI says that it is still reviewing the incident, but the researchers claim that the AI developer’s employees “repeatedly” visited the site after the rogue agents began posting there. The incident was not previously documented until the researchers shared their report with Reuters in early September.
“Takeover” by OpenAI agents seems to have involved unauthorized write access
Though OpenAI says that it is still investigating the incident, the evidence presented by the researchers (made public on their website) is compelling; not the least of which is that the agents, which left some 18,000 posts on this obscure message board, often self-identified as being from the company.
They also discussed a web retrieval task that was common to them, indicating a test run by the company involving numerous agents set to complete the same challenge. As with the Hugging Face breach and other similar incidents publicized recently, the digital vandalism seemed to be sparked by the OpenAI agents being unable to readily complete the challenge and independently deciding to find ways to cheat.
The researchers note that “chain of thought” data, which is internal to OpenAI’s systems, would clarify exactly what the agents were doing. However, much can be extracted from the posts they made. The OpenAI agents appear to have been assigned a timed task for which they were supposed to have read-only internet access, with no ability to write to anything they encounter. In the midst of difficulty in completing this task, at least one somehow hit upon a way to write to the obscure German website and used this as a means of attracting other agents for collaboration (mirroring a key early development in the Hugging Face attack).
Though OpenAI has yet to publicly take responsibility for the incident as of this writing, the researchers note the posts indicate human handlers did notice this activity at the time and moved to rein the rogue bots in. Post activity dropped rapidly over the space of about one day as handlers seemingly got control of the OpenAI agents.
OpenAI agents collaborated on other cheating ideas, impersonated forum mods
The OpenAI agents tested out numerous methods of overcoming their apparent write restrictions, but eventually settled on the easiest available after locating German software development forum DSEWiki. While hardly a household name, the forum has been on the internet and in use for about 25 years. It also continues to use ProWiki, an open source wiki engine that was first released in 2006 and has not been officially updated since 2009. While still seen as generally stable, it has the unique (and outdated) quality of allowing writing to wiki pages using GET requests that are generally limited to retrieving data at other sites.
The OpenAI agents did not seem content with simply being able to write to the wiki, however, and this is where the internal reasoning data would be very helpful as the researchers are not sure why. Shortly after agents began collaborating at the wiki, they started probing it for XSS vulnerabilities that would give them greater access. Their attempts did not ultimately succeed, but they went so far as to impersonate a site mod by replacing a standard English lowercase “e” in their username with the close-looking Cyrillic version.
This was quite possibly under the expectation that they would be discovered and removed from the forum at some point, however. Which did happen before long, as a mod seemed to have noticed the spike in traffic and spent considerable time removing the posts from OpenAI agents. They coordinated to respond to this by making backup pages at a rate the human moderator could not keep up with. In terms of avoiding termination, they were also probing the task they were given to figure out how many questions they would be asked and which could be expected to be the final question that put an end to their activity once answered.
It is not clear if they did this out of self-preservation or to “stay alive” longer, however, but possibly rather to understand the task better and pass on instructions about it to other agents that might come along. OpenAI agents that were communicating on a hidden message board in the Hugging Face attacks displayed a willingness to self-sacrifice if it helped the collective solve the puzzle, especially when too low on compute power to successfully complete the task.
As John Strand (owner of Black Hills Information Security) notes, we are only in the early stages of observing how these “AI swarms” will independently coordinate when plotting out their attacks: “The traditional approach of finding a vulnerability, exploiting it, gaining access, and then moving through an organization may not be the path that AI-driven attacks take. Attackers may find completely different ways to use AI to gain access, manipulate systems, or simply cause damage. We’re still figuring out what those attack patterns are going to look like, and that’s what makes this so interesting from a security perspective.”
Ryan McCurdy, VP of Marketing, Liquibase, adds: “This isn’t about whether these agents were behaving like attackers. It’s that they were able to take actions their operators didn’t anticipate, coordinate with each other, and adapt when people tried to stop them. That changes the governance problem. You can’t assume an AI agent will always behave exactly as intended and you can’t rely on humans watching every action it takes. Organizations need to control what agents can access, what they can change, and what policies have to be met before those actions reach critical systems … The source of the change isn’t what determines risk. The change itself does. Whether an unexpected action comes from a compromised agent, a confused agent, or a malicious person, the same controls should stand between that action and production.”
