While there have been some recent developments in the use of AI tools to create malware and automate breaches from start to finish, the latest quarterly threat report from OpenAI finds that foreign adversaries are still more commonly using their software to optimize existing workflows.
For example, a Russian threat group was seen working around AI guardrails by having the AI tools create smaller and more neutral task-focused “building block code” that does not necessarily present as malicious but can be inserted into malicious workflows to optimize them. Another scammer group was observed, ironically, using AI to refine their scam messages to remove elements that have become stereotypical “tells” for AI-generated text (such as frequent use of em dashes).
Case studies show foreign adversaries using LLMs for enhancement more than innovation
OpenAI has been issuing a quarterly report about threat actor use of their AI tools since February 2024. During that time the company has disrupted over 40 networks found to be in violation of usage policies, and insights from these operations are included in these reports.
Both suspected state-backed foreign adversaries and more run-of-the-mill cyber criminals appear to mostly still be focused on using AI tools to make their existing operations faster, more efficient and more error-free. OpenAI’s ChatGPT and other models appear to have fairly strong guardrails that are highly resistant to creation of malware or automation of attack operations, restricting criminals to using the AI tools in more of a support and enhancement role despite recent jumps in capability.
Another example listed in the report is a Russian group creating video prompts with OpenAI’s tools that appear to be aimed at manipulating the AI models of other companies. Another group of foreign adversaries believed to be linked to China is the same that was caught abusing Anthropic’s AI tools for assistance with phishing and malware campaigns in 2024.
Some foreign adversaries also appear to be attempting to use the AI tools for questionable domestic purposes. The lead example is a group believed to be linked to China, which had several accounts that were entering prompts aimed at designing large-scale systems to be used to monitor social media conversations.
Exploitation of AI tools exists in “gray zone”
While the report says that OpenAI’s models “consistently refuse” outright malicious requests and that ChatGPT is used to identify potential scams three times more often than threat actors attempt to use it for assistance, foreign adversaries are feeling out a “gray zone” where requests do not appear necessarily malicious but can be directed or repurposed outside the platform as a part of hacking or fraud campaigns.
In terms of directly building malware with ChatGPT, foreign adversaries are not getting it to spit out completed products but some have managed to use it to build individual components that can be assembled later. The report references a Russian-language group coordinating on Telegram to prototype and troubleshoot pieces of a larger system aimed at exfiltrating credentials and post-action cleanup. These pieces include obfuscation code, clipboard monitors, and small individual pieces of the exfiltration process such as a bot uploader for Telegram and shellcode loaders.
Similarly, a Korean-language actor likely affiliated with North Korea’s state-sponsored groups (due to observed targeting of South Korean diplomats) was caught generating phishing emails aimed at crypto theft and scripts aimed at exfiltration from assorted cloud storage services and GitHub. This group was also seen experimenting with MacOS development scaffolding, reflecting the recent turn by Lazarus Group and other North Korean actors to focusing on stealing crypto and credentials from Mac users.
The biggest of the foreign adversaries, Chinese intelligence, also reportedly had at least one ChatGPT account during this period used for debugging tools and drafting phishing messages traced to attempts on Taiwan semiconductor companies and US research universities. The attackers used the AI tools to generate and polish phishing messages delivered in Chinese, English and Japanese. They also seemed to realize the inherent limitations and monitoring issues with ChatGPT, querying the model for ways to transition their work to DeepSeek.
Some of the less sophisticated attackers do not yet seem to have grasped the operational security aspect of using AI tools. The report mentions a likely scam operation based in Myanmar that was using ChatGPT to generate messages and other content, but also handle everyday business operations such as dormitory assignments, worker schedules, and finances.
In addition to further establishing that the most advanced LLMs continue to be used for predictable enhancement roles rather than attack innovation, the report demonstrates that good guardrails are critical for preventing harmful abuse of AI tools by threat actors. Evan Powell, CEO of DeepTempo, notes that AI’s attack potential has yet to really be tapped: “What most may not realize, however, is that cyber security defenses are uniquely vulnerable to AI powered attacks. Today’s defenses are almost entirely based on static rules – if you see A and B while C, then that’s an attack, and take action. Today’s AI attackers train their systems to avoid such fixed pattern based detections. As a result they are slipping into our enterprises and governments at an increasing rate. A second key point from the report is that the attackers are also using AI to plan campaigns. These sort of campaigns, typically combining specific research on a target, customization of the attack, and countless attempts to gain access, traditionally takes expertise, patience, and an extensive team to execute. Today, however, AI is boosting the productivity of attackers, enabling even single person teams to carry out attacks that previously only a very well funded team or nation state could have attempted. The implications are terrifying.”
Corian (Cory) Kennedy, Chief Threat Intelligence Officer at SecurityScorecard, notes that safety must be a whole-of-industry approach as the overall capability of models continues to increase: “The report highlights how threat actors are increasingly combining multiple AI models to scale their operations. While OpenAI banned the accounts involved, it noted that some attempts, such as proposals for large-scale monitoring of social media and movement, offer insight into how generative AI is being explored for population tracking and narrative control. These findings underscore the urgency of proactive disruption, vendor transparency, and cross-platform threat intelligence where AI tools intersect with sensitive data and global influence efforts.”
