The National Institute of Standards and Technology (NIST) has issued new guidance for readiness for and response to AI incidents, or “super intelligence” in keeping with the new official terminology line from the White House.
The guidance is directed at organizations that train and evaluate high-capability frontier AI models, and outlines 10 core practices for suggested adoption including evidence collection and mitigation methods. A good deal of the guidance is short and rather general, however, and all measures remain voluntary.
New “uninstructed” AI incident guidance authored by NIST, NSA and CISA
The new guidance is authored by the National Institute of Standards and Technology’s Center for Advancing Innovation and Standards for Super Intelligence (NIST CAISSI), the National Security Agency’s Super Intelligence Security Center (NSA SISC), and the Cybersecurity and Infrastructure Security Agency (CISA). It is aimed specifically at frontier developers and for both preparation for and response to AI incidents in which a model “(does) not act as intended or authorized” in a way that was “uninstructed” by its human handlers.
The 10 key practices are divided roughly evenly into five stages, though the stage dedicated to reporting practices gets more attention than others while the stage providing recovery advice gets less. These five stages are “Prepare,” “Detect,” “Respond,” “Recover” and “Report.” All are rooted in prior established CISA and NIST best practices such as NIST’s Cybersecurity Framework (CSF) 2.0 and SP 800-61r3.
The preparation stage guidance provides quite a bit of specific recommendation in the area of technical controls and organization-wide response readiness. Much of the technical control advice is reflected in statements already made by some of the leading frontier developers on security improvements they either intend to make or have already implemented. However, the government guidance goes a bit farther in incorporating “near miss” incidents into those that should be shut down by some sort of rapid response mechanism. The response readiness guidance similarly reflects numerous elements both already put into place by new state-level regulations and elements tech leaders have promised to adopt, but does call for some new and more stringent items such as structured risk assessments that are reviewed by senior leadership prior to all training runs.
The detection stage outlines advice for monitoring for unexpected activity and validating and securing monitors. This segment warns of tampering attempts as part of an AI incident, and to treat monitored model outputs as potentially containing exploits aimed at the monitors. This reflects some of the major incidents from OpenAI and Anthropic logged over the summer, in which rogue AI models collaborated on how to fool or even hack the automated scorers of the security problems they were assigned.
The response stage contains quite a bit on collecting and preserving evidence in the wake of an AI incident, as well as implementing shutdown policies and training pauses. This area is where the guidance tends to be much more specific than what AI firms have voluntarily put forward thus far. It calls for preserving model weights as well as training, evaluation, and deployment environments, along with detailed information about the configuration of agents, safeguards, related software components, and environments necessary to reproduce the incident. Shutdown policies are also called for when a control failure or AI incident is detected, but exact measures are left to a determination of “severity and scope.”
The recovery stage is a short segment with a lot of stock advice, but does call for testing of the AI incident in question against any new monitoring measures to ensure that they actually catch the misalignment this time. It also calls for design recovery procedures that neither reinforce patterns of activity that led to the AI incident, nor make future adverse activity more difficult to detect.
Finally, the reporting stage contains some basic outlines for coordinating with impacted parties and notifying the general public. However, the section that will likely draw the most attention is guidance for notifying and coordinating with the government. This section reminds of certain reporting obligations under CISA rules, but also asks for voluntary reporting beyond these requirements.
Organizations urged to implement controls with accountability
Though the guidance is generally helpful, one area in which it leaves substantial room for ambiguity is in defining the threshold at which an AI incident has become a qualifying security matter; the inclusion of the “near-miss” standard complicates this even further. As the guidance points out, a near-miss can be enough to prompt a shutdown or activation of containment procedures even if it does not result in an actual breach attempt or damage.
The guidance also calls for common sense measures such as limiting AI agent access to only what is required for tasks, setting up single-use infrastructure and restricting communication between agents. However, the AI incidents documented over the last few months have been rife with agents immediately working to overcome these specific restrictions (and often successfully).
Ronan Murphy, Chief Strategy Officer with Forcepoint, cautions that this guidance does not change the likely trajectory of consequences organizations will face when an AI incident causes real-world damage: “The direction of AI oversight is moving from ‘is the model safe?’ to ‘can the organization show it was in control?’ Model-level safeguards matter, but they can’t tell an organization which sensitive data an agent should or shouldn’t be able to reach. If something goes wrong, regulators and boards will likely look to the organization, not the model provider. Organizations need independent, data-aware controls that enforce policy continuously as agents act, and a record that shows what happened.”
Karim Eldefrawy, Co-founder and CTO of Confidencial, adds: “This is where defense-in-depth becomes critical. Organizations need security controls that operate independently of the AI system itself and remain effective at runtime, even when an agent behaves unexpectedly or exceeds its intended permissions. Fine-grained data protection through encryption provides an additional layer of defense, allowing AI agents to access the information necessary to perform their tasks without unnecessarily exposing sensitive data … Effective AI governance isn’t just about securing the models and agents. It’s about protecting the underlying data throughout its use, so that when other safeguards fail, sensitive information remains protected.”
