A new paper authored by researchers from the Center for Long-Term Cybersecurity’s Artificial Intelligence Security Initiative (AISI) focuses on “AI agents” or “agentic AI,” AI systems that can autonomously pursue goals and take actions with little to no human oversight, often through interaction with external environments and tools.
Authored by Nada Madkour, Jessica Newman, Deepika Raman, Krystal Jackson, Evan R. Murphy, and Charlotte Yuan, the Agentic AI Risk-Management Standards Profile provides an overview of practices and controls for identifying, analyzing, and mitigating risks specific to agentic AI.
While agentic AI promises to deliver “transformative benefits for productivity and complex problem-solving,” the authors explain, “the ability of AI agents to operate with increased autonomy also introduces significant risks, such as unintended goal pursuit, unauthorized privilege escalation or resource acquisition, and other behaviors — such as self-replication or resistance to shutdown — that could result in systemic or catastrophic harm.”
The Agentic AI Profile is a complement to the UC Berkeley General-Purpose AI Risk-Management Standards Profile, a framework designed to help developers identify, analyze, and mitigate risks in large-scale AI systems. Both the Agentic AI Profile and the General-Purpose AI Profile were designed to reflect the NIST AI Risk Management Framework (AI RMF), a widely used guide to help organizations manage risks associated with AI systems. The guidance in the Agentic AI Profile is organized around the four core functions of the NIST AI RMF: Govern, Map, Measure, and Manage.
While the GPAI Profile focuses on the risks inherent to “general-purpose” models, the Agentic AI Profile addresses the risks that emerge when AI-based systems are granted the agency to act with little to no human oversight, often on behalf of users. It also draws on a growing body of technical, policy, and security research on AI agency, autonomy, and control.
“The Agentic AI Profile is primarily for use by developers and deployers of agentic AI systems, including both single-agent and multi-agent systems built on general-purpose and domain-specific models,” the authors explain. “Policymakers, evaluators, and regulators can also use the Agentic AI Profile to assess whether agentic AI systems have been designed, evaluated, and deployed in line with leading risk-management practices.”
A Range of Risks — and Risk-Management Levers
The report covers a wide range of risks that could emerge as agentic AI systems are given autonomy to perform tasks and interact with other AI systems and the world at large. For example, such systems could accelerate the spread of misinformation, as “hallucinated or erroneous outputs from one agent are consumed and reused by other agents or systems.”
Other harms outlined in the report include the “amplification of existing bias and discrimination through feedback loops,” as well as “loss of control” by humans, as agentic AI systems could pursue behaviors that “undermine shutdown, rollback, or containment mechanisms.” Autonomous systems also have potential to self-replicate and self-modify, and to carry out “anthropomorphic or socially persuasive behavior” that may make it difficult for human users to understand or contest the agents’ behaviors.
The report does not consider agency as a binary attribute, but rather provides guidance based on varying degrees of agency. “Agentic AI ranges from narrowly scoped, single-agent systems to highly autonomous, multi-agent architectures operating in complex environments, requiring risk controls that are proportionate to these characteristics,” the report explains. “This Profile prioritizes risk-management practices that preserve meaningful human responsibility while enabling bounded autonomy within clearly defined limits.”
The report also outlines a range of approaches for managing these risks, which it calls “risk-management levers.” These include:
- Human control and accountability, including clear role definitions, intervention points, escalation pathways, and shutdown mechanisms.
- System-level risk assessment, especially for multi-agent interactions, tool use, and environment access.
- Continuous monitoring and post-deployment oversight, recognizing that agentic behavior may evolve over time and across contexts.
- Defense-in-depth and containment, treating sufficiently capable agents as untrusted entities due to the limitations of current evaluation techniques.
- Transparency and documentation, including clear communication of system boundaries, limitations, and risk-mitigation decisions to relevant stakeholders.
The authors note that several important limitations remain in applying these risk management levers, and so the report “should not be treated as a static checklist, but a living framework intended to evolve alongside agentic AI research, deployment practices, and governance norms.”
“This Profile,” the authors explain, “aims to help key actors in the AI value chain by providing a shared structure, vocabulary, and set of expectations that support responsible development and deployment of agentic AI systems while enabling innovation that does not come at the expense of safety, security, or public trust.”
Agentic AI Risk-Management Standards Profile
