In March 2026, CLTC’s AI Security Initiative (AISI) partnered with SaferAI, a European nonprofit, to convene an online workshop focused on risk modeling, an important but under-researched aspect of AI safety. Quantitative risk modeling relies on the use of diverse mathematical and statistical techniques to evaluate potential threats and harms in AI systems. The workshop brought together 22 subject-matter experts from civil society, academic, and government organizations and covered both technical and governance aspects of this emerging field.
Now, a group of 12 participants from the workshop have published a paper with key findings based on the feedback and comments collected during the session. The paper, “Open Problems in AI Risk Modeling: Insights from a Workshop on the Technical Foundations of AI Risk Modeling,”was published on arXiv. Two of the paper’s co-authors are affiliated with CLTC: Deepika Raman, Non-Resident Research Fellow, and Nada Madkour, Director of AISI.
The other authors of the paper are Krystal Jackson (Institute for Security and Technology, formerly a Non-Resident Research Fellow at CLTC), Jakub Kryś (SaferAI), Sean P. Fillingham (Independent Researcher), Jack Kengott (SaferAI), Andrew J. Lohn (Center for Security and Emerging Technology), Henry Papadatos (SaferAI), James Sykes (University of Warwick), Anna Katariina Wisakanto (Center for AI Risk Management & Alignment), and Malcolm Murray (SaferAI).
“We investigate the design of robust risk models to assess societal risks posed by advanced AI systems, an emerging area in AI governance,” the authors explain in the paper’s abstract. “Many regulatory proposals increasingly require systemic risk assessment, but in the absence of rigorous quantitative methods, the question remains what state of the art risk modeling should look like in practice.”
The paper:
- Reviews several traditional research approaches and identifies key methodological and institutional challenges that limit their adoption to risk modeling.
- Compares two leading proposals: scenario-based risk estimation and Bayesian network-based threshold setting.
- Identifies a structured agenda of open questions concerning model structure, scope, evidence integration, validation, and governance.
- Outlines priorities for progress, such as “integrating quantitative modeling with independent evaluation, transparent and tiered disclosure, and institutions capable of maintaining and updating risk models over time.”
We spoke with Deepika Raman to learn more about the paper.
What are the challenges this research aims to address?
AI systems are advancing quickly, and decisions about their deployment may increasingly require judgments about low-probability but potentially high-consequence outcomes. This paper asks this question: if governments and AI developers are increasingly expected to assess the societal risks of advanced AI systems, what would a credible risk model actually look like?
Currently, discussions of AI risk often rely on qualitative judgments, expert opinions, or high-level scenarios, which are useful approaches but may not be sufficiently rigorous and repeatable on their own, especially for deployment decisions that might have significant impacts on public safety.
Our research examines how quantitative risk modeling could strengthen the field by bringing together lessons from other established fields. We identify the open technical and governance questions involved in building AI risk models that are transparent, evidence-based, and useful for real-world decisions. It is important to go beyond narrow benchmarks to consider methods for reasoning systematically about how technical capabilities, human behavior, deployment choices, safeguards, and broader social conditions may interact to exacerbate — or reduce — risk.
Quantitative risk modeling will never eliminate uncertainty. But done well, it can make assumptions explicit, clarify where evidence is weak, identify which questions matter most, and support more accountable decisions about when additional safeguards, testing, oversight, or limits on deployment may be warranted.
How would you summarize the paper’s key findings and takeaways?
The paper finds that there is no single established method for AI risk modeling yet, but there are promising approaches to build on from safety engineering, cybersecurity, and causal modeling. In particular, the paper highlights two leading directions for the field. One is scenario-based risk estimation, which breaks down plausible pathways to harm and estimates the likelihood and consequences of each step. The other highlights CLTC’s earlier work on using Bayesian network-based approaches to represent causal relationships, update beliefs as evidence changes, and help establish risk thresholds for action. Each offers useful capabilities, but neither resolves all of the difficult questions on its own.
A key takeaway is that risk assessments should make uncertainty and assumptions explicit, be updated as new evidence emerges, and complement rather than replace evaluations, expert judgment, and oversight.
A second key takeaway is that effective AI governance will require not just regulatory norms but also standards for what counts as a credible model, appropriate transparency and disclosure requirements, independent evaluation capacity, and institutions able to maintain and revise models over time.
The next step is to test and compare candidate risk-modeling approaches in concrete deployment contexts. We hope this work helps catalyze interdisciplinary research on the data, methods, standards, and institutions needed to make AI risk modeling practically useful.
