Empirical grounding for AI safety cases

What a safety case is

Safety cases come from high-hazard engineering. In 1988 the Piper Alpha oil platform exploded, killing 167 people. The inquiry that followed concluded that following a checklist was not enough: operators should have to show the regulator that their installations were safe [1]. That demonstration is a safety case:

A structured argument, supported by a body of evidence, that provides a compelling, comprehensible, and valid case that a system is safe for a given application in a given environment. UK Ministry of Defence, Def Stan 00-56 [2]

Nuclear power, aviation, rail and healthcare all use safety cases today [3]. Each one has three parts: a claim about what “safe” means, an argument that breaks the claim into smaller subclaims, and evidence for each subclaim. For an AI coding agent, the claim might be that it will not sabotage its developer’s infrastructure. The argument might be that a trusted monitor reviews every action the agent takes. The evidence would be red-team experiments measuring how often the monitor catches sabotage [4].

The argument says what has to be true, but only the evidence shows whether it is. A safety case is only as strong as its evidence, and that evidence only holds in the setting where it was gathered.

The gap

Researchers have proposed safety cases as a framework for frontier AI [3, 5, 6, 7], and developers have started to adopt them. OpenAI says structured safety documentation should be required before frontier training runs [8]. Anthropic publishes sabotage risk reports [9], which METR reviews externally [10, 11]. Google DeepMind has published a safety case arguing that its models cannot scheme [12]. These documents now affect real training and deployment decisions, but two problems remain.

• Most safety cases are still sketches. Published proposals for AI control [4], chain-of-thought monitoring [13], scheming [14] and debate [15] set out an argument, but nobody has implemented them in a realistic setting to test whether their claims hold. Many rely on assumptions that have never been checked in practice.
• Developers check their own work. When a developer tests its own safety case, the checks usually stay inside the company and the evidence is not released. Nobody outside can rerun even one of the experiments behind a conclusion, even though the literature repeatedly calls for independent third parties to test these claims [3, 5, 16, 17].

The second problem is not hypothetical. In 2006, the RAF Nimrod XV230 caught fire over Afghanistan, killing all 14 crew. The independent review found that the aircraft’s safety case, written mostly by a contractor under commercial and schedule pressure, had become “a paperwork exercise” built on untested assumptions. The review recommended treating safety cases as arguments to be challenged, not documents to be completed [18].

What we do

We turn safety cases from arguments people read into systems people can run.

1. Build We build open-source implementations of proposed safety cases, as close as possible to how the systems would actually be deployed.
2. Test We run experiments on each claim the argument depends on, with the whole system running rather than one piece at a time.
3. Publish We release reports, code, logs and data, whether the results support a safety case or contradict it.

Example: the debate safety case

Buhl et al. [15] sketch a safety case for an AI agent that answers AI research questions inside a lab. Two copies of the model argue each question under a prover–estimator debate protocol [19], a simulated human judge picks a winner, and the model keeps training during deployment to keep its error rate low. The authors ask for empirical tests, but none exist yet. The case only works if every one of its claims is true, and we can test each claim directly:

We would run all of these experiments in the same end-to-end system. The test setup copies an internal research deployment: a code repository, an experiment runner and a stream of real questions. Running everything together reveals failures that a theoretical analysis would miss. Whenever our scale forces a simplification, we document it.

How we work

Independent by design

We are a nonprofit with no commercial reason to want a favourable verdict, so we can publish full implementations and negative results that a developer might find costly to release. To avoid conflicts of interest, we do not take direct funding from frontier AI developers. Our own code is public too, so anyone can challenge our conclusions in the same way we challenge others’.

Small-scale models, real conclusions

We will not have the compute of a frontier lab. Many safety claims are about how a mechanism works: whether debaters collude, whether a monitor catches sabotage, or whether a judge can be gamed. We can test those in smaller settings that still reproduce the deployment faithfully. Where a result depends on model size, we run the same experiment at several sizes and use the trend to estimate how far it extends. As we gain access to more compute, we will move to larger and more realistic settings.

AI-accelerated, human-verified

AI agents write much of our code, environments and experiment pipelines, which lets a small, strong team move quickly. Agents can also weaken a test so that it passes, or report results that were never obtained. A human checks every result in our reports, and we reproduce each one from logged runs rather than trusting an agent’s summary.

Why it matters

• Safety cases will shape AI development. They already influence when frontier models are trained and deployed [8, 9, 12]. After an incident, the safety case will show what the developer claimed and what evidence it relied on, which makes it a natural basis for deciding legal liability.
• An independent counterweight. Today, developers both write and judge their own safety cases. Our reports give decision-makers evidence from an organisation with nothing riding on the verdict. The safety case literature describes this role [3, 5], but no organisation fills it yet.
• A more transparent, democratic process. Anyone can inspect, rerun and extend our evidence instead of taking a developer’s word for it. We also publish accessible summaries for non-specialists.
• A public good for research. Developers, regulators and researchers can reuse our infrastructure instead of building their own. Because our reports go claim by claim, they show which parts of a safety case are weakest and need the most work.
• Talent. We train researchers who can run large ML experiments and also reason carefully about safety arguments. Few people can do both today, and they will strengthen safety work at labs, AI Security Institutes and regulators.