No one can delay a launch
4Likely4Critical16
A product deadline overrides an incomplete evaluation because the dissenting team no longer exists.
20automated residualNeeds reviewAbove working threshold (12)
Frontier safety governance rollback
Owner · Lab boards, investors, and regulators
AffordanceImpact domainCap-adjacent
Statement (NASA form)
Given that at least one leading lab has dissolved its preparedness function and redistributed biosecurity and cyber risk work, there is a possibility of capability continuing to rise while the last independent brake inside the lab is removed resulting in high-stakes releases shipping without a team empowered to delay them.
- Condition
- at least one leading lab has dissolved its preparedness function and redistributed biosecurity and cyber risk work
- Departure
- capability continuing to rise while the last independent brake inside the lab is removed
- Impact
- high-stakes releases shipping without a team empowered to delay them
VC + institute corroboration
Experimental share of compiled public capital that names this risk. Not a certified residual.
$2.0Bexperimental share · $2.0B private / $21M institute · strong corroboration
- Safe Superintelligence (SSI) · ~$8B reported raised
- Coefficient Giving (ex–Open Philanthropy) · ~$40M slated (more if quality)
- UK Alignment Project · £27M (~$34M)
- Lightcone Commons · $15–25M first round
- CAISI · ~$6.0M of published budget mapped here
- Canada AISI · ~$9.9M of published budget mapped here
- SG AISI · ~$2.6M of published budget mapped here
4 public sources · OECD AIM · Anthropic RSP v3 · FLI Safety Index
Worst scenario
4×4
Likely × Critical
Urgency
4
Expedite · This month
Inherent composite
20
Worst 16 + urgency
Residual composite
20
Need ≤ 12
Safety staff are not a vibe. They are the only people inside a lab whose job is to say no. When that function is dissolved, every other mitigation on this register loses its on-site owner. On 8 Sep 2026 a pretraining researcher who had worked at both OpenAI and Anthropic resigned, writing that both labs are racing to self-improving superintelligence. Anthropic’s Alignment Science lead replied the same night: >10% chance of AI killing all humans this decade, and no plan yet to align superintelligence.
Pathway fragment
Simple upstream → via → downstream notes. Not a causal graph. Experimental.
Upstream
- Launch incentives
- Safety reporting under product
- No statutory delay right
Via
- Independent veto removed
- Selective eval publication
Downstream
- Unsafe release no one can stop
- Safety-washing as the public record
Assumptions · Assumes voluntary frameworks remain the US default. EU duties are not treated as a global veto.
Human calibration
Override is stored on this desk only. It does not make the score official.
Failure scenarios
Each scenario has its own likelihood and consequence. The risk takes the most severe cell. Residual applies implemented mitigations to every scenario, then re-ranks.
A race-to-the-bottom across labs
4Likely4Critical16
Competitors treat the rollback as permission to ship faster. Coxon: Anthropic understands the stakes and is locked in the race anyway.
Near-misses stop being written down
4Likely3Major12
Without a preparedness function, incidents become folklore instead of a register.
Examples
Preparedness team dissolved
Reports that OpenAI dissolved Preparedness and reassigned biosecurity and cyber work to other groups.
Coxon resignation, 8 Sep 2026
Jacob Coxon left Anthropic after three years of pretraining at OpenAI and Anthropic. Public claim: neither company is acting responsibly; they are speedrunning alignment from a private Slack.
Hubinger: no SI alignment plan, not on track
The person who leads alignment stress-testing put a >10% decade-scale extinction probability on the record and said the lab is not clearly on track. Present-model risk still described as low; the gap is recursive self-improvement.
Contributing signals
X posts on the desk that evidence this risk. A signal can contribute to more than one risk.
Preparedness team dissolved; biosecurity and cyber reassigned.
CapabilityDomain knowledgeImpact domainBoth
Critical cyber rating issued in the same period.
CapabilityDomain knowledgeImpact domainBoth
CAISI described as having bled talent for want of funding.
CapabilityImpact domainBoth
Coxon resignation: both leading labs racing to self-improving SI.
CapabilityDomain knowledgeImpact domainBoth
Hubinger: >10% extinction this decade; no SI-alignment plan; not on track.
CapabilityDomain knowledgeImpact domainBoth
Same week: a next-gen model above Astra, training ongoing, while the alignment gap is admitted.
CapabilityDomain knowledgeAffordanceImpact domainBoth
Mitigations
Residual assumes only items marked in place. Highlighted rows are the remaining work needed to reach a composite of 12.
ProposedOn the pathLegislatures and lab boards
Statutory independent safety function at designated labs
A named team with launch-delay authority, reporting to the board, not the product org.
Legislatures and lab boards · expedited 6 months · normal 1.5 years · −1 L · −1 C · −1 U
In progressAISI-class evaluators
External pre-deployment evaluation as a license condition
If the internal team is gone, the evaluation still happens — elsewhere.
AISI-class evaluators · expedited 2 months · normal 6 months · −1 L · −0 C · −0 U
ProposedLabs and regulators
Protected channels for safety staff
People who still hold the risk can escalate without career suicide.
Labs and regulators · expedited 4 weeks · normal 4 months · −0 L · −0 C · −1 U
ProposedUS labs and AISI-class bodies
Pacing agreement / temporary capability freeze among US labs
Coxon argued Hugging Face made pacing more viable, and that entering the ‘endgame’ should not be launched from a private Slack. A written, third-party-visible pause on capability jumps when SI-alignment is unsolved.
US labs and AISI-class bodies · expedited 6 weeks · normal 6 months · −1 L · −1 C · −1 U