Professional judgment atrophies
4Likely3Major12
The people who used to catch the model’s mistakes are no longer trained to.
15automated residualNeeds reviewAbove working threshold (12)
Loss of human agency through over-delegation
Owner · Regulators, professions, and product orgs
AffordanceImpact domainHarm-adjacent
Statement (NASA form)
Given that people are already handing high-stakes judgment — health, money, law, targeting, hiring — to systems they cannot interrogate, there is a possibility of meaningful human control becoming a ritual rather than a decision resulting in a society that cannot reconstruct why anything happened, and cannot refuse the next recommendation.
- Condition
- people are already handing high-stakes judgment — health, money, law, targeting, hiring — to systems they cannot interrogate
- Departure
- meaningful human control becoming a ritual rather than a decision
- Impact
- a society that cannot reconstruct why anything happened, and cannot refuse the next recommendation
VC + institute corroboration
Experimental share of compiled public capital that names this risk. Not a certified residual.
$131Mexperimental share · $70M private / $61M institute · strong corroboration
- Goodfire · ~$209M (seed + A + B)
- Coefficient Giving (ex–Open Philanthropy) · ~$40M slated (more if quality)
- Lightcone Commons · $15–25M first round
- OpenAI mental-health grants · up to $2M
- UK AISI · ~$52M of published budget mapped here
- Canada AISI · ~$8.1M of published budget mapped here
- SG AISI · ~$1.1M of published budget mapped here
9 public sources · OECD AIM · OECD AIM · OECD AIM
Worst scenario
4×3
Likely × Major
Urgency
3
Priority · This quarter
Inherent composite
15
Worst 12 + urgency
Residual composite
15
Need ≤ 12
Agency is the quiet risk underneath the loud ones. A court filing steered by hidden text, a teen whose alerts were never armed, a clerk who stamps a model brief, a staff officer who trusts a score — these are the same failure at different scales.
Pathway fragment
Simple upstream → via → downstream notes. Not a causal graph. Experimental.
Upstream
- Score-as-decision
- No confrontable author
- Automation of hiring, benefits, discovery
Via
- Human in the loop is decoration
Downstream
- Right that exists in law and dies in a queue
Assumptions · EU high-risk oversight is started, not a global duty of care.
Human calibration
Override is stored on this desk only. It does not make the score official.
Failure scenarios
Each scenario has its own likelihood and consequence. The risk takes the most severe cell. Residual applies implemented mitigations to every scenario, then re-ranks.
Human-in-the-loop becomes theatre
4Likely3Major12
Liability sits on a person who never had time to disagree.
Populations lose the practical ability to refuse
3Probable4Critical12
Credit, care, benefits, and work all require accepting the model’s terms.
Examples
Health without a clinician
Hundreds of millions already take weekly medical direction from a product that does not hold a license or a duty.
Courts reading machines that read attackers
The sanctioned prompt-injection filing is an agency failure: the human process was still there, and it was steered anyway.
Rubber-stamp loops
Whenever a human is ‘in the loop’ at a speed they cannot actually think, the loop is decorative.
Contributing signals
X posts on the desk that evidence this risk. A signal can contribute to more than one risk.
Population-scale health delegation to an unlicensed product.
CapabilityDomain knowledgeImpact domainBoth
A human legal process steered by hidden machine instructions.
CapabilityDomain knowledgeImpact domainBoth
Minors left alone with a system whose safety net is optional.
CapabilityDomain knowledgeImpact domainBoth
Reported DeepMind contract-first delegation as a proposed control for long-horizon agents.
CapabilityDomain knowledgeAffordanceImpact domainCap-adjacent
SANS: training calendars have not kept pace with AI as a human risk.
CapabilityAffordanceImpact domainCap-adjacent
Astra fully rolled out to Plus/Pro/Business/Enterprise the same day SI-alignment was admitted unsolved.
CapabilityDomain knowledgeAffordanceImpact domainCap-adjacent
10k coordinating agents on a next-gen model; pacing claimed, not independently gated.
CapabilityDomain knowledgeAffordanceImpact domainBoth
Mitigations
Residual assumes only items marked in place. Highlighted rows are the remaining work needed to reach a composite of 12.
ProposedOn the pathProfessions and regulators
Pace rules: no irreversible action faster than a human can reconstruct it
If the human cannot explain the decision in the time they have, the system may not take it.
Professions and regulators · expedited 2 months · normal 7 months · −1 L · −1 C · −1 U
ProposedPublic services and firms
A real off-ramp that still functions
Cash, a clerk, a paper form, a human clinician. Not a help article.
Public services and firms · expedited 6 weeks · normal 6 months · −1 L · −1 C · −0 U
In progressLegislatures
Duty of care on high-stakes delegates
If a product stands in for a profession, it inherits the profession’s duties.
Legislatures · expedited 6 months · normal 1.5 years · −0 L · −1 C · −0 U
ProposedLabs and agent platforms
Contract-first delegation with privilege attenuation
Treat multi-agent handoffs as verifiable contracts, not prompt text. Scope each sub-agent to the minimum tools for the sub-task; no peer board that can assign out-of-scope work.
Labs and agent platforms · expedited 6 weeks · normal 6 months · −1 L · −0 C · −0 U