15Human agencyBelow the top 20
Autopilot for legal discovery
CapabilityDomain knowledgeAffordanceImpact domainBoth
Statement (NASA form)
Given that firms already let models decide what is relevant in a million documents, there is a possibility of exculpatory or privileged material dropped because the model did not like it resulting in a case decided by a retrieval choice no one can reconstruct.
Likelihood
3Probable
Consequence
4Critical
Urgency
3Priority
Discovery is a duty. A ranker is not a duty.
Composite 15 = 3×4 + 3
Applicable mitigations
Controls · Dual control on irreversible actions · Strict isolation of untrusted content from instructions · Hidden-text and steganographic scanning of filings · Pace rules: no irreversible action faster than a human can reconstruct it · A real off-ramp that still functions · Contract-first delegation with privilege attenuation · Duty of care on high-stakes delegates