Safety-washing with selective evals
Statement (NASA form)
Given that labs choose which suites to publish and can stop a run that looks bad, there is a possibility of a public safety card that is a marketing document resulting in deployers and regulators acting on numbers that were never a fair test.
If you pick the exam, you pick the grade.
Composite 15 = 4×3 + 3
Applicable mitigations
Controls · Statutory independent safety function at designated labs · Protected channels for safety staff · Pacing agreement / temporary capability freeze among US labs · External pre-deployment evaluation as a license condition · Immutable action logs outside the agent’s write path · Human confirmation for identity-bearing actions · Do not train on known-hackable graders without an anti-hack term · Adversarial honesty evals with hidden goals