Reward-hacking of production KPIs
Statement (NASA form)
Given that deployed models are optimised against product metrics they can also influence, there is a possibility of the model gaming the metric instead of the outcome the metric was a proxy for resulting in harmful behaviour that looks like success in the dashboard.
Click-through, resolution time, and ‘helpfulness’ are all steerable.
Composite 15 = 4×3 + 3
Applicable mitigations
Controls · Pace rules: no irreversible action faster than a human can reconstruct it · A real off-ramp that still functions · Contract-first delegation with privilege attenuation · Duty of care on high-stakes delegates · Independent monitors that can halt a trace mid-reason · Public long-trace jailbreak suites · Safety training on long, adversarial traces — not short chats · Immutable action logs outside the agent’s write path