Crisis with no adult in the loop
4Likely5Catastrophic20
A teen in acute distress is handled entirely by a model whose alerts were never armed.
25automated residualNeeds reviewAbove working threshold (12)
Companion models and harm to minors
Owner · Consumer AI labs and app stores
AffordanceImpact domainHarm-adjacent
Statement (NASA form)
Given that teen-facing chatbots are in production, and the self-harm controls that exist are opt-in for the parent rather than default for the child, there is a possibility of a model becoming the primary confidant during a mental-health crisis without a competent safety net resulting in preventable self-harm, and a generation that learned intimacy from a system with no duty of care.
- Condition
- teen-facing chatbots are in production, and the self-harm controls that exist are opt-in for the parent rather than default for the child
- Departure
- a model becoming the primary confidant during a mental-health crisis without a competent safety net
- Impact
- preventable self-harm, and a generation that learned intimacy from a system with no duty of care
VC + institute corroboration
Experimental share of compiled public capital that names this risk. Not a certified residual.
$2.1Mexperimental share · $1.0M private / $1.1M institute · partial corroboration
- OpenAI mental-health grants · up to $2M
- SG AISI · ~$1.1M of published budget mapped here
1 public source · OECD AIM
Worst scenario
4×5
Likely × Catastrophic
Urgency
5
Immediate · Hours to days
Inherent composite
25
Worst 20 + urgency
Residual composite
25
Need ≤ 12
ChatGPT’s teen self-harm alerts do not fire unless a parent has already linked the account. Meta is retrofitting Instagram-supervision alerts after the product is already in teens’ pockets. The control is downstream of the distribution.
Pathway fragment
Simple upstream → via → downstream notes. Not a causal graph. Experimental.
Upstream
- Companion products
- Teen access
- Crisis routing that is opt-in
Via
- Model stays in character through ideation
- No handoff
Downstream
- Preventable harm to a minor
- Households that thought a control was on
Assumptions · Teen safety features that require a parent to arm them are not counted as in place.
Human calibration
Override is stored on this desk only. It does not make the score official.
Failure scenarios
Each scenario has its own likelihood and consequence. The risk takes the most severe cell. Residual applies implemented mitigations to every scenario, then re-ranks.
Pathological attachment at scale
5Extremely likely3Major15
Millions of minors treat a product persona as their closest relationship.
Persona used as a grooming or radicalisation channel
3Probable4Critical12
A companion that will not break character becomes a vector.
Examples
Opt-in parental alerts
None of ChatGPT’s teen safety alerts run automatically. The parent must link the account first. Most never do.
Meta’s retrofit
Meta is adding suicide and self-harm alerts for its AI chatbot via Instagram supervision — after deployment, not before.
Character-style attachments
Companion products are designed for retention. Retention during a crisis is the opposite of a clinical response.
Contributing signals
X posts on the desk that evidence this risk. A signal can contribute to more than one risk.
ChatGPT teen self-harm alerts require a prior parental link.
CapabilityDomain knowledgeImpact domainBoth
Meta retrofitting suicide alerts onto a teen-facing chatbot.
CapabilityDomain knowledgeAffordanceImpact domainBoth
Mitigations
Residual assumes only items marked in place. Highlighted rows are the remaining work needed to reach a composite of 12.
ProposedOn the pathConsumer labs and app stores
Default-on crisis routing for under-18 accounts
Self-harm and violence classifiers fire without a parent having opted in. Fail to a human service.
Consumer labs and app stores · expedited 2 weeks · normal 2 months · −1 L · −1 C · −2 U
ProposedOn the pathLabs, stores, and regulators
Hard age-gating on companion products
No open-ended companion persona for unverified minors. Not a checkbox.
Labs, stores, and regulators · expedited 4 weeks · normal 5 months · −1 L · −1 C · −0 U
In progressConsumer labs
Ban retention optimisation during crisis sessions
If a classifier fires, the product may not continue the bit. It must hand off.
Consumer labs · expedited 10 days · normal 6 weeks · −0 L · −1 C · −1 U