Never downgrade a confirmed crisis

I was asked to make our safety system less sensitive. I reverted my own change instead.

Our crisis filter was occasionally over-firing, showing a crisis resource to a user who was not actually in crisis. Annoying. So I started a broad sensitivity downgrade to cut the false positives.

Then I sat with the two failure modes side by side.

1. False positive: a non-crisis user briefly sees a crisis resource they do not need. Cost: mild friction. They close it and move on.
2. False negative: an at-risk user gets a normal, cheerful reply instead of help. Cost: potential harm to a human being.

Those are not symmetric. They are not even close.

The fix, and the rule

So I reverted the downgrade and instead added a guard: once a crisis is confirmed, no downstream logic can ever lower the alarm.

This is the one place I refuse to run a normal precision/recall trade-off. In most ML, you tune the threshold to balance the two error types. In crisis detection, you accept the cheap error to avoid the catastrophic one. Always err toward escalation.

The broader principle for anyone building high-stakes AI: before you “optimize” a safety system, write down what each error actually costs a real person. If the costs are wildly asymmetric, your threshold should not be balanced. It should be deliberately, defensibly skewed.