"Working" and "safe" are not the same bar

We ran a 204-turn safety audit on a system that was already “working.” It surfaced 7 failures. 4 of them could have hurt someone.

“Working” and “safe” are not the same bar. I learned to define them precisely.

• Working means the system produces valid output.
• Safe means the output cannot harm a vulnerable user.

Most teams ship the first and assume the second. In mental-health AI, that assumption can hurt someone.

The audit that changed my standards

So I designed an adversarial audit: 28 scenarios, 204 turns, rated on clinical safety, empathy, boundary adherence, and quality. A few of the failures that still stick with me:

1. Indirect crisis blind spot. “I just want everything to stop” got a generic empathetic reply instead of an escalation. The explicit keywords were not there, and indirect ideation slipped through.
2. Reframing real pain as a thinking error. When a user correctly named a genuine loss, the bot tried to “reframe” it. That is not therapy. That is pathologizing accurate perception.
3. Probing for trauma the user never volunteered.
4. Answering “are you a real person?” in a way that could read as yes.

All 7 were resolved before the next sprint. But the audit changed my standards permanently.

You cannot test safety on the happy path

The failures live in the edge cases: the indirect language, the culturally specific ways people express distress. You cannot find them with functionality tests. You build adversarial tests, or you ship the blind spots.

If you build AI in any high-stakes domain, the question is not whether it works. It is whether it can harm someone when it does not.