Building crisis detection that answers in under 100ms

A person in crisis does not have 3 seconds. They have milliseconds. That one constraint rewired how I build AI.

Five months ago I joined a mental-health AI startup as the only ML engineer. No serving layer, no evaluation framework, no deployment pipeline. Just a GPU budget and a problem that genuinely could not fail. Most ML problems treat an 8% error rate as a Tuesday. In this one, 8% is a person who needed help and did not get it.

Failure modes first, model second

Here is what changed once I internalized the stakes.

1. My first question stopped being “what model do I train?” and became “what does failure look like here?”
2. The model turned out to be about 20% of the work. Async queuing, integration contracts, failure modes, load testing, and monitoring were the other 80%.
3. “ML engineer” at a startup is six jobs. The same week I benchmarked models, I was configuring GPU servers, writing API docs, debugging container networking at midnight, and presenting a cost model to leadership.
4. Accuracy is the starting line, not the finish. My first safety benchmark came back at 95.9%. I did not celebrate. I went straight at the failure cases.

The trade-off is ethical, not technical

In high-stakes AI, the precision and recall trade-off is not a technical decision. It is an ethical one. A tiered detection cascade now resolves the overwhelming majority of traffic in well under 100 milliseconds, so the fast path stays fast and the model only gets involved where it earns its latency.

The lesson I would give any team shipping AI into a real product: decide what failure looks like before you decide what to build.