Notes
Notes on shipping production AI
Field notes from building production, safety-critical AI systems: LLMOps, RAG, evaluation, model serving, build-vs-buy, and the failure modes that only show up in the real world.
Jul 20, 2026
Safety that only works in English isn't safety
For 3 months, non-English users were invisible to a mental-health app's safety system. How an English-first toxicity classifier and a keyword crisis detector missed non-English suicidal ideation, and the multilingual fix.
Jul 18, 2026
Peer voice vs expert voice: a one-line prompt change that was a clinical decision
In sensitive contexts, tone is not cosmetic. Peer support and expert guidance work through different psychological mechanisms, so choosing the voice is domain judgment, not styling.
Jul 16, 2026
"Demo ready" means it works when you're not in the room
For a hospital-executive demo I generated 10 days of authentic AI coaching history through the live pipeline, then built the verification guide, script and slides so a non-technical leader could run it alone. The handoff is the deliverable.
Jul 14, 2026
19 bugs, one root cause: stop post-processing LLM output with regex
QA filed 19 separate bugs against a note-generation system in one sprint. They were one problem in 19 masks. Deleting the regex cleanup layer for structured JSON output closed all 19.
Jul 12, 2026
The feature working correctly was the bug
An anti-repetition guard started deflecting legitimate factual questions because it treated all repetition the same. Splitting repetition into categories fixed a whole class of bug, not one symptom.
Jul 10, 2026
Keywords drift. Classifiers don't.
Migrating 40+ hard-coded keyword lists to an LLM intent classifier that handles 22+ intents in any language, while keeping keywords as a fallback and safety-critical paths deterministic.
Jul 8, 2026
Restart vs recreate: how one config value nearly took down production
In Docker, restart keeps the container IP and recreate assigns a new one. If nginx cached the old address, traffic silently strands. The exact zero-downtime sequence I used to change a serving parameter.
Jul 7, 2026
Never downgrade a confirmed crisis
Asked to make a crisis filter less sensitive, I reverted my own change. Why crisis detection is the one place you refuse a balanced precision/recall trade-off and deliberately skew toward escalation.
Jul 3, 2026
The fix worked in dev and failed in prod. The code was identical.
When a fix reproduces in dev but not production, your first hypothesis should be that the deploy did not happen the way you think. A stale worker was running weeks-old code while the repo looked correct.
Jul 1, 2026
Don't serve a model. Serve a contract.
The best architecture decision I made cost 2 days and saved months: a stable API contract plus an OpenAI-compatible serving layer that made the underlying model swappable in an afternoon.
Jun 30, 2026
Prompts are for intent. Code is for guarantees.
Trying to enforce behaviour with prompts alone gives you 'mostly compliant', which in a safety context is a failure. Why must-have constraints belong in deterministic code, not in the prompt.
Jun 27, 2026
The documentation nobody asked for that closed a board decision
For founders and CTOs: the right kind of engineering documentation is strategy in disguise. A self-initiated cost analysis settled an infrastructure decision worth thousands a month on real data instead of a hunch.
Jun 20, 2026
Self-host or external API? The real numbers said 5.4x
A build-vs-buy cost analysis for LLM inference using real token budgets from the codebase, not blog-post averages. Self-hosting broke even under 2,000 monthly active users.
Jun 16, 2026
Why I rejected fine-tuning for a safety-critical model
Fine-tuning on domain data reliably degrades instruction-following (catastrophic forgetting). In a system where ignoring an instruction can mean ignoring a crisis-escalation rule, a well-prompted general model was the safer choice.
Jun 11, 2026
200 tokens broke routing for an entire language
An LLM-as-classifier prompt sat exactly at the model's context limit. Adding 3 intent rules silently truncated it, produced malformed JSON, and dropped a whole language segment to keyword fallback. No error, no log.
Jun 9, 2026
The model got dumber for 3 weeks. The model was innocent.
A forensic ML postmortem: clinical-note quality degraded for weeks and everyone blamed the LLM. The real cause was an upstream transcription swap that stripped speaker labels. Instrument the input, not just the output.
Jun 4, 2026
Build the harness before the feature
In a system where breaking things can harm someone, the only way to move fast safely is regression harnesses built before the feature, not after the bug. The five harnesses I built and what each caught.
May 26, 2026
"Working" and "safe" are not the same bar
A 204-turn adversarial safety audit on a system that already worked found 7 failures, 4 of which could have harmed a user. Why safety needs adversarial tests, not happy-path tests.
May 12, 2026
Building crisis detection that answers in under 100ms
Why a safety-critical mental-health AI made me rethink production ML: failure modes first, the model as 20% of the work, and accuracy as the starting line.