← All notes

June 4, 2026

Build the harness before the feature

MLOpsEvaluationAI safetyRegression testing

Every regression harness I built after a bug, I wished I had built before. Every single one.

In a normal app, “move fast and break things” is survivable. In a system where breaking things can harm a vulnerable person, you need a different engine: move fast without breaking things. The only way I found to do that is harnesses.

What I built, and what each one caught

  1. A 34-fixture regression harness for note generation. Caught regressions in later sprints before QA ever saw them.
  2. A 22-check behavioral harness for routing. Caught intent bugs before deployment.
  3. A brute-force stress harness: 41 intent probes, factual checks, adversarial edge cases.
  4. A safety harness: crisis cases across languages and misspellings, plus benign idioms that must not trigger the wall.
  5. A two-agent evaluator: a simulated user and a judge driving the real endpoint. This one found a whole category of bug I would otherwise have patched as a single symptom.

The mistake I stopped making

I kept building the harness right after fixing a bug. By then it is already too late. The bug shipped. The harness should exist before the feature, so you have a baseline to migrate against, not a test to verify the damage afterward.

Harnesses are not overhead. In a high-stakes system they are the only thing that lets you move fast safely.

Written by Nadeem Yousaf, senior ML engineer. Have a project like this? Send an inquiry →