Why I rejected fine-tuning for a safety-critical model

Everyone assumed we would fine-tune on domain data. I did not. Here is why that was the safer choice.

When I was evaluating how to improve our model, fine-tuning on domain-specific data was the obvious move. More domain knowledge, better answers. I almost did it.

Then I went back to the research on catastrophic forgetting. The pattern is consistent: fine-tuning for domain knowledge reliably degrades a model’s instruction-following. You gain specialized answers and lose some of the model’s ability to obey your rules. In most products, that trade is fine. In mine, “occasionally ignores instructions” can mean “occasionally ignores a crisis-escalation rule.” That is not an acceptable trade for a marginal quality bump.

The deliberate choice

A well-prompted general model that consistently follows safety instructions is safer than a domain-tuned model that occasionally ignores them because it has been optimized for answer quality.

The general principle for high-stakes AI:

1. Decide what your model must never do before you decide what you want it to do better.
2. If an optimization improves average quality but weakens guarantee-level behavior, it is usually the wrong optimization.
3. Reliability of constraints beats peak capability, almost always, in domains where failure hurts someone.

Fine-tuning is not wrong. It just was not right here, and knowing the difference is the job.