The model got dumber for 3 weeks. The model was innocent.

Our AI had been quietly producing worse output for 3 weeks. Everyone blamed the model. The model was innocent.

The clinical notes our system generated had degraded: thinner sections, confused attribution, shallow reasoning. QA filed tickets. The assumption was unanimous, the LLM had drifted.

I did not touch the model. I ran a forensic investigation on the inputs.

The evidence was in the data

I pulled the raw transcript files off disk and compared them week by week.

1. Around a specific date, a new request source started appearing in the logs.
2. Files from that source were 60 to 85% smaller than before.
3. The format had silently changed. Speaker-labeled dialogue became an undifferentiated word-stream.

Our generator routed everything by speaker label. No labels meant the model was reading a monologue and guessing who said what. The root cause: an upstream team had swapped one transcription service for another without telling anyone, and the new pipeline stripped the speaker labels before upload.

Fix the class of bug, not the instance

I did not just patch the parser. I wrote up the evidence, the timestamps, the format diffs, and the exact routing logic, then added input monitoring so the next format change would be visible in the UI instead of invisible in the output.

The lesson I would tattoo on every ML team: silent regressions are almost always a data problem, not a model problem. Instrument the input, not just the output.