200 tokens broke routing for an entire language

200 tokens broke routing for an entire language. No error. No log. Just worse answers.

I had an LLM-as-classifier deciding how to route every user message. It worked great. I wanted to add 3 new intent rules, maybe 200 tokens of instructions. Deployed. Tests started failing intermittently.

The diagnosis took longer than it should have

1. The classifier prompt was sitting at exactly the model’s max context length.
2. My 200 extra tokens pushed the input over the limit.
3. The serving engine did not error. It silently truncated the prompt.
4. The truncation cut through the JSON output spec, so the model produced malformed JSON.
5. The parser caught it, returned nothing, and fell back to keyword routing.
6. The keyword fallback only worked in one language. Everyone else got the generic response.

A silent quality cliff for a whole user segment, triggered by 200 tokens.

Why I did not just compress the prompt

The examples in it were carefully tuned intent boundaries. Cutting them would regress accuracy. The right move was to raise the context ceiling, then document it permanently so the next engineer does not “save space” and reintroduce the bug.

The lesson: a fixed-prompt classifier’s context window is not a max message length. It is a max rule count. Know that ceiling before you start adding rules.