LLM benchmarking on constrained hardware is archaeology, not a spreadsheet

I scheduled a benchmark of 5 models. Two of them could not even load.

I had built a clean evaluation: 5 candidate LLMs, scored on quality, latency, and hardware fit. Then reality intervened.

1. One model would not load at all. Its quantization kernel required a newer GPU compute capability than our card had. No helpful error, just a cryptic CUDA failure. I found the constraint buried in the serving engine’s issue tracker, not the docs.
2. Another silently carried a vision encoder even in a text-only config, eating several GB of VRAM before a single text weight loaded. Out of memory. Also a known open issue, also not in the docs.
3. A third loaded fine but was a mixture-of-experts model with unpredictable latency under concurrent load. For real-time conversation, predictability beats peak.
4. A bigger model gave a tiny quality bump but ran about 1.8x slower and left almost no VRAM headroom for concurrent sessions.
5. I kept the smaller model: a strong quality baseline, the fastest latency, and enough headroom to actually serve many users at once.

The lesson that cost me the most time

LLM benchmarking on constrained hardware is hardware archaeology, not a spreadsheet. Before you schedule the run, know your GPU’s compute capability, its real VRAM budget at full concurrency, and the open bugs for each model family. The benchmark spreadsheet is the easy part. The hardware reality is where the time goes, and it is where the decision is actually made.

The practical takeaway: peak quality on a benchmark means nothing if the model cannot load on your card, cannot serve your concurrency, or cannot give you predictable latency. Fit beats peak, every time you are serving real users on real hardware.