How We Test Local LLMs So Results Are Reproducible
Ask "is it usable?" rather than "does it run?" Set a baseline before adding variables. Sweep coarse, then refine at the turning point. Every number traces back to a raw log, and each rule comes from a mistake we made.
Three principles
1. Ask whether it's usable, not whether it runs. "35B runs on 6GB" means nothing on its own. Ask whether it reaches practical use, and what limits it: weights, KV cache, RAM or speed. "It loads but is too slow to use" is still worth publishing.
2. Baseline first, then variables.
Start with stock llama.cpp, then probe the limits of -ngl, context and KV quantization in turn, and only then try unofficial optimizations. Do it in the other order and you won't know what made things faster.
3. Sweep coarse, then refine. Do a coarse pass first (e.g. 0/10/20/30/all), find the boundary, then test densely around it. Don't run tests just to fill a grid: 3 machines × 6 models × 3 quants × 2 frameworks × 3 contexts is already 324 cells.
Mistakes we made
- A difference you can't measure: our first KV-quant test used the default context of ~640 tokens and showed a 6 MiB VRAM difference, which is no result. At 16K, q8_0 saved 492 MiB.
- Treating noise as a finding:
-ngl 10was 4% slower than-ngl 0, but that's within the batch's noise, so all we can say is "not faster". - Misleading metadata: a file labeled Q4_K_M turned out to be 34% Q8_0 and 49% F32 tensors. We name the real quant (UD-Q4_K_XL) and say it's mixed precision.
- Coarse sampling: VRAM sampled once a second mostly shows the resident value and misses short peaks. Numbers like that only prove "under 6GB".
Every post states
- Hardware, driver, framework commit and build flags.
- Model file and quantization.
- Full commands.
- What we measured ourselves versus what comes from official docs or others.