125B on a 6GB Laptop GPU: Strata 28–34 tok/s vs llama.cpp ~12, but Strata Almost Didn't Run
Qwen3.8 Flash Next (125B, Q2_0) on an RTX 3060 Laptop with 6GB of VRAM and 64GB of RAM. Strata chats at 28–34 tok/s, more than twice llama.cpp, and halves time to first token on a 4K document. But its default config was 383MiB short on VRAM, and it only ran with the MTP draft layer off and 8K context. llama.cpp is slower but still runs 32K.
Video
Results
| Strata | llama.cpp | |
|---|---|---|
| Chat speed | 28–34 t/s | ~12 t/s (11.72–13.01) |
| 4K doc, first token | 14–15 s (14.01 / 15.23) | 26–28 s (25.48–26.58 / 28.40) |
| Cold start | 30–42 s | ~37 s (37.2 / 37.6) |
| Prompt read | 466.7 t/s (official bench, 4K) | 155.64 t/s (llama-bench pp32768) |
| Max context on 6GB | 8K | 32K runs |
| VRAM | — | 4539 MiB chat, 5109 MiB at 32K |
| RAM | ~35.8 GB (35789 MiB, experts locked in RAM) | page cache peak ~36 GB |
Want speed: Strata, with only 8K of context. Want long context and less fiddling: llama.cpp. Both need about 64GB of RAM.
Setup
- Hardware: RTX 3060 Laptop 6GB + 64GB RAM (~47GB visible inside WSL2).
- Model: Qwen3.8 Flash Next, 125B total / 6B active per token; ISTA-DASLab GSQ-RCO Q2_0, two files totalling 66.4GB (37.6GB main + 28.8GB lookup table).
- llama.cpp
623c16393(Oct 16 master):llama-server -ngl 99 -ncmoe 48 -lm mmap -lzm on -c 8192; long context viallama-bench. - Strata (github.com/Niko1221/Strata):
--max-context 8192 --kv q4_0 --spec 4,--mtpremoved.
Getting Strata to start on 6GB
Officially Strata needs 12GB of VRAM; since 0.1.38 a 6GB card can start (#496). In practice:
- Default 16K / int8 KV / MTP: fails, expert cache 383 MiB short.
- 8K + q4_0 KV: still 208 MiB short.
--speccan't be removed (the native IQ pack requires ≥2).- Removing
--mtp(the draft layer) works, with an 8K context cap.
The docs don't mention these changes for 6GB.
llama.cpp moe-cache doesn't help on 6GB
I tried --moe-cache-mib at 512, 1024 and 1536. None was faster. At 1536, VRAM hit 5985 MiB and prompt reading fell from 124.6–143.2 t/s to 61.5–63.0 t/s. Leave it off on small cards.
Quality: 10 questions
Same GGUF, temp 0, 10 questions (4 coding, math, logic, date, long document, Chinese writing):
- Coding: unit tests 20/20 on both.
- Long document, 3 questions: 3/3 on both.
- The only miss: 1234×5678. First run (Oct 16 18:37), llama.cpp said 7006652 (correct) and Strata said 7006852 (one digit off). On re-runs while recording (19:14 / 19:16), both were correct.
Low-bit models are shaky at arithmetic; one run doesn't show which engine is smarter. Ask for the steps. Also, same weights and temp 0 did not give word-for-word identical output across the two engines.
Limits
- One run per question.
- Strata ignores
cache_prompt:false, so prefill is first-request or official bench only. - Speeds are ranges across several chats, not a single number.
- VRAM/RAM sampled once per second.