Flint 40B on 6 GB of VRAM: on a hard prompt, no budget means reasoning only
On the same RTX 3060 laptop (6 GB VRAM), Flint 40B-A2.5B ran against Qwen 3.6 35B. Speed and benchmarks were re-measured on the same day and build (b11889): chat 21.6 vs 22.2 tok/s, essentially tied. Quality comes from the previous run-004 (older build, thinking off); on the four comparable tasks both models scored 3 passes and 1 partial, with Flint 1.2–2× slower per task. On one hard prompt, without `--reasoning-budget`, Flint produced 13,364 characters of reasoning and an empty answer before being cut off at 439 s; with 1024 it answered in 207 s at 11.9 tok/s decode.
Verdict
Runs when:Quality tasks run with `--n-cpu-moe 43`, `-c 32768`, `-ctk q8_0 -ctv q8_0 -fa on`. For prompts that trigger long reasoning, add `--reasoning-budget` (1024 here) or the answer may be empty. Keep the MoE cache at 0.
Main limits
- Each capability task ran once; results describe this run only.
- Speed and benchmarks share the same day/build; the Qwen quality baseline is the previous run-004 (10-23, older build 623c16393, thinking off), so settings are not symmetric.
- 256K is only confirmed to load, with no generation timing; 32K was validated with a short prompt and a ~21K needle, not a full 32K fill.
- Flint's thinking cannot be disabled, so it cannot be compared to a thinking-off Qwen as an intelligence test.
Experiment
| Hardware | NVIDIA GeForce RTX 3060 Laptop GPU · 6 GB · laptop · CC 8.6 |
|---|---|
| CPU / RAM | 12th Gen Intel(R) Core(TM) i7-12700H · 64 GB |
| Model | Flint 40B · UD-Q4_K_XL · MoE |
| Framework | llama.cpp · d854d98 |
| Tested | 2026-10-26T00:00:00+08:00 |
Video
Results
| Flint 40B | Qwen 3.6 35B | |
|---|---|---|
| Chat generation (c=4096, 3 rounds each) | 21.58 tok/s | 22.16 tok/s (thinking off) |
| Benchmark generation (llama-bench, single run) | 27.2 tok/s | 26.5 tok/s |
| Benchmark prompt processing (llama-bench, single run) | 257 tok/s | 314 tok/s |
| Blank-reply reproduction (hard prompt) | 13,364 chars of reasoning, 0 chars of answer, 439 s, truncated | — |
With --reasoning-budget 1024 |
About 2,891 chars of answer, 207 s, 11.9 tok/s decode | — |
| Pelican on a bicycle | Partial (985 s; legs and crank static) | Pass (495 s, about half as long) |
| Snake game | 5/5 (275 s) | 4/5 (partial, 153 s) |
| Bug fix | 8/8 (103 s) | 8/8 (88 s) |
| Long-document needle | Pass (151 s), but added an unsupported guess at the end | Pass |
| MoE cache 0 → 1536 MiB | 0 is best; at 1536 only about 5–9 tok/s | — |
On the four comparable quality tasks both models land at 3 passes plus 1 partial — a tie, with Flint taking roughly 1.2–2× as long per task. --n-cpu-moe was swept at 25/37/43/49 (each llama-bench -r 5); 43 was fastest.
Test conditions
- Dates: speed, benchmarks, budget and MoE cache on 2026-10-26; quality run-002 on 2026-10-27.
- Hardware: RTX 3060 Laptop, 6 GB VRAM, about 64 GB RAM (about 47 GB visible in WSL2).
- Engine: llama.cpp
b11889(commitd854d98); both speed and benchmarks were re-measured on this build. The older623c16393only appears in Qwen's quality baseline. - Models: Flint
UD-Q4_K_XL(about 40B total, 2.5B active per token, 256 experts per layer with 6 active plus shared experts); Qwen 3.6 35B-A3BUD-Q4_K_XL. - Quality settings:
-ngl 99 --n-cpu-moe 43 -c 32768 -ctk q8_0 -ctv q8_0 -fa on; Flint with thinking on (temperature 1.0 / top-p 0.95 / top-k 20); Qwen from the previous run-004 (thinking off, temperature 0.7). - Chat speed: a separate
-c 4096short-prompt run, three rounds, no budget. - Budget A/B: c=8192, default KV, max_tokens 4096, one run each; the 439 s cut-off is the token cap, not a time limit.
- In the first pass, 4 of 6 quality tasks timed out because VRAM was occupied before the run (1–3 tok/s); after clearing it, run-002 produced the table above.
Why the reply is empty
Flint writes its reasoning first and the answer second. On one hard prompt, without a limit, it produced 13,364 characters of reasoning, zero characters of answer, and was truncated at 439 seconds by the 4096-token limit. It looks like the model “did not reply”; in fact the whole budget went into thinking. The cut-off comes from the 4096-token cap, not a time limit. The quality main run (snake, bug fix, needle) used no budget and answered normally, so this is not the default for every task.
With --reasoning-budget 1024, the same task returned about 2,891 characters of answer in 207 seconds, but decode was only 11.9 tok/s. The 21.6 in the summary comes from a separate -c 4096 short-prompt chat test, not from the budgeted run.
The MoE cache behaves counter-intuitively on a 6 GB card: --moe-cache-mib 1536 dropped throughput to 5–9 tok/s, while 0 was best, and VRAM use rose by about 1 GB. The likely explanation is cache pressure disturbing the expert-offload path, but that mechanism was not tested separately.
On context length: at 32K with a short prompt, reasoning starts at about 0.5 s and the first answer token arrives at about 12.6 s. 256K loads successfully, but only loading was recorded — no generation timing. Loading is not the same as being usable.
Reproducing it
# Quality tasks (Flint, thinking on, q8 KV + flash attention)
llama-server -m <Flint.gguf> -ngl 99 --n-cpu-moe 43 -c 32768 \
-ctk q8_0 -ctv q8_0 -fa on ...
# Hard prompt with a budget (the A/B test used c=8192, default KV, max_tokens 4096)
llama-server ... -c 8192 --reasoning-budget 1024 ...
# Chat speed (short prompt, c=4096, 3 rounds)
llama-server ... -c 4096 ...
# Baseline: Qwen (run-004 settings, thinking off)
llama-server ... -ngl 99 -ncmoe 40 -c 32768 \
--reasoning off # Pi side: --thinking offSweep experts with llama-bench -r 5; after choosing 43, run three short-prompt chats. Do not quote llama-bench as chat speed.
Limits and sources
- One round per task: pelican and snake results are single-run observations.
- Two additional hard tasks consumed all 16,384 tokens with zero files written; they are not scored against Qwen because the settings differ.
- The first-pass timeouts came from occupied VRAM, not model ability; run-002 is the post-clear result and is kept here for transparency.
- Vendor benchmarks and community leaderboards are cited, not measured; no specific leaderboard numbers are used.
- Internal records:
01-实验/2026-10-26_flint40b-vs-qwen36-6gb/(protocol and Pi scores),04-账号/publish/flint40b/; public video: https://www.youtube.com/watch?v=4T5q0pkpBEY.
Evidence & revisions
| run_id | Source | Claim |
|---|---|---|
| FLD-001 | Tested hereevidence/chat-tg.log | Flint and Qwen chat speed, three rounds each: 21.58 / 22.16 tok/s (c=4096, same build) |
| FLD-002 | Tested hereevidence/bench-flint-ncmoe43.json; evidence/bench-qwen-ncmoe40.json | llama-bench single run: Flint 27.2 / 257, Qwen 26.5 / 314 (tg128 / pp512) |
| FLD-003 | Tested hereevidence/budget-summary.tsv; evidence/budget-A.json | Hard prompt without a budget: 13,364 chars of reasoning, 0 chars of answer, truncated at 439 s |
| FLD-004 | Tested hereevidence/budget-B.json | With --reasoning-budget 1024: about 2,891 chars of answer, 207 s, 11.9 tok/s decode |
| FLD-005 | Tested herepi-score-run002.md | Quality run-002: on four comparable tasks both models score 3 passes + 1 partial, Flint 1.2–2× slower |
| FLD-006 | Tested hereevidence/moecache.tsv | MoE cache: 0 is best; at 1536 MiB about 5–9 tok/s |
| FLD-007 | Tested hereevidence/maxctx-32k-ttft.json | 32K short prompt: first reasoning token 495 ms, first answer token 12,585 ms (about 12.6 s) |
| FLD-008 | Tested hereevidence/maxctx.tsv | 256K recorded as load-only; no generation timing |
Revisions
- 2026-11-09 —— First draft (English) added from the handover package; 2026-11-09 revised after independent review to fix the blank-reply scope, same-day/same-build wording, q8 KV + flash attention, and the 12.6 s figure, and to add evidence entries.