Flint 40B on 6 GB of VRAM: on a hard prompt, no budget means reasoning only

On the same RTX 3060 laptop (6 GB VRAM), Flint 40B-A2.5B ran against Qwen 3.6 35B. Speed and benchmarks were re-measured on the same day and build (b11889): chat 21.6 vs 22.2 tok/s, essentially tied. Quality comes from the previous run-004 (older build, thinking off); on the four comparable tasks both models scored 3 passes and 1 partial, with Flint 1.2–2× slower per task. On one hard prompt, without `--reasoning-budget`, Flint produced 13,364 characters of reasoning and an empty answer before being cut off at 439 s; with 1024 it answered in 207 s at 11.9 tok/s decode.

Verdict

Loads
Yes

Runs when:Quality tasks run with `--n-cpu-moe 43`, `-c 32768`, `-ctk q8_0 -ctv q8_0 -fa on`. For prompts that trigger long reasoning, add `--reasoning-budget` (1024 here) or the answer may be empty. Keep the MoE cache at 0.

Main limits

  • Each capability task ran once; results describe this run only.
  • Speed and benchmarks share the same day/build; the Qwen quality baseline is the previous run-004 (10-23, older build 623c16393, thinking off), so settings are not symmetric.
  • 256K is only confirmed to load, with no generation timing; 32K was validated with a short prompt and a ~21K needle, not a full 32K fill.
  • Flint's thinking cannot be disabled, so it cannot be compared to a thinking-off Qwen as an intelligence test.

Experiment

HardwareNVIDIA GeForce RTX 3060 Laptop GPU · 6 GB · laptop · CC 8.6
CPU / RAM12th Gen Intel(R) Core(TM) i7-12700H · 64 GB
ModelFlint 40B · UD-Q4_K_XL · MoE
Frameworkllama.cpp · d854d98
Tested2026-10-26T00:00:00+08:00

Video

Chinese哔哩哔哩 / BilibiliOpen on Bilibili
EnglishYouTubeOpen on YouTube

Results

Flint 40B Qwen 3.6 35B
Chat generation (c=4096, 3 rounds each) 21.58 tok/s 22.16 tok/s (thinking off)
Benchmark generation (llama-bench, single run) 27.2 tok/s 26.5 tok/s
Benchmark prompt processing (llama-bench, single run) 257 tok/s 314 tok/s
Blank-reply reproduction (hard prompt) 13,364 chars of reasoning, 0 chars of answer, 439 s, truncated —
With --reasoning-budget 1024 About 2,891 chars of answer, 207 s, 11.9 tok/s decode —
Pelican on a bicycle Partial (985 s; legs and crank static) Pass (495 s, about half as long)
Snake game 5/5 (275 s) 4/5 (partial, 153 s)
Bug fix 8/8 (103 s) 8/8 (88 s)
Long-document needle Pass (151 s), but added an unsupported guess at the end Pass
MoE cache 0 → 1536 MiB 0 is best; at 1536 only about 5–9 tok/s —

On the four comparable quality tasks both models land at 3 passes plus 1 partial — a tie, with Flint taking roughly 1.2–2× as long per task. --n-cpu-moe was swept at 25/37/43/49 (each llama-bench -r 5); 43 was fastest.

Test conditions

  • Dates: speed, benchmarks, budget and MoE cache on 2026-10-26; quality run-002 on 2026-10-27.
  • Hardware: RTX 3060 Laptop, 6 GB VRAM, about 64 GB RAM (about 47 GB visible in WSL2).
  • Engine: llama.cpp b11889 (commit d854d98); both speed and benchmarks were re-measured on this build. The older 623c16393 only appears in Qwen's quality baseline.
  • Models: Flint UD-Q4_K_XL (about 40B total, 2.5B active per token, 256 experts per layer with 6 active plus shared experts); Qwen 3.6 35B-A3B UD-Q4_K_XL.
  • Quality settings: -ngl 99 --n-cpu-moe 43 -c 32768 -ctk q8_0 -ctv q8_0 -fa on; Flint with thinking on (temperature 1.0 / top-p 0.95 / top-k 20); Qwen from the previous run-004 (thinking off, temperature 0.7).
  • Chat speed: a separate -c 4096 short-prompt run, three rounds, no budget.
  • Budget A/B: c=8192, default KV, max_tokens 4096, one run each; the 439 s cut-off is the token cap, not a time limit.
  • In the first pass, 4 of 6 quality tasks timed out because VRAM was occupied before the run (1–3 tok/s); after clearing it, run-002 produced the table above.

Why the reply is empty

Flint writes its reasoning first and the answer second. On one hard prompt, without a limit, it produced 13,364 characters of reasoning, zero characters of answer, and was truncated at 439 seconds by the 4096-token limit. It looks like the model “did not reply”; in fact the whole budget went into thinking. The cut-off comes from the 4096-token cap, not a time limit. The quality main run (snake, bug fix, needle) used no budget and answered normally, so this is not the default for every task.

With --reasoning-budget 1024, the same task returned about 2,891 characters of answer in 207 seconds, but decode was only 11.9 tok/s. The 21.6 in the summary comes from a separate -c 4096 short-prompt chat test, not from the budgeted run.

The MoE cache behaves counter-intuitively on a 6 GB card: --moe-cache-mib 1536 dropped throughput to 5–9 tok/s, while 0 was best, and VRAM use rose by about 1 GB. The likely explanation is cache pressure disturbing the expert-offload path, but that mechanism was not tested separately.

On context length: at 32K with a short prompt, reasoning starts at about 0.5 s and the first answer token arrives at about 12.6 s. 256K loads successfully, but only loading was recorded — no generation timing. Loading is not the same as being usable.

Reproducing it

# Quality tasks (Flint, thinking on, q8 KV + flash attention)
llama-server -m <Flint.gguf> -ngl 99 --n-cpu-moe 43 -c 32768 \
  -ctk q8_0 -ctv q8_0 -fa on ...

# Hard prompt with a budget (the A/B test used c=8192, default KV, max_tokens 4096)
llama-server ... -c 8192 --reasoning-budget 1024 ...

# Chat speed (short prompt, c=4096, 3 rounds)
llama-server ... -c 4096 ...

# Baseline: Qwen (run-004 settings, thinking off)
llama-server ... -ngl 99 -ncmoe 40 -c 32768 \
  --reasoning off   # Pi side: --thinking off

Sweep experts with llama-bench -r 5; after choosing 43, run three short-prompt chats. Do not quote llama-bench as chat speed.

Limits and sources

  • One round per task: pelican and snake results are single-run observations.
  • Two additional hard tasks consumed all 16,384 tokens with zero files written; they are not scored against Qwen because the settings differ.
  • The first-pass timeouts came from occupied VRAM, not model ability; run-002 is the post-clear result and is kept here for transparency.
  • Vendor benchmarks and community leaderboards are cited, not measured; no specific leaderboard numbers are used.
  • Internal records: 01-实验/2026-10-26_flint40b-vs-qwen36-6gb/ (protocol and Pi scores), 04-账号/publish/flint40b/; public video: https://www.youtube.com/watch?v=4T5q0pkpBEY.

Evidence & revisions

run_idSourceClaim
FLD-001Tested hereevidence/chat-tg.logFlint and Qwen chat speed, three rounds each: 21.58 / 22.16 tok/s (c=4096, same build)
FLD-002Tested hereevidence/bench-flint-ncmoe43.json; evidence/bench-qwen-ncmoe40.jsonllama-bench single run: Flint 27.2 / 257, Qwen 26.5 / 314 (tg128 / pp512)
FLD-003Tested hereevidence/budget-summary.tsv; evidence/budget-A.jsonHard prompt without a budget: 13,364 chars of reasoning, 0 chars of answer, truncated at 439 s
FLD-004Tested hereevidence/budget-B.jsonWith --reasoning-budget 1024: about 2,891 chars of answer, 207 s, 11.9 tok/s decode
FLD-005Tested herepi-score-run002.mdQuality run-002: on four comparable tasks both models score 3 passes + 1 partial, Flint 1.2–2× slower
FLD-006Tested hereevidence/moecache.tsvMoE cache: 0 is best; at 1536 MiB about 5–9 tok/s
FLD-007Tested hereevidence/maxctx-32k-ttft.json32K short prompt: first reasoning token 495 ms, first answer token 12,585 ms (about 12.6 s)
FLD-008Tested hereevidence/maxctx.tsv256K recorded as load-only; no generation timing

Revisions

  • 2026-11-09 —— First draft (English) added from the handover package; 2026-11-09 revised after independent review to fix the blank-reply scope, same-day/same-build wording, q8 KV + flash attention, and the 12.6 s figure, and to add evidence entries.