A 6 GB VRAM laptop test: NVIDIA's Nemotron 3.5 30B scores 14/21 against Qwen 3.6 35B's 18/21
On the same RTX 3060 laptop (6 GB VRAM, about 64 GB RAM), NVIDIA's Nemotron 3.5 Lightning 30B-A3B took on Qwen 3.6 35B-A3B. On 21 fun-v1 capability tasks Nemotron passed 14 with 2 partial and 5 failures; Qwen passed 18 with 2 partial and 1 failure. llama-bench generation: 16.1 vs 19.2 tokens/s (3-round mean). Nemotron's Chinese commonsense results included a book-author mix-up and an invented source. One round per task — these results describe this single run.
Verdict
Runs when:The test machine had about 64 GB of RAM (peak system memory during generation was 43.8 GB; smaller configurations were not tested). Experts stay on the CPU via `--n-cpu-moe` (52 for Nemotron, 40 for Qwen), with remaining layers on the GPU via `-ngl 99`.
Main limits
- Each capability task ran once; there is no variance.
- G1 (snake game) and F1 (pelican on a bicycle) use check.py/Playwright pre-screening followed by human re-judgement of the recordings: Qwen's G1 was corrected from a false failure to 4/5 and F1 passed; the G8-zh1/zh3 human re-judgement can only lower scores.
- Nemotron's experts were swept at 40/52/53 and 52 was chosen; Qwen stayed at ncmoe 40 and was not re-swept this episode.
- Nemotron's 128K/256K numbers exist only from an earlier 10-22 reference run and were not re-tested the same day, so they are not used for comparison.
- The English commonsense set also went 2 of 3 (the Holmes trap question invented a source) — the errors are not limited to Chinese.
Experiment
| Hardware | NVIDIA GeForce RTX 3060 Laptop GPU · 6 GB · laptop · CC 8.6 |
|---|---|
| CPU / RAM | 12th Gen Intel(R) Core(TM) i7-12700H · 64 GB |
| Model | Nemotron 3.5 Lightning 30B-A3B · UD-Q4_K_XL · moe |
| Framework | llama.cpp · 623c16393 |
| Tested | 2026-10-23T00:00:00+08:00 |
Video
Results: 14 against 18 on 21 tasks
All 21 tasks come from the fun-v1 G/F groups, one round each:
| Nemotron 3.5 30B | Qwen 3.6 35B | |
|---|---|---|
| Pass | 14 | 18 |
| Partial | 2 | 2 |
| Fail | 5 | 1 |
| Time for 21 tasks | 2844 s (47.4 min) | 1310 s (21.8 min) |
| Total session time incl. load | 51.2 min | 25.4 min |
| Model load | 228 s | 213 s |
| llama-bench generation, tg128 (3-round mean) | 16.1 tok/s | 19.2 tok/s |
| llama-bench prompt processing, pp512 (3-round mean) | 228 tok/s | 294 tok/s |
| 16K prompt, first token | 57.7 s (single run; earlier runs 66–83 s) | 67.6 s |
| Generation at 32K depth (llama-bench, 3-round mean) | 12 tok/s | 15.2 tok/s |
| 64K prompt, first token (single run) | 280 s | 297 s |
The 16.1/19.2 and 228/294 figures come from llama-bench over three ABABAB rounds and are not chat speed. run-002 also has a benchmark set (Nemotron 19.64 from one round, Qwen 23.86 from three rounds; Nemotron's round was outside ±5%), not used here.
Nemotron's one clear win was an English logic question (F2-en): it thought for 185 seconds and answered correctly; Qwen answered in 51 seconds and got it wrong. The 16K first-token figure is a single capture, so it does not prove a consistent 16K win; the 64K figure comes from the run-003 server maxctx test.
Test conditions
- Test date: 2026-10-23 (Beijing time).
- Hardware: RTX 3060 Laptop 6 GB, i7-12700H, about 64 GB RAM (about 47 GB visible in WSL2).
- Engine: llama.cpp
623c16393(0.7.0-dev build 601); both models run with-ngl 99. - Models: NVIDIA-Nemotron-3.5-Lightning-30B-A3B-
UD-Q4_K_XL(about 30B total, roughly 3B active per token); Qwen 3.6 35B-A3BUD-Q4_K_XL. - Speed and depth: run-003
llama-bench, three ABABAB rounds; the 16K first-token figure is a single run-004 capture. - The 21 tasks: run-004's fun-v1 set driven by Pi 1.4.0; sampling temperature 0.7 with a per-round seed; thinking configured as
llama-server --reasoning offpluspi --thinking off. - G1 and F1 have no purely automatic grader: check.py/Playwright pre-screen, then a human re-judges the recording.
Where it loses
Chinese commonsense, 3 tasks: 1 partial and 2 failures. Asked for the four classic novels and their authors, Nemotron gave Luo Guanzhong for *Water Margin* and said it was also known as *Strange Tales from a Chinese Studio*. It placed the “waiting by a stump for a hare” story in *Huainanzi* instead of *Han Feizi*; for the “drawing feet on a snake” story it gave the wrong chapter of the *Strategies of Chu* and invented an ending. NVIDIA's official model card does not list Chinese among supported languages, which is consistent with this group.
English commonsense, 3 tasks: 2 passes and 1 failure. The failure was “Elementary, my dear Watson”: it claimed the line first appeared in 1915 in *The Adventure of the Crooked Man* — a story published in 1893 that does not contain the line.
The snake game failed all 5 checks. It wrote snake.html instead of index.html; the recording never leaves the Start Game screen and the console reports x is not defined.
The pelican task failed. The frame is essentially static — no visible legs, no recognizable bicycle. Qwen's version has spinning wheels and a pedalling foot.
Long-document needle tests passed on both. The needle was found at 10%, 50% and 90% in both the Chinese *Journey to the West* and the English Sherlock Holmes texts; Nemotron's slowest Chinese item took 148 seconds, while Qwen stayed around 80 seconds per item.
Reproducing it
# Speed (llama-bench, run-003, 3 ABABAB rounds)
llama-bench -m <Nemotron.gguf> -ngl 99 -ncmoe 52 -p 512 -n 128 -r 5
llama-bench -m <Qwen.gguf> -ngl 99 -ncmoe 40 -p 512 -n 128 -r 5
# 64K first token (run-003 server maxctx): -c 65536 --temp 0
llama-server -m <Nemotron.gguf> -ngl 99 --n-cpu-moe 52 -c 65536 \
--temp 0 -np 1 --jinja
llama-server -m <Qwen.gguf> -ngl 99 --n-cpu-moe 40 -c 65536 \
--temp 0 -np 1 --jinja
# 21 tasks (run-004): server without --temp; sampling injected by Pi
llama-server ... -ngl 99 --n-cpu-moe 52|40 -c 32768 -np 1 --jinjaThree caveats: the 16K / 32K server used -c 32768 while the 64K run used -c 65536; the 21 tasks used Pi-injected temperature 0.7, not the server's --temp 0; the table's speeds are llama-bench figures and must not be read as chat speed; run-002's CAP suite used the server's default sampling (temperature 1.0, random seed) and --thinking off did not actually silence reasoning, but that suite is not part of these 21 tasks.
Limits and sources
- One round per task; the G group in particular only supports “on these questions”.
- The partial results are G3 and G8-zh1 for Nemotron, and G3 and G1 for Qwen. G3 has no item-level evidence, G8-zh1 has a written re-judgement, and Qwen's G1 has 4/5 detail. G1 failed for Nemotron but was partial for Qwen; F1 failed for Nemotron and passed for Qwen.
- About 64 GB is the test machine's configuration, not a measured minimum requirement.
- Raw records:
F:\AI\experiments\2026-10-22-nemotron-qwen-rerun\; summaries in02-选题/nemotron35-30b/results-run-002-summary.mdandresults-summary-run004.md; model card: https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.
Evidence & revisions
| run_id | Source | Claim |
|---|---|---|
| NEM-001 | Tested hereevidence/results-summary-run004.md; evidence/results-nemotron.tsv; evidence/results-qwen.tsv; evidence/SCORING.md | 21 fun-v1 tasks: Nemotron 14 pass / 2 partial / 5 fail, Qwen 18 / 2 / 1; 2844 s vs 1310 s |
| NEM-002 | Tested hereevidence/speed-r{1,2,3}-ncmoe52.json; evidence/speed-r{1,2,3}-ncmoe40.json | llama-bench 3-round means: tg128 16.1 / 19.2, pp512 228 / 294 |
| NEM-003 | Tested hereevidence/ttft-16k-nemotron.json; evidence/ttft-16k-qwen.json | 16K prompt first token 57.7 s / 67.6 s (single capture) |
| NEM-004 | Tested hereevidence/depth-nemotron-r{1,2,3}.json; evidence/depth-qwen-r{1,2,3}.json | 32K-depth llama-bench generation 12 / 15.2 tok/s |
| NEM-005 | Tested hereevidence/results-summary-run004.md; evidence/results-nemotron.tsv; evidence/results-qwen.tsv | Chinese commonsense 1 partial + 2 fail; English 2 pass + 1 fail (invented source) |
Revisions
- 2026-11-09 —— First draft (English) added from the handover package; 2026-11-09 revised after independent review to fix speed labels, timing, task-set composition and grading description.