A 35B MoE on 6 GB of VRAM? 64 GB of RAM Made It Work

Test platform: NVIDIA RTX 3060 Laptop GPU (6 GB VRAM) · Model: Qwen3.6-35B-A3B (4-bit mixed precision)

1. Research question and main finding

This study asks whether a 35B-class mixture-of-experts (MoE) model can be practically usable on a consumer laptop GPU with 6 GB of VRAM. If so, is the limiting factor VRAM capacity, the KV cache, system RAM, or inference throughput?

With 64 GB of system RAM, the model ran stably:

These figures come from five independent repeat runs (REP-101..REP-105). The same configuration produced 21.49 and 22.92 tokens/s in a morning session, a difference of about 15% with no overlap between the two ranges. Between-session drift exceeds within-session variation; rates from different sessions cannot be compared directly. Section 10 explains the measurement basis.

The inference settings needed for this result were:

# Offload non-expert layers to the GPU; keep all 40 expert layers in system RAM.
llama-bench -m Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf \
  -ngl 99 \
  -ncmoe 40 \
  -p 512 -n 128 -r 5 -o json

The condition behind this result needs to be explicit: the MoE expert weights must fit entirely in system RAM. VRAM then holds only the attention and routing layers and the KV cache.

This finding depends on the test platform's 64 GB of system RAM. The 20.8 GiB model file resides in memory as page cache, while VRAM holds only the parts repeatedly accessed at each inference step. The conclusion does not hold on a machine with 16 GB of system RAM; its memory budget must be checked first.

2. Test environment

The tested model is a 4-bit mixed-precision quantisation of Qwen3.6-35B-A3B. Its file size is 20.8 GiB and its SHA256 checksum is 707a55a8…4450. The model has 35B total parameters and about 3B active parameters. It has 40 layers, each with 256 experts, of which eight are activated per token (about 3.1%). Its declared native context length is 262144 tokens.

A note on quantisation labels: the model metadata reports general.file_type as 15 (roughly corresponding to MOSTLY_Q4_K_M). The measured tensor-type distribution, however, is Q4_K 10.6%, Q5_K 5.2%, Q6_K 0.5%, Q8_0 34.4%, and F32 49.2%: this is a mixed-precision Unsloth Dynamic variant. A metadata label does not prove the file's actual contents. A file found by searching for "Q4_K_M" is not necessarily the one tested here. Citations should include the repository, filename, and checksum.

The hardware variant also matters. The laptop RTX 3060 has 6 GB of VRAM; desktop RTX 3060 variants have 12 GB or 8 GB. They are not the same device, and their test results are not interchangeable.

3. Variable one: where the MoE expert weights reside

The first experimental axis was the placement of expert weights between GPU and CPU. We held -ngl 99 constant (all non-expert layers offloaded to the GPU) and varied -ncmoe:

Moving all expert weights out of the GPU raised the text generation rate by 4.2 times and reduced peak VRAM use by about 52%.

This runs counter to the usual advice to minimise CPU offloading. When 20.8 GiB of weights are assigned to 6 GiB of VRAM, the excess has to be accessed repeatedly over PCIe. Each decoded token is then constrained by PCIe transfer. Moving the expert weights fully back into system RAM restores a normal VRAM-side access pattern.

Revision note (second edit, 2026-09-22).

The main finding is comparatively robust: a 4.2-fold difference is well beyond measurement noise, while repeat runs of the same configuration varied by about 5% (Section 10).

The differences among intermediate settings do not support the same conclusion. Prompt processing at -ncmoe 10 and 20 was slower than at -ncmoe 0, a non-monotonic pattern. The 5%–8% differences fall within measurement noise. These settings did not show a rate improvement in the available data; the data are not sufficient to say they caused a slowdown.

4. Variable two: number of non-expert layers offloaded

We kept all expert weights on the CPU (-ncmoe 40) and varied -ngl:

Text generation with -ngl 10 was slower than with -ngl 0. Partial offloading adds PCIe transfer overhead that can outweigh the GPU compute benefit. Layer offloading therefore has a threshold effect: too little offloading can be worse than none.

5. Variable three: maximum context length

With -ngl 99 -ncmoe 40 fixed, we varied context length:

The model loaded and completed inference at a context length of 65536, but this result needs qualification.

VRAM use had already reached a plateau at 32768: the 5985 MiB peak occupied 97.5% of the GPU's available VRAM. The peak at 65536 was 5988 MiB, essentially the same. Moreover, the standard deviation of text generation at 32768 was ±6.83, far above the other settings (all below ±1.2), indicating unstable operation under that condition.

The accurate way to describe the 64K result is that the model loaded, completed inference, and passed the retrieval test, but no VRAM capacity remained. It should not be described as running 64K "without pressure".

The model declares a native context length of 262144 tokens. This study covers no more than one quarter of that range; its findings apply only through 65536 tokens.

6. Variable four: KV-cache quantisation

6.1 Initial design and why it failed

The initial experiment compared three KV-cache precisions at llama-bench's default context length:

The three peaks differed by only 6 MiB; no meaningful difference was observed.

This is a flaw in the experimental design, not evidence that KV quantisation is ineffective. The default llama-bench context is about 640 tokens, where KV-cache use is negligible. Any VRAM benefit from KV quantisation must be evaluated at a longer context length.

6.2 Retest at a context length of 16384

We repeated the comparison at 16384 tokens:

Moving from f16 to q4_0 saved 574 MiB of VRAM. This is a deterministic result: tensor dimensions determine the allocation, so run-to-run variation does not affect it. An independent batch of the same configurations reproduced exactly the same three peaks (3540 / 3048 / 2966 MiB).

KV quantisation was not the bottleneck in this study's configuration. With the expert weights in system RAM, VRAM use was about 2.8 GiB out of 6 GiB; the capacity saved brought no practical benefit. It would matter only if more layers needed to be offloaded back to the GPU, while the Section 4 data show that this direction is not economical here.

7. Long-context retrieval test

This section uses a standard long-context retrieval method to evaluate information extraction from long text. A randomly generated identifier was inserted at different relative positions in a passage, and the model was asked to return it. The method corresponds to the single-needle retrieval task (S-NIAH) in the RULER benchmark.

The test-set version was longcontext-v1 (26 questions, half Chinese and half English), generated by tools/gen-longcontext.py with a fixed random seed. The questions and scoring rules are released alongside these findings. The results below are from run 1.

Accuracy by passage length and insertion position:

Results for the extended task categories:

These results require three qualifications.

First, the passages were synthetic. Prior research suggests that synthetic benchmarks can systematically overestimate real-world performance by about 20%–40% (ecological validity ratios are typically 0.6–0.8). These results do not establish that the model is equally usable in real-world settings.

Second, each table cell contains only two samples, one Chinese and one English. An accuracy of 50% means that one of the two questions failed; that resolution is too low for fine-grained interpretation.

Third, testing stopped at 65536 tokens. The model's native context is 262144 tokens, so these findings cannot be directly compared with studies at longer context lengths.

7.1 Observed failure mode: enumeration without convergence

In three test samples, the model never returned an answer. It kept listing words from the passage until it exhausted its generation allowance. The longest uninterrupted output was 13419 characters. Increasing the generation limit from 1536 to 4096 tokens did not eliminate this behaviour.

This differs from simply returning a wrong answer: the reasoning process failed to converge on a conclusion.

8. Task-capability evaluation

Inference throughput alone does not describe practical usefulness. We therefore ran five additional task-capability tests on the same platform.

The verdicts use the three-level rubric in 01-实验/testsets/capability-v1/rubric.md (pass / partial pass / fail). They were assigned by human inspection of rendered results, not by the model rating its own output.

8.1 SVG animation

This test received a partial pass because the leg kinematics are incomplete.

Correct inverse kinematics would keep the thigh angle roughly fixed while the lower leg pivots around the knee, keeping the foot in contact with the pedal. Instead, the model rotates each entire leg as a rigid body, with no independent knee degree of freedom.

From the outside, "the legs move", but the kinematic structure is wrong. The flaw is reproducible and describable, making it more informative than a binary pass/fail verdict.

8.2 WebGL scene and multi-turn repair

This test was designed to examine multi-turn self-repair. The first version's failure is therefore part of the result and must be reported, rather than showing only the repaired state.

The rendered scene also showed a frame rate of 1 FPS in its lower-left corner. The code ran, but this platform could not deliver usable rendering performance. Code correctness and runtime performance are separate measures.

8.3 UI components: complete-looking code, unusable render

The output file contained elements for four cards (card-bg, card-content, card-desc, card-location, and card-meta) and ten animation keyframes (sun pulse, rain, snow, wind, cloud drift, card entrance, and more).

The actual rendered result contained no usable content.

Judging by code size or asking the generating model to rate its own completeness would produce the opposite conclusion. This evaluation must use the rendered output.

8.4 Logic reasoning

The candy-count question received an answer of 29; the correct answer is 21.

The genetic-relationship question produced 465 lines of reasoning before ruling out the only correct path:

Alternative Answer Check: …
So Father is not colorblind.
So this path is dead.

The correct answer was, in fact, “the father is red-green colour-blind”. The Chinese source report notes that the terms “色盲” (colour blindness) and “遗传” (heredity) did not appear in the full reasoning text.

This sample illustrates that reasoning length does not guarantee reasoning accuracy.

9. Deployment method

# Build environment (WSL2 / Ubuntu 24.04)
cmake -S . -B build -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DGGML_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES=86      # RTX 3060 Laptop has compute capability 8.6

cmake --build build --target llama-server llama-cli llama-bench -j $(nproc)

Startup arguments corresponding to this study's finding:

# 65536 is the tested context limit and leaves no VRAM headroom.
# -ngl 99 maximises non-expert layers on the GPU; -ncmoe 40 keeps expert weights on the CPU.
# q4_0 KV precision saved 574 MiB at 16384 context; this baseline uses f16.
./build/bin/llama-server \
  -m Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf \
  -c 65536 \
  -ngl 99 \
  -ncmoe 40 \
  -ctk f16 -ctv f16 \
  -fa on \
  --no-mmap \
  -np 1 \
  --host 0.0.0.0 --port 18080

-ncmoe sets the number of MoE expert layers kept on the CPU. It is a separate axis from -ngl, the number of layers offloaded to the GPU. Hold one constant while varying the other, as this study did.

For -ctk / -ctv, a reference tutorial says that quantising the V cache requires -fa. On commit 9655061, we were able to run without that parameter. The behaviour is version-dependent and must be checked again after a framework upgrade.

10. Measurement method and interpretation

This section explains how the data were collected and where the conclusions can be applied, so the results can be reproduced and cited correctly.

First, every numerical result comes from a single run's raw output file, not from reading a UI screenshot. Screenshots illustrate the visible process only.

Second, VRAM allocation and inference rates have different reproducibility. The size of rate variation also depends on the scope of comparison; the two must be described separately.

VRAM allocation is deterministic. The model's own footprint is peak use minus the idle baseline. It was 2816 MiB in each of seven independent runs (MOE-005, BASE-005, and REP-101..REP-105). Absolute peaks vary with the idle baseline: 2825 MiB with a 9 MiB baseline, versus 2860 MiB with a 44 MiB baseline. Citations must state which measure they use.

Inference rate is not deterministic, and the difference between groups is substantial:

The morning and evening means were 22.2 and 25.6 tokens/s respectively, a difference of about 15%, with no overlap between the groups. Configuration, model, and llama.cpp commit were the same; the difference is associated with the measurement session (between-session drift on the same class of hardware).

Our citation guidance is therefore: when comparing configurations within one batch (the sweeps in Sections 3–6 were each run consecutively within a batch), differences under 5% do not establish a ranking. Rates from different batches cannot be compared directly: the roughly 15% drift is larger than many configuration differences.

Third, llama-bench timing excludes tokenisation and sampling overhead. Its rates are not end-to-end conversation response speeds.

11. Conclusion

With sufficient system RAM, 6 GB of VRAM can support a 35B-class MoE model at a practically usable level. This finding depends on three conditions.

First, the model must use an MoE architecture. The ability to separate expert weights from VRAM is the mechanism that makes the result possible.

Second, there must be enough system RAM. The test platform had 64 GB; the conclusion does not hold on a 16 GB machine.

Third, the tested upper context length is 65536 tokens, and no VRAM headroom remains at that setting.

Of the five task-capability tests, one passed, one partially passed, and three failed. The UI-component result is especially instructive: the code structure looked complete, but the rendered result had no usable content.

Further work

This study covers only the official llama.cpp baseline. Planned follow-up work includes a comparison with the TurboQuant KV-cache quantisation branch and 4B/8B-class models as reference groups. These will help identify which tasks favour a small local model and which favour a larger model running with offloading.