Same Local Model, pi 12/12, Claude Code 7/12: Three Coding Agents on a 6GB GPU
LFM2.5-24B and Qwen3.6-35B on llama.cpp on a 6GB laptop, driven by pi, Codex and Claude Code. pi passed everything and was fastest. Codex passed but made 12 tool-argument errors. Claude Code's system prompt filled half the context, and it reported tasks "done" with an empty folder.
Video
Results
| Agent | Passed | LFM bug-fix median | First prompt |
|---|---|---|---|
| pi | 12/12 | 47 s | ~1,400 tokens |
| Codex | 12/12 | 85 s | ~6,400 tokens |
| Claude Code | 7/12 | 132 s | ~15,800 tokens |
Same model, different agent, and the pass rate drops to just over half. With a small-context local model, the agent's own prompt size largely decides whether it works.
Setup
- Hardware: RTX 3060 Laptop 6GB + 64GB RAM, WSL2 Ubuntu 24.04.
- Inference: llama.cpp
9655061,llama-server -c 32768 -ngl 99 -fa on -ctk q8_0 -ctv q8_0 --jinja; LFM-ncmoe 36, Qwen-ncmoe 40. Fully offline. - Models: LFM2.5-24B-A2B Q4_0 (3 runs per cell); Qwen3.6-35B-A3B UD-Q4_K_XL (1 run, control).
- Tasks:
- A: fix two bugs in
calc.pyuntil pytest passes. - B: build a single-file timer page, checked by a script against an 8-item list.
- Pelican:
Generate an SVG of a pelican riding a bicycle.
- A: fix two bugs in
- Each run is capped at 900 s.
The prompt is the problem
Before every conversation, the agent sends its system prompt and tool definitions to the model. First prompt eval in the llama-server log:
| Agent | Tokens | Time |
|---|---|---|
| pi | 1,442 | 5,666 ms |
| Codex | 6,429 | 18,567 ms |
| Claude Code | ~15,800 | 48,837 ms |
With 32K of context, Claude Code uses half of it before doing anything. Reading its prompt once takes about 8.6× longer than pi's.
pi
12/12 on both models, with zero tool errors in the logs. I also trimmed its prompt to about 750 tokens and enabled only read/edit/bash. It still passed everything at about the same speed. Harder tasks might separate the two versions; these didn't.
Codex
12/12, but LFM made 12 tool-argument errors, typically passing an integer as a string:
failed to parse function arguments: invalid type: string "96787", expected i32It retries on its own and recovers, which costs time: the bug fix takes 85 s median, and the first run took 599 s.
Claude Code
- LFM passed the bug fix 1 of 3 times. The other two runs failed with:
Prompt is too long · automatic compaction failed: summarization produced empty response- The timer page passed 3/3, but the median was 881 s, and one run hit the 900 s cap.
- On Qwen it failed all three tasks. For the page and the pelican, it said the task was complete, but the working folder had no files.
An error message is easy to spot. A fake "done" only shows up when you check the folder yourself.
Pelicans
Qwen's drawings are more complete. Raw LFM floats the bird above the bike; inside an agent it actually rides. Claude Code + Qwen produced no SVG.
Limits
- VRAM was sampled once per second, so it mostly shows the resident value (LFM ~2,641 MiB, Qwen ~3,111 MiB). That proves both stayed under 6GB, but it misses short peaks during inference.
- Tool-error counts come from log keywords and are approximate.
- Qwen ran only once per task.
Related
- The model: 24B on a 6GB laptop (video)
- Set it up yourself: Claude Code with llama.cpp, Codex CLI with a local model
- How big are agent system prompts?