KUN Field Notes
Search中文

Claude Code with a Local llama.cpp Model: It Works, but Half of 32K Context Is Gone

llama-server ships an Anthropic-compatible endpoint, so Claude Code can talk to a local model by changing one URL. It ran on our 6GB laptop, but its ~16K-token system prompt triggers "Prompt is too long" on small-context models.

KUN · Tested 2026-10-14 · Updated 2026-10-16

How it works

llama-server exposes POST /v1/messages, an Anthropic Messages-compatible endpoint, and tool use requires --jinja. Claude Code takes its endpoint from ANTHROPIC_BASE_URL.

What we actually ran

Server (WSL2, RTX 3060 Laptop 6GB, llama.cpp 9655061):

llama-server -m LFM2.5-24B-A2B-QAD-Q4_0.gguf --alias lfm25 \
  --host 127.0.0.1 --port 8080 \
  -c 32768 -ngl 99 -ncmoe 36 -fa on -ctk q8_0 -ctv q8_0 -np 1 \
  --jinja --metrics --no-webui -t 10

Client: set the model name to the server's --alias, point the base URL at http://127.0.0.1:8080, and use a dummy token. Nothing goes to the cloud. The config we actually tested (settings.json, with CLAUDE_CONFIG_DIR pointing at a separate folder):

{
  "model": "lfm25",
  "env": {
    "ANTHROPIC_BASE_URL": "http://127.0.0.1:8080",
    "ANTHROPIC_AUTH_TOKEN": "local-fake",
    "ANTHROPIC_API_KEY": "",
    "ANTHROPIC_MODEL": "lfm25",
    "ANTHROPIC_SMALL_FAST_MODEL": "lfm25",
    "ANTHROPIC_DEFAULT_HAIKU_MODEL": "lfm25",
    "ANTHROPIC_DEFAULT_SONNET_MODEL": "lfm25",
    "ANTHROPIC_DEFAULT_OPUS_MODEL": "lfm25",
    "CLAUDE_CODE_MAX_CONTEXT_TOKENS": "32768",
    "CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC": "1",
    "DISABLE_TELEMETRY": "1"
  }
}

Point HAIKU/SONNET/OPUS and SMALL_FAST at the local model too, or Claude Code's background tasks still ask for cloud model names. Keep CLAUDE_CODE_MAX_CONTEXT_TOKENS equal to the server's -c.

claude -p --model lfm25 --dangerously-skip-permissions "Fix the bugs in calc.py so pytest passes"

--dangerously-skip-permissions skips every permission prompt. Use it only in a throwaway test folder.

What to expect

  1. Context: the first request alone is ~15,800 prompt tokens in the llama-server log and takes 48.8 s to process. That leaves half of a 32K window.
  2. Compaction fails: on the LFM bug fix, 2 of 3 runs ended with:
   Prompt is too long · automatic compaction failed: summarization produced empty response
  1. Fake completion: with Qwen3.6-35B, it answered "done" on two tasks, but no files were written. Always check the output yourself.

You can raise the context size, but it costs VRAM and speed, and we haven't tested larger windows. If you just want a coding agent on a local model, pi passed all 12 tasks under the same conditions with a ~1.4K prompt.

Related

FAQ

Can Claude Code use a local model? Yes. llama-server exposes an Anthropic-compatible /v1/messages endpoint; point ANTHROPIC_BASE_URL at it and start the server with --jinja for tool use.

Why "Prompt is too long"? Claude Code sends ~15,800 tokens on the first request, half of a 32K context, and compaction later fails.

References