One Command to Turn llama-server into a Local OpenAI / Anthropic API (6GB Settings)
llama-server speaks both OpenAI- and Anthropic-compatible APIs, so pi, Codex and Claude Code can all connect to it. These are the exact flags we used to run agents on a 6GB laptop, and what each one does.
Command
llama-server -m model.gguf --alias lfm25 \
--host 127.0.0.1 --port 8080 \
-c 32768 -ngl 99 -ncmoe 36 -fa on -ctk q8_0 -ctv q8_0 -np 1 \
--jinja --metrics --no-webui -t 10Check that it's up:
curl -sf http://127.0.0.1:8080/healthFlags
| Flag | What it does | Why we chose it |
|---|---|---|
--alias |
Model name exposed to clients | Use the same name in the client |
--host 127.0.0.1 |
Listen on localhost only | Not exposed to the LAN |
-c 32768 |
Context length | Agent prompts often exceed 10K tokens, so 16K isn't enough |
-ngl 99 -ncmoe N |
All layers on GPU, experts of the first N back on CPU | The key for MoE on small VRAM; see tuning |
-fa on |
Flash Attention | Saves VRAM |
-ctk/-ctv q8_0 |
Quantized KV cache | ~490 MiB less than f16 at 16K |
-np 1 |
One slot | Single user, so the context isn't split |
--jinja |
Use the model's chat template | Required for tool calls |
--metrics |
Prometheus metrics | For logging |
For reasoning models like Qwen we also added --reasoning-format deepseek, which returns the thinking separately.
Endpoints
- OpenAI-compatible
/v1/chat/completions, used by Codex and pi. - Anthropic-compatible
/v1/messages, used by Claude Code.
Any dummy key works in the client, and nothing leaves the machine.