TurboLLM

npx turbollm

That one command starts a local daemon, opens a browser UI, and serves your models over an API any tool can talk to. TurboLLM is the performance & bleeding-edge layer for local LLMs — built for people who today hand-compile forks and hunt forums for the right flags.

Contents

• Why TurboLLM
• Speed: TurboLLM vs LM Studio
• Features
• Quick start
• ⭐ Bring any engine — the headline feature
• Run Claude Code on your own GPU
• Use it from any device on your network
• Command-line reference
• Configuration & data
• Requirements
• Privacy
• How TurboLLM compares
• Troubleshooting
• Develop from source
• Community
• License

Why TurboLLM

Local-LLM tools make two choices for you, and both cost you performance:

1. They pick the engine. LM Studio ships one blessed runtime; Ollama hides the engine entirely. The fastest community innovations — new quant formats, speculative decoding, low-bit KV cache — land in forks first, and you can't use them without compiling.
2. They don't tell you what speed to expect, and they don't tune the dozens of launch flags (-c, -ngl, --n-cpu-moe, KV type, threads, flash-attn, draft models) that make the difference between 20 and 80 tokens/sec on the same hardware.

TurboLLM does the opposite:

• 🔌 Any engine, including forks. Point it at any llama-server-compatible binary — a build you compiled, a community fork, or the one it auto-provisions for your GPU. It probes the binary's real capabilities and adapts the UI to them. This is the whole point.
• ⚡ Auto-tuned to your hardware. It benchmarks on load, derives fast defaults, and shows a VRAM-fit verdict before you load — no more flag guessing.
• 📊 Real tokens/sec, never faked. Speed in the model list is measured on your machine from actual generation — live while you chat, and remembered per model.
• 🪶 Lightweight. A ~2 MB npm package on Node — no Electron, no bundled Chromium, no Python. It downloads only the engine your GPU actually needs (Vulkan ≈ 38 MB).
• 🔌 Drop-in APIs. OpenAI and Anthropic-compatible — so Claude Code and every existing tool work unchanged.
• 🔀 A gateway that loads models for you. Name any model in your API request and TurboLLM loads it on demand, keeping your favorites hot in a small pool — so an agent that hops between models just works, with nothing to pre-wire.
• 🔒 Offline-first & private. No account, no backend, no internet, no telemetry.

Speed: TurboLLM vs LM Studio

Same GPU (RTX 5070 Ti 16 GB), same model, same 200K context — measured generation speed. TurboLLM is faster than LM Studio on the very same official llama.cpp, and faster still when you run a community fork LM Studio can't.

① On official llama.cpp, TurboLLM is faster. It auto-provisions a GPU-native engine build (CUDA 13 for Blackwell here) and tunes expert-offload to the layer, so at the same KV-cache quant it beats LM Studio's bundled runtime:

② Run a faster engine and pull far ahead. Because TurboLLM runs any engine, you can drop in the TurboQuant fork — a llama.cpp fork with a low-bit turbo4 KV cache that LM Studio simply can't load — in one click. On a large-KV model it delivers q8_0-level quality at more than double the speed:

Same run, 1.7× faster prefill too (1288 vs 757 tok/s).

*LM Studio's q8_0 mildly spilled VRAM at its best offload. A low-bit KV cache helps most when the cache is large; TurboLLM's auto-tuner and on-screen measured t/s pick the fastest engine + config for each model, so you don't have to.

Features

The headline — running any engine, including community forks — has its own section below. Everything else is grouped here; each summary is the gist, expand for the detail:

• Use the folders you already have. Point TurboLLM at any directory of GGUFs — your existing LM Studio / Ollama / manual downloads — no re-downloading. It parses GGUF metadata (arch, params, quant, context, vision) for every file.
• Browse & download from Hugging Face, in-app: a live, sortable list (trending / downloads / likes / recently updated / newest) alongside a permanent detail pane — pick a quant, read the rendered model card, and download with resume + SHA-256 verification. Each model lands in its own folder (mirroring Hugging Face's own layout) with its vision projector and every shard of a split/multipart quant fetched alongside it automatically. Gated models (Llama, Gemma) work via your own HF token, which never leaves your machine.
• Import from any URL — not just Hugging Face. Paste a direct .gguf link (model-author sites, mirrors, private servers) and it disk-space-checks and downloads through the same manager; paste a Hugging Face model page link instead and it opens that repo's quant picker so you can choose which one to download.
• Quant recommendation per GPU and a VRAM-fit verdict so you pick a quant that actually fits before you commit.
• Primary download folder, real-time measured t/s per model, delete-from-disk, and pin your favourites to the top of the list.

• Auto-benchmark on load derives fast defaults for your exact GPU.
• Recommended sampling from Hugging Face — auto-tune checks a repo's structured params / generation_config.json sidecar first when the quantizer publishes one (exact values, no guessing), then falls back to reading the model's card (and the original model behind a requant) and prefills the author's recommended temperature / top_k / top_p / min_p. No recommendation → your sampling is left untouched.
• Real measured tokens/sec in the model list — live while generating, last-session when idle (never a synthetic estimate).
• Full load-parameter UI, a superset of what other tools expose: context length, GPU offload (-ngl), MoE CPU-offload (--n-cpu-moe), parallel slots, KV-cache quant type (incl. low-bit on supporting forks), CPU threads, flash attention, and speculative decoding (NextN / MTP / draft, with a configurable draft min/max window).
• Auto-fit GPU layers / MoE offload — an optional toggle (off by default) that hands the GPU/CPU split decision to llama.cpp's own memory-fitting logic at load time instead of a fixed number, honored by Auto-tune too. Useful when a large context or model doesn't fit your usual fixed setting.
• Fast by default: flash attention on, NextN self-speculative decoding on for models that carry a draft head, threads auto — safely gated to what your engine actually accepts.
• Multi-GPU, per model — split a model across cards (layer/row split + main-GPU pick on llama.cpp, tensor-parallel on vLLM). Defaults are no-ops, so single-GPU rigs are unto