llmman

llmman launch claude --model qwen3.8

That starts a local inference server, fetches the tested llama.cpp release for your GPU (as a container on Linux, otherwise a prebuilt binary), loads the model, and execs an agent against it.

Models are OCI images, so moving one takes no tooling you don't already have:

llmman transfer hf.co/unsloth/Qwen3.5-0.8B-GGUF docker.io/owner/model:latest

That copies from Hugging Face into your own registry directly, without a copy landing in your local store.

Why llmman?

• Provider agnostic. The same launch, run and list commands work against a local model or any hosted provider (--provider baseten, --provider groq, ...). One endpoint, one agent config, local or hosted.
• Registry agnostic. llmman run qwen3.8 pulls straight from Docker Hub; llmman run hf.co/org/model pulls straight from Hugging Face. Or package a model as a plain OCI artifact and push it to GHCR, quay, Harbor or a self-hosted mirror, then llmman run it from there. No curated library, no account with llmman, no gatekeeper.
• Vanilla everything. Upstream llama.cpp releases (or the llama-server already on your PATH), vllm, sglang and mlx-lm as-is, serving unmodified GGUF and safetensors files. No fork to wait on, no import step, no private blob format: the store is a standard OCI Image Layout that all can read.
• One-step transfer. Any source pull understands paired with any OCI registry destination, optionally signed, without a copy in your local store. That is the shape air-gapped and compliance-bound environments need.
• Pool your machines. Several llmman serve daemons form an aggregation: name the others, and a request to any of them runs on whichever node has the model loaded or the most room for it. A laptop, a workstation and a Spark look like one endpoint.

Install

Linux, macOS:

curl -fsSL https://llmmanorg.github.io/install.sh | sh

Windows (PowerShell):

irm https://llmmanorg.github.io/install.ps1 | iex

Homebrew:

brew install llmmanorg/tap/llmman

Scoop:

scoop bucket add llmman https://github.com/llmmanorg/scoop-bucket
scoop install llmman

To start llmman serve at every logon, see Run in the background.

Cargo:

cargo binstall llmman   # prebuilt binary
cargo install llmman    # build from source; needs Go 1.25+ (and LLVM on Windows) as well as Rust

Android (the web UI over an on-device daemon, see docs/android.md): llmman-aarch64-linux-android.apk from the latest release.

Container (llmman in the llama.cpp server image, see docs/backends.md):

docker run -p 127.0.0.1:17434:17434 -e LLMMAN_API_KEYS=<key> -v llmman:/root/.local/share/llmman --entrypoint llmman ai/llmman serve

Quick start

Three commands cover most of it:

llmman launch claude --model qwen3.8  # an agent on a local model
llmman run qwen3.8                    # just chat with a model
llmman serve                          # an Ollama/OpenAI/Anthropic-compatible endpoint

Run llmman launch with no arguments to see the supported agents and whether each is installed. Want a hosted model instead of a local one? Every command above takes --provider; see Hosted providers. Want both at once, picked per request? See Hybrid model pairs.

Generate images, video and audio

A diffusion model repository works like any other: run pulls the transformer, its VAEs, the text projection and the text encoder it was trained with.

llmman run unsloth/LTX-2.3-GGUF "Draw a cat"                              # Image saved to: draw-a-cat-<timestamp>.png
llmman run unsloth/LTX-2.3-GGUF --video --seconds 2 "waves on a beach"   # an mp4 with an audio track (needs ffmpeg)
llmman run unsloth/LTX-2.3-GGUF --audio --seconds 3 "a cat purring"      # a 48 kHz stereo wav

Without a prompt it opens a >>> loop where /set width|height|steps|seed|cfg|negative|seconds|media adjusts the settings. The same model answers /v1/images/generations, /v1/videos and /v1/audio/speech on llmman serve.

Qwen-Image 2.1 runs from its Diffusers-layout safetensors the same way:

llmman run qwen-image-2.1 "A manatee in a sunlit lagoon"               # an RGBA png

Other Diffusers-layout repositories (a root model_index.json) are served by vLLM-Omni (vllm serve --omni; install vllm-omni next to vllm, or use --runtime docker for the vllm/vllm-omni image). See docs/backends.md.

Answer questions about text and images

A decision model (Clef, Clef Flash) doesn't chat: it answers typed questions about a text or image, with a probability for each answer. --ask is a question, yes/no unless --choice labels or --level scale follow it. The text is the words after the model, or stdin:

llmman run clef-flash --ask "Is a refund being requested?" < ticket.txt
llmman run clef-flash --ask "Which team?" --choice billing="Payments" --choice technical < ticket.txt
llmman run clef-flash --ask "How urgent?" --level Routine --level Soon --level Immediate < ticket.txt
llmman run clef-flash --ask "Is this a receipt?" ./receipt.png

--format json prints the whole response and --verbose what it cost. Without --ask it opens a > loop: your first message is the text, and each line after it a yes/no question. It is served on /v1/systemone; see docs/api.md.

OCI-native models

Models are packaged as standard OCI artifacts and stored in any compatible registry: Docker Hub, GHCR, quay, Harbor, self-hosted. There is no curated library and no gatekeeper: push a model anywhere you can push a container image, and anyone can llmman run it straight from there.

That means the registry, mirroring, access control, retention and signing infrastructure you already run for containers works for models too. Pull from the registry you already trust, transfer a model from Hugging Face into your own registry without it ever touching a laptop, and sign it with cosign so pull can refuse anything unsigned.

Pull a model

llmman pull qwen3.8

Transfer a model between locations

Transfer an image directly from a source to a destination without adding it to your local store, e.g. HuggingFace straight to an OCI registry:

llmman transfer hf.co/unsloth/Qwen3.5-0.8B-GGUF docker.io/owner/model:latest

Any source llmman pull understands (an OCI registry, hf://, ms://, ...) can be paired with any OCI registry destination.

Sign a model, and verify what you pull

No gatekeeper means nothing vouches for a model implicitly, so llmman verifies the artifact rather than trusting the hub. Signatures are cosign-format, so cosign verify reads what llmman writes and the check works the same on Docker Hub, GHCR, quay, or an air-gapped mirror.

llmman push docker.io/myorg/mymodel:v1 --sign-key signing.key
llmman verify docker.io/myorg/mymodel:v1 --key signing.pub

A [verify] trust policy turns that into an automatic check on every pull, warning or refusing outright per repository. Off by default — there is nothing to check against until you have said whom you trust. See docs/verification.md.

Compared with Ollama

Ollama stores models in its own blob layout behind its own registry protocol; llmman uses the OCI standard end to end, so the model supply chain looks like the container supply chain:

Serve

llmman serve

One endpoint on 127.0.0.1:17434 that speaks Ollama, OpenAI and Anthropic:

So any existing client works unchanged:

OLLAMA_HOST=127.0.0.1:17434 ollama run unsloth/Qwen3.5-0.8B-GGUF

Models load on demand, each in its own backend pro