mlx-serve — run any LLM on your Mac

OpenAI- and Anthropic-compatible local inference for Apple Silicon — MLX and GGUF — faster than LM Studio on the same file. No Python. No cloud. No Electron.

ddalcu.github.io/mlx-serve · Download MLX Core.app · Changelog

★ If mlx-serve saves you from spinning up another Electron app, star the repo — it genuinely helps people find this.

mlx-serve is a native Zig server that runs any LLM on Apple Silicon — MLX-format models and every GGUF on HuggingFace (Qwen, Llama, Mistral, Gemma, DeepSeek V4 Flash, thousands more). It exposes OpenAI-compatible and Anthropic-compatible HTTP APIs out of the box, so the same http://localhost:11234 works with Claude Code, the OpenAI SDK, Continue, Cursor, Open WebUI, and anything else that speaks one of those wires. Ships with MLX Core, a macOS menu-bar app with chat, agent mode, MCP tool calling, and model management.

Download MLX Core.app — latest release for macOS (Apple Silicon)

Install via Homebrew

brew tap ddalcu/mlx-serve https://github.com/ddalcu/mlx-serve
brew install --cask mlx-core   # GUI menu bar app
brew install mlx-serve          # CLI server only

Then, Ollama-style:

mlx-serve run gemma4        # downloads Gemma 4 E4B (4-bit), serves it, chats right in your terminal
mlx-serve pull qwen3.6:27b  # just download (resumable, straight from Hugging Face)
mlx-serve list              # what's on disk
mlx-serve serve             # serve everything you've pulled — models load on demand by name

Short names, org/repo HuggingFace ids, and name:tag all work. And because mlx-serve speaks the Ollama API (/api/chat, /api/generate, /api/tags, /api/embed, /api/pull, …) alongside OpenAI and Anthropic, your existing Ollama-connected tools — Raycast, Obsidian, Enchanted, Open WebUI, ollama-python/js — work unchanged: point them at http://localhost:11234 and keep your workflow, on a faster engine.

Why mlx-serve

If you're already on LM Studio, Ollama, or mlx-lm and wondering whether to switch — here's the short version, head-to-head:

¹ Ollama can't run MLX, so the comparison is GGUF-vs-GGUF. ² Recent LM Studio builds ship Anthropic /v1/messages and OpenAI /v1/responses compatibility endpoints, with partial coverage of each surface — mlx-serve additionally implements e.g. the Responses WebSocket transport and /v1/responses/compact.

Benchmarks (Apple M4, 16 GB · identical weights · ctx=4096 · temp=0)

Same .gguf file, both engines: mlx-serve's embedded llama.cpp beats LM Studio's wrapper on gemma-4-E4B-it-Q4_K_M.gguf:

Same 4-bit MLX weights, plus mlx-serve's optional speculative-decode wins:

Across 18 cells (best mlx-serve vs best LM Studio, geomean): +35%. Reproduce with tests/bench.sh --family gemma --lmstudio --omlx.

Features

• Run any LLM — every supported MLX architecture and the entire GGUF universe via embedded llama.cpp. DeepSeek V4 Flash runs through the dedicated antirez/ds4 engine.
• OpenAI-compatible API — /v1/chat/completions, /v1/completions, /v1/embeddings, /v1/models, streaming SSE, tools, JSON-schema constrained decoding, logprobs.
• OpenAI Responses API — /v1/responses with previous_response_id chains, per-event sequence_number, the /v1/responses/compact opaque history blob, and a WebSocket transport on the same endpoint.
• Anthropic Messages API — /v1/messages works with Claude Code (ANTHROPIC_BASE_URL=http://localhost:11234) and the Anthropic SDK.
• Ollama-compatible API — /api/chat, /api/generate, /api/tags, /api/show, /api/ps, /api/embed, /api/pull speak the Ollama wire (NDJSON streaming, tool calls with object arguments, thinking, format JSON schemas, name:latest model names), so the whole Ollama client ecosystem works against mlx-serve unchanged.
• Ollama-grade CLI — mlx-serve run gemma4 downloads (resumable), serves, and drops you into a streaming chat REPL; pull / list / serve manage a shared ~/.mlx-serve/models store the GUI app uses too.
• Speculative decoding — PLD (model-agnostic n-gram lookup, on by default) + the Gemma 4 cross-attention drafter. Adaptive prompt-time and runtime gates keep novel-content workloads at parity; agentic code loops see up to 1.6×.
• Native multi-token prediction (Qwen 3.6) — checkpoints shipping a trained mtp/ sidecar (like Qwen3.6-27B-4bit-MTP) speculate with the model's own head automatically: up to 1.8× on agent-style edit loops, self-tuning draft depth, zero setup.
• KV-cache quantization — 4-bit / 8-bit / TurboQuant variants shrink KV memory ~4× / ~2× / further still, so 16K contexts fit on hardware that couldn't hold them dense.
• Continuous batching — --max-concurrent N batches decode requests through one forward pass for ~1.6× throughput at 4-way parallel.
• Prefix cache — shared system-prompt KV reuse across turns and across conversations. v26.5.7 adds an LRU of llama.cpp KV sessions so multi-doc agent loops stay warm.
• Tokenize cache — chat-template render + tokenize cached per request; the second hit on a long conversation is a memcpy. Warm TTFT 7.7× faster on 1.8K-token prompts.
• Vision — Gemma 4 SigLIP encoder; send images via image_url content blocks.
• Reasoning / thinking — full streaming of thinking tokens as reasoning_content.
• No Python — single Zig binary, no pip, no venv. The MLX Core app ships everything signed and notarized.

MLX Core (macOS App)

Menu-bar app that wraps the server with a full UI:

• Model browser — download from HuggingFace with resumable transfers, auto-discovers LM Studio's existing model folder (~/.lmstudio/settings.json) so you don't re-download what's on disk, GGUF rows show a min–max RAM-estimate range.
• Chat interface — multi-session chat with markdown rendering. Drop in PDFs (PDFKit-extracted) or images alongside text.
• Agent mode — 10 built-in tools (shell, cwd, readFile, writeFile, editFile, searchFiles, listFiles, browse, webSearch, saveMemory) with automatic tool calling loop and a per-tool approval dialog (*