Rapid-MLX

Single-user end-to-end throughput (B=1: one request at a time, 256 max output tokens, output_tokens / wall-clock incl. first-token latency), median of 3 rounds. chat_template_kwargs.enable_thinking=False passed where the engine honours it. Tested on M3 Ultra 256 GB / rapid-mlx v0.6.83 (fused top-p sampler). ¹ carried over from 2026-04 bench — disk-constrained on this refresh.

• tok/s (tokens per second) — roughly how many words the AI generates per second. Higher = faster.
• 4bit / 8bit — compression levels for models. 4bit uses less memory (recommended); 8bit is higher quality.
• TTFT (Time To First Token) — how long before the AI starts responding.
• Tool calling — the AI can call functions in your code. Used by Cursor, Claude Code, and coding assistants.
• OpenAI API compatible — Rapid-MLX speaks the same language as ChatGPT's API, so any app that works with ChatGPT can work with Rapid-MLX by just changing the server address.
• Ollama / llama.cpp — other popular tools for running local AI. The only apples-to-apples row in our table is GPT-OSS 20B (identical weights both sides) — Rapid-MLX runs it 2.3x faster than Ollama under B=4 concurrent load. On the Qwen3 closest-tag rows (Qwen3.5/3.6 DeltaNet isn't on llama.cpp yet, so we compare against qwen3:Nb) Rapid-MLX leads 1.7–2.4x. The Gemma 4 row is tied at parity with Ollama's Gemma 3 (different architectures, 1.0x). Against mlx-lm serve (same MLX weights) Rapid-MLX is 1.2–1.5x faster. Full caveats in Benchmarks.

Quick Start

Step 1 — Install (pick one):

# uv (recommended — one command, isolated env, auto-manages Python)
uv tool install rapid-mlx@latest
# Don't have uv yet? Install it first: curl -LsSf https://astral.sh/uv/install.sh | sh

# Or one-liner with auto-setup (installs Python if needed)
curl -fsSL https://raullenchai.github.io/Rapid-MLX/install.sh | bash

# Homebrew (Mac-native — needs tap + trust before install on Homebrew 4.x)
brew tap raullenchai/rapid-mlx
brew trust raullenchai/rapid-mlx
brew install rapid-mlx

# pip (requires Python 3.10+ — macOS ships 3.9, so install Python first if needed)
pip install rapid-mlx

Upgrade later: uv tool upgrade rapid-mlx / brew upgrade rapid-mlx / pip install -U rapid-mlx.

Vision/multimodal models (Gemma 4, Qwen-VL, etc.) need extras: pip install 'rapid-mlx[vision]'. Text-only install is ~460 MB; vision adds ~322 MB. See Optional Extras for the full list.

"No matching distribution" error? Your Python is too old. Run python3 --version — if it says 3.9, install a newer Python: brew install python@3.12 then python3.12 -m pip install rapid-mlx

Refusing to load formula ... from untrusted tap? Homebrew 4.x requires third-party taps to be explicitly trusted before install. The brew trust raullenchai/rapid-mlx line above is what marks the tap as trusted — without it, even after brew tap, the install is refused. Trust is per-machine and persists across upgrades.

Tapping homebrew/core / Operation not permitted during brew install? Brew 5.x's install sandbox can't auto-tap homebrew/core mid-install. Pre-tap it once, then retry:

brew tap homebrew/core --force   # ~1.3 GB, one-time
brew tap raullenchai/rapid-mlx
brew trust raullenchai/rapid-mlx
brew install rapid-mlx

Step 2 — Talk to a model right now (one command, no second terminal):

rapid-mlx chat

Defaults to qwen3.5-4b-4bit. First run downloads the model (~2.5 GB) — you'll see a progress bar. Drops you into a REPL when it's ready. Type /help for slash commands, /exit to quit. Pass --think to surface chain-of-thought.

Step 2b — Or serve a model for use from other apps:

rapid-mlx serve qwen3.5-4b-4bit

Same model, same download — but this starts an OpenAI-compatible HTTP server instead of a REPL. Wait for Ready: http://localhost:8000/v1.

Want vision? pip install 'rapid-mlx[vision]' then rapid-mlx serve gemma-4-26b-4bit (~14 GB).

Step 3 — Hit the API (from a second terminal tab):

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"default","messages":[{"role":"user","content":"Say hello"}]}'

That's it — you now have an OpenAI-compatible AI server on localhost:8000. Point any app at http://localhost:8000/v1 and it just works.

Step 4 — Share it publicly (optional — get a https:// URL anyone can hit):

rapid-mlx share qwen3.6-27b-8bit

This spawns the same local serve and tunnels it through rapidserver.quicksilverpro.io over a WebSocket. Your terminal prints a public OpenAI-compatible endpoint plus a bearer key — point any chat UI or OpenAI SDK at it. Bearer auth, a locked-down CORS allowlist, and a default 120 RPM rate-limit are wired on the spawned child; closing the terminal tears the tunnel down.

The default chat surface is our hosted Big-AGI fork (tool calling, personas, voice — no signup); any OpenAI-compatible client also works, e.g. OPENAI_API_BASE_URL=<share-url>/v1 OPENAI_API_KEY=<bearer> open-webui serve.

Pick a 27B-class model or larger for a usable share experience — 4B is fine for local dev but too small for live chat (rapid-mlx models lists all aliases).

Want a Claude Code-like TUI? Rapid-MLX is the backend — pair it with an open-source agent CLI like OpenCode or codex for the full slash-commands / tool-use / multi-turn experience. Run rapid-mlx agents opencode --setup (or codex --setup) to wire it up automatically.

Tip: Run rapid-mlx models to see all available model aliases. For a smaller/faster model, try rapid-mlx serve qwen3.5-9b-4bit (~5 GB).

From source (for development):

git clone https://github.com/raullenchai/Rapid-MLX.git
cd Rapid-MLX && pip install -e .

Vision models (adds mlx-vlm + opencv + torch, ~322 MB extra):

pip install 'rapid-mlx[vision]'

Audio (TTS/STT via mlx-audio):

pip install 'rapid-mlx[audio]'

Not into the terminal? Rapid-MLX Desktop is a Mac app that bundles the same rapid-mlx engine inside a one-click GUI — drag to Applications, pick a model, chat. No Python, no pip, no brew. The CLI here is still the source of truth for serving and scripting; the desktop app is the friendlier on-ramp.

Try it with Python (make sure the server is running, then pip install openai):

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")  # any value works, no real key needed

response = client.chat.completions.create(
    model="default",
    messages=[{"role": "user", "content": "Say hello"}],
)
print(response.choices[0].message.content)

Works With

Agent Harnesses (MHI-tested)