esp32-thinks

A multi-model Muse — a personal AI assistant on ESP32, talking out of the box.

Think of it as a Muse that thinks with your models: say "Hi, ESP" (or press the knob), ask anything, and it answers out loud — streaming sentence-by-sentence while the model is still writing, in English, Chinese or Korean, on a small gadget with a mic, a speaker and a round display.

No vendor lock on the brain: speech, transcription and chat are three independent endpoints, each pointed at any OpenAI-compatible server — one provider for all three, or a different one per task.

• ESP32-S3 · built on Meta's open-source muse-gadget-sdk · license notices
• Models: OpenAI, QuickSilver, or any mix — presets for the common ones, fully custom for the rest

Why

Cloud assistants rent you a brain: one vendor's models, one vendor's pricing, one vendor's uptime, and your voice on their servers. The gadget hardware is fine — the lock is in the firmware.

esp32-thinks flips that. The device runs your stack: pick the cheapest transcription, the smartest chat model, the voice you like — per task, changeable in one config line and a rebuild. The implementation is a reviewable patch against a pinned open-source SDK, so you can read it, change it, or point it at a local server.

Performance (measured on device)

QuickSilver preset: qwen3.7-flash chat, whisper-large-v3-turbo STT and chatterbox-turbo / qwen3-tts speech. v0.2.10 allows one or two natural sentences, rather than the old 15-character answers. Longer replies take longer to synthesize.

These are individual samples, not latency guarantees. The final flashed firmware returned a 29-word English answer in 7.1 s, 72 Chinese characters in 15.1 s and 76 Korean characters in 10.3 s. A greeting took 1.7 s and an immediate post-wake question took 1.9 s. The Chinese sample contained 66 Han characters: the prompt’s approximate length bound can be exceeded. Typed timings include chat, speech fetch and decoding; the console drains audio without playing it through the speaker. They do not measure voice-to-first-sound latency.

See v0.2.10 validation and known limits for exact test coverage, and REFERENCE for the configuration and historical latency receipts. Earlier 2–3 s figures used much shorter replies and should not be applied to the new default persona.

What you get

• Wake word + button — say "Hi, ESP" (esp-sr WakeNet10, runs on-chip) or hold the knob; both start a turn. Wake turns end on a voice-activity timer; the button works exactly like before.
• Bring your own brain — three independent OpenAI-compatible endpoints (STT / chat / TTS) in one config file. Mix providers freely; presets: openai, quicksilver, custom.
• Streaming voice replies — the chat stream is parsed at sentence boundaries (ASCII + CJK); each sentence is synthesized the moment it lands and appended to the audio already playing. First word arrives while the model is still generating.
• Long-lived brain connection — TLS is prepared before you ask, probed every 180 s while idle, and resumed with session tickets after an idle close. A failed probe triggers background preparation; a disconnect between probes may still need turn-time reconnection.
• Barge-in — press the knob while Muse is speaking: the reply stops mid-sentence, in-flight requests are cancelled, and your next words start a fresh turn immediately.
• Silence guard — a quiet recording (interrupted turn, pocket press) is discarded instead of becoming a hallucinated answer.
• Automatic stall recovery — outstanding I2S or LCD work gets a 20 s deadline, followed by a zero-allocation task dump and reboot. Normal sleep and slow cloud responses don't count as stalls.
• Language following — answer in the user’s language, defaulting to English when unclear. English, Chinese and Korean are tested; STT detects language automatically. Replies aim for one or two sentences, about 40 words or 60 Chinese characters.
• Language-aware voice — Chinese and Korean reply text selects the optional multilingual TTS model. The QuickSilver preset uses qwen3-tts for these replies and chatterbox-turbo for English.
• Headless provisioning — put your Wi-Fi in the config; the device self-provisions on boot. No app, no BLE dance, no vendor account needed.
• Typed turns exercise chat and TTS — the serial console path runs the same inference and decoding pipeline without microphone capture.

At a glance

Quick start

# 0) One-time: ESP-IDF v6.x for esp32s3 (docs/REFERENCE.md §Prerequisites)

git clone https://github.com/facebookincubator/muse-gadget-sdk.git
git -C muse-gadget-sdk checkout 812c46f
git clone https://github.com/raullenchai/esp32-thinks.git
cd esp32-thinks

# 1) Back up the factory NVS once — your restore path if anything goes sideways
./scripts/backup-nvs.sh ../muse-gadget-sdk /dev/ttyACM1

# 2) Apply the patch + configure your brain (pick a preset)
./apply.sh ../muse-gadget-sdk --preset quicksilver   # or openai / custom

# 3) Build + flash (CH342-safe pacing; allow several minutes)
./build-and-flash.sh ../muse-gadget-sdk /dev/ttyACM1

With a preset + headless Wi-Fi in step 2, that's the whole setup: unplug, re-plug, and talk — press the knob, or just say "Hi, ESP".

How a turn flows

 "Hi, ESP" / knob ──▶ record (pre-roll holds the last second)
                        │  ends: voice VAD (~1.4 s silence) or knob release
                        ▼
              trim to speech ──▶ POST WAV/ADPCM ─▶ STT endpoint
                        │                         (¼-size upload, keep-alive)
                        ▼
              POST /chat/completions (SSE) ──────────────▶ chat endpoint
                        │  split at sentence boundaries (ASCII + 。！？…)
                        ▼
              per sentence: POST /audio/speech ─▶ MP3 appends to the buffer
                        │                          already decoding + playing
                        ▼
                 first word out loud while the model is still writing

• Connection care is off the critical path: the first TLS handshake happens after Wi-Fi comes up (and before a recording if needed), a HEAD probe re-checks the client every 180 s while idle, and tickets enable TLS 1.3 resumption. A keep-alive FIN surfaces as an empty 200 and gets one transparent retry before any reply data has started.
• Short replies (≤ 128 UTF-8 bytes) reuse the completed chat connection for speech; longer replies stream sentences on a separate speech connection. Each endpoint pins its own root CA, and tickets never cross origins.
• On-device the screen follows the turn: listening (mic accent + level meter) → thinking (ring spinner, caption shows the transcript) → speaking (level follows the audio, caption follows the text).

Connection preparation starts after Wi-Fi comes up. A first question can still pay connection costs if preparation has not finished, or when a longer reply needs a separate speech connection. See REFERENCE for historical measurements and connection behavior.

Serial console

The USB serial port (/dev/ttyACM1) is a full console, handy for bench testing: