J.A.R.V.I.S — Voice + HUD for Hermes Agent

A self-hosted, Iron-Man-style voice assistant and command center built on top of Hermes Agent (NousResearch's open-source autonomous agent). Talk to a real agent — one with persistent memory, terminal access, web search, file tools, and 80+ skills — through a glowing arc-reactor HUD in any browser on your LAN, or a push-to-talk client.

Everything runs on your own hardware. The only cloud calls are your LLM provider (via Hermes) and ElevenLabs for the voice. Speech-to-text is fully local (Whisper on CPU).

Demo

▶ Watch the demo on YouTube — live transcription, agent tool calls, holographic media panels, and the cinematic boot, all in real time.

What it does

Click the ring and speak. Your words transcribe live on screen while you talk. The transcript goes to Hermes Agent, which actually does things — reads and writes files, runs commands, searches the web, remembers you across sessions — and the reply streams back as speech, sentence by sentence, while the rest is still being generated. Typical round trip: 3–5 seconds.

The HUD around the ring is a real control center:

• Live agent activity — watch tool calls happen with command previews
• STOP button — halt a runaway agent turn mid-tool-call (Esc works too)
• Approval cards — dangerous commands pause for your ALLOW/DENY
• Interrupt-aware barge-in — cut it off mid-sentence; it knows exactly what you heard and what you didn't
• Embedded dashboards — Hermes' kanban board and session browser pop up in animated viewers, fully interactive
• Holographic media panels — say "show me a video of how arc reactors work, on screen" and a panel swoops in from Z-depth, traces its frame, materializes through a scanline, and plays the video. The agent drives it through a bundled Hermes plugin (hud_display); panels can fly into left/right thirds, and "clear the screen" sweeps them away
• Usage tracking — tokens/day, turns, ElevenLabs quota bar
• Machines panel — live CPU/GPU stats for the host and remote workers
• Cinematic boot — press B: panels flicker in, ring spins up, "Systems online. Good morning."
• Privacy filter — secret-shaped strings are redacted before any text reaches cloud TTS
• Optional GPU ears — point it at any NVIDIA machine on your LAN running the included sidecar and transcription jumps to large-v3-turbo at ~0.2 s, with automatic fallback to local Whisper when that machine is off
• Mobile-ready — responsive layout + Add to Home Screen = full-screen Jarvis app on your phone

Architecture

 Browser HUD (any LAN device)          Host machine (tested on macOS / Apple Silicon)
 ── https/wss :443 ──────────┐   ┌──────────────────────────────────────┐
   mic · speaker · panels    ├──►│ voice pipeline server (this repo)    │
                             │   │  STT: faster-whisper (local, free)   │   ┌─────────────────┐
 Push-to-talk client         │   │  TTS: ElevenLabs Flash (streaming)   ├──►│ Hermes Agent     │
 ── ws :8765 ────────────────┘   │  HUD + auth + dashboard TLS proxy    │   │  API :8642 (lo)  │
                                 └──────────────────────────────────────┘   │  memory · tools  │
                                                                             │  skills · cron   │
                                                                             └─────────────────┘

One brain, many faces: voice and typed chat share a single persistent Hermes session, so each knows what you said to the other — and memory survives every restart.

Requirements

• A machine for the server (tested: Mac mini, Apple Silicon). Linux should work with minor changes (launchd → systemd).
• Hermes Agent installed and configured with an LLM provider
• Python 3.11+
• An ElevenLabs API key (free tier works; ~0.5 credits/char on Flash)
• Any modern browser on the LAN

Install

Full walkthrough in docs/SETUP.md. Short version:

# 1. Enable the Hermes Agent API server
cat >> ~/.hermes/.env <<EOF
API_SERVER_ENABLED=true
API_SERVER_KEY=$(python3 -c 'import secrets;print(secrets.token_urlsafe(32))')
JARVIS_HUD_TOKEN=$(python3 -c 'import secrets;print("jarvis-"+secrets.token_hex(3))')
ELEVENLABS_API_KEY=your-key-here
EOF
hermes gateway   # or set up its LaunchAgent / service

# 2. This repo
git clone https://github.com/YOURNAME/jarvis-hermes-hud
cd jarvis-hermes-hud/server
python3 -m venv .venv
.venv/bin/pip install fastapi uvicorn requests pyyaml numpy anthropic \
    RealtimeSTT faster-whisper silero-vad websockets psutil
cp config/server.example.yaml config/server.yaml   # edit: your ElevenLabs voice_id etc.
scripts/make-certs.sh                              # self-signed TLS (browser mic needs it)
scripts/make-boot-audio.sh YourName                # one-time boot greeting synthesis

# 3. Run
.venv/bin/python server.py
# open https://YOUR_HOST/hud/ → accept cert → enter your JARVIS_HUD_TOKEN → talk

For auto-start on boot, see launchd/ (macOS) — the plists document two non-obvious macOS traps (external-drive TCC and log paths) that cost us an evening.

Usage

Repo layout

server/          FastAPI voice pipeline + HUD host (the core of this project)
server/hud/      single-file HUD (vanilla JS, no build step)
server/scripts/  start/stop/health/smoke + cert & boot-audio generators
client/          optional Windows/Linux push-to-talk Python client (wake word capable)
worker/          optional GPU sidecars: big-model STT server + stats agent for the Machines panel
hermes-plugin/   Hermes tool plugin: lets the agent summon/dismiss HUD media panels
launchd/         macOS auto-start templates with hard-won TCC + FD-limit notes
docs/            SETUP, ARCHITECTURE (protocols/endpoints), TROUBLESHOOTING

Security model

• The Hermes API key never reaches the browser: the HUD talks through a strict allowlist proxy on the voice server.
• All HUD endpoints + dashboard proxy + browser WebSockets are gated by a token (cookie, entered once per device).
• Hermes' API binds to loopback only; the dashboard binds to loopback only.
• Secret-shaped strings are redacted before text leaves for cloud TTS.
• LAN-only by design — do not port-forward this to the internet.

Credits & license

Built on Hermes Agent by Nous Research. HUD aesthetics inspired by jarvis-dashboard. STT by faster-whisper / RealtimeSTT. Voice by ElevenLabs.

MIT — see LICENSE. Use it, fork it, build your own Jarvis.