Demo
What Can You Do?
"Ok Vision, what am I looking at?"
| Use Case | Example |
|---|---|
| Instant Translation | Point at a menu or sign abroad and get real-time translation |
| Hands-Free Cooking | "What's the next step?" while your hands are covered in dough |
| Smart Shopping | "Is this a good deal?" - get price comparisons and reviews |
| Travel Guide | "Tell me about this building" - instant history and facts |
| Accessibility | Describe surroundings, read text aloud, identify objects |
| Meeting Assistant | "Remember this person is John from Marketing" |
| Learning | "What plant is this?" - identify flora, fauna, landmarks |
| DIY Helper | "How do I fix this?" - show the problem, get step-by-step help |
| Fitness Coach | "Is my form correct?" - real-time posture feedback |
With OpenClaw's 56+ tools: Send emails, control smart home, set reminders, search the web - all hands-free through your glasses.
Features
Five AI Backends — Cloud or Fully On-Device
- Local (MLX): A choice of on-device models via Apple MLX — Qwen 2.5 (0.5B/3B), Gemma 2 2B, Gemma 4 E2B, SmolVLM2 2.2B — so you can trade capability for memory/speed. Private, offline, zero API cost. Pick SmolVLM2 to unlock on-device vision — photo Q&A and a fully-offline live video mode (see below).
- Apple Intelligence: Apple's on-device Foundation Model (iOS 26+). No download, no memory pressure (OS-managed), private and offline. Uses guided generation + Apple's native tool-calling.
- OpenClaw: Wake word activation, 56+ tools, task execution via WebSocket
- Gemini Live: Real-time voice + vision with native audio streaming
- OpenAI: GPT-4o text + vision over the Chat Completions API — works with any OpenAI-compatible endpoint (OpenRouter, Groq, local servers, etc.). Also drives live video via the Realtime API (
gpt-realtime) — continuous voice + camera frames (see below).
Live Video — Real-Time Voice + Vision
Say "Ok Vision, start video stream" to enter a live mode where the glasses camera stays on and the AI answers questions about what you're seeing. Ask freely — no wake word between questions — until you say "stop video". Live video routes to whichever backend you've selected:
- Gemini Live — native continuous audio + 1fps video (cloud).
- OpenAI — the Realtime API (
gpt-realtime): streaming voice + camera frames over WebSocket. Uses the same OpenAI key/base URL you already set (so OpenAI-compatible gateways work too). - Local (SmolVLM2) — fully on-device live video: Apple speech-to-text in, on-device SmolVLM2 answers on the latest frame, spoken back with your chosen voice. No cloud, no cost, works with no signal.
On-Device Photos (SmolVLM2)
With SmolVLM2 selected as your local model, "Ok Vision, take a photo and tell me what this is" captures a frame from the glasses and answers entirely on-device — nothing leaves the phone. Other local models stay text-only and hand camera questions to a cloud backend. (Images are resized on-device to keep the vision encoder within iOS memory limits.)
On-Device Neural Voice (Kokoro)
- A natural, offline, private voice (Kokoro-82M) running on-device via MLX — selectable from a Speech Engine dropdown with a voice picker.
- Apple's system voice stays the default (with Premium/Enhanced voice support); Kokoro is the upgrade when you want lifelike speech with nothing leaving the phone.
Agentic Web Search — Real Live Information
- The models search the web whenever they're unsure or asked about current things — news, weather, prices, scores — and never answer "I can't access real-time data" without trying.
- Tavily (free tier) returns real live content for the model to summarize; DuckDuckGo is the keyless fallback.
- Smart flow: local models reformulate + retry a weak query; the OpenAI backend runs a real function-calling loop (call
web_search→ refine → answer).
Conversation Memory
- Multi-turn context on the on-device and OpenAI backends: "What's the capital of France?" → "What's its population?" just works.
- Bounded per session so local memory stays safe; Apple keeps context via a reused native session.
Hands-Free Productivity Tools
- Run real actions by voice: timers, Pomodoro sessions, reminders (Apple Reminders), calendar events (read today/upcoming, add), notes (auto-tagged with place + time), and copy to clipboard.
- Built on stable Apple frameworks (EventKit, UserNotifications, CoreLocation, UIPasteboard) — deterministic, no hallucination surface.
- Pixel-perfect times: the tool does the date math, not the model. Say "remind me at 6 PM" and it lands at exactly 6:00 PM — even on the tiny on-device model.
- One tool registry, four backends: OpenAI and Gemini Live (function-calling), Apple Intelligence (
Toolprotocol), and on-device Gemma 4 (JSON tool-calls) all share the same tools. - See docs/native-tools.md for the full design.
On-Device Face Recognition (Apple Vision)
- Teach it faces hands-free: "Ok Vision, remember this person as Sara"
- Recognize them later: "Ok Vision, who is this?"
- Runs entirely on-device (Apple Vision
computeDistance) — no cloud, no photos leave your phone - Intent is parsed by the on-device model (agentic) — any phrasing works, and it only triggers for a person actually in view
Smart Voice Control
- Reliable wake word activation ("Ok Vision") for privacy — primed recognition + self-restart so it keeps listening (survives idle, replies, and glasses off/on)
- Barge-in support - interrupt AI anytime by saying "Ok Vision"
- Conversation mode - follow-up questions without wake word
- "Ok Vision stop" - stop AI mid-speech
- Audio routes correctly whether you're using the glasses or the phone alone (loud speaker, not the earpiece)
On-Device Model Management
- Pick a local model, download it on demand, and delete it to reclaim storage anytime from Settings — swap between a tiny 0.5B model and a larger one as you like.
Glasses Integration
- Photo capture on voice command ("take a photo")
- Live video streaming to Gemini (1fps)
- Seamless glasses registration via Meta AI app
Production-Ready
- Auto-reconnect with exponential backoff (12 attempts)
- Network monitoring (auto-pause on WiFi drop)
- App lifecycle handling (suspend/resume connections)
- Secure credential storage
Zero Hardcoding
- All API keys configurable in-app
- No code changes needed to use
- Example config files included
Screenshots
| Screen | Description |
|---|---|
| Voice Assistant | Tap the orb or say "Ok Vision" — live transcripts, distinct listening/thinking/speaking states |
| Settings | Configure AI backend, web search, glasses, voice control, and advanced options |
| AI Backends | Choose Local (MLX), Apple Intelligence (on-device), OpenClaw (tools), Gemini Live (low latency), or OpenAI |
| Local Models | Download and manage on-device models (Qwen, Gemma, SmolVLM2, FastVLM) with real sizes and one-tap switching |
Quick Start
Prerequisites
- macOS with Xcode 15+
- Physical iOS 18+ device (simulator doesn't support Bluetooth; on-device MLX models need iOS 18)
- Meta Ray-Ban smart glasses
- Meta Developer account for glasses registration
- An AI backend — one of:
- Local (MLX) — no account/key needed; a choice of on-device models (Qwen 2.5, Gemma, SmolVLM). Runs on a recent iPhone (e.g. 15 Pro/16/17)
- Apple Intelligence — no key or download; needs iOS 26+ on an Apple-Intelligence device (iPhone 15 Pro and newer)
- OpenClaw instance
- Gemini API key
- OpenAI API key (or any OpenAI-compatible endpoint)
Step 1: Clone & Configure
git clone https://github.com/rayl15/OpenVision.git
cd OpenVision/meta-vision
# Copy config templates
cp Config.xcconfig.example Config.xcconfig
cp OpenVision/Config/Config.swift.example OpenVision/Config/Config.swift
Step 2: Get Meta Credentials
- Go to Meta Developer Console
- Create an app or use existing one
- Enable "Wearables" capability
- Copy your App ID and Client Token
Step 3: Edit Config.xcconfig
# Your Apple Team ID (fro