mirroir-mcp

Give your AI eyes, hands, and a real iPhone. An MCP server that lets any AI agent see the screen, tap what it needs, and figure the rest out — through macOS iPhone Mirroring. Experimental support for macOS windows. 38 tools, any MCP client.

Vision: agents that test — and play

mirroir is growing archetype-aware coverage for games. A game splits into two surfaces. The shell — menu, shop, loadout, settings, win/lose screens — is ordinary text-labelled UI that today's skills already drive. The scene — the playfield — is a vision-driven loop an agent follows one touch at a time. The honest targets are game QA smoke tests, tutorial walkthroughs, and scripted demos, not real-time play: iPhone Mirroring exposes a single pointer, so twin-stick and twitch games stay out of reach (see Games).

Token reductions make the scene loop affordable. Screenshots are ~90% of an agent's cost, so text-rich shell screens read through describe_screen with no image at all, and a region/scale capture knob trims the per-frame image cost where a screenshot is unavoidable. The cost model is simple: frames × ~2.7k tokens, at ~3 frames/min.

The rollout is tracked as Gaming Surface A–E — shell coverage ships first; the scene loop is measured on a real device before it widens.

Requirements

• macOS 15+
• iPhone connected via iPhone Mirroring

Install

/bin/bash -c "$(curl -fsSL https://mirroir.dev/get-mirroir.sh)"

or via npx:

npx -y mirroir-mcp install

or via Homebrew:

brew tap jfarcand/tap && brew install mirroir-mcp

The first time you take a screenshot, macOS will prompt for Screen Recording and Accessibility permissions. Grant both.

Claude Code

claude mcp add --transport stdio mirroir -- npx -y mirroir-mcp

GitHub Copilot (VS Code)

Install from the MCP server gallery: search @mcp mirroir in the Extensions view, or add to .vscode/mcp.json:

{
  "servers": {
    "mirroir": {
      "type": "stdio",
      "command": "npx",
      "args": ["-y", "mirroir-mcp"]
    }
  }
}

Cursor

Add to .cursor/mcp.json in your project root:

{
  "mcpServers": {
    "mirroir": {
      "command": "npx",
      "args": ["-y", "mirroir-mcp"]
    }
  }
}

OpenAI Codex

codex mcp add mirroir -- npx -y mirroir-mcp

Or add to ~/.codex/config.toml:

[mcp_servers.mirroir]
command = "npx"
args = ["-y", "mirroir-mcp"]

git clone https://github.com/jfarcand/mirroir-mcp.git
cd mirroir-mcp
./mirroir.sh

Use the full path to the binary in your .mcp.json: <repo>/.build/release/mirroir-mcp.

How it works

Every interaction follows the same loop: observe, reason, act. describe_screen gives the AI every text element with tap coordinates (eyes). The LLM decides what to do next (brain). tap, type_text, swipe execute the action (hands) — then it loops back to observe. No scripts, no coordinates, just intent.

Beyond the basics, the server exposes higher-level navigation and lifecycle tools: press_back navigates back by OCR-tapping the "<" chevron (with a canonical-position fallback), press_home/press_app_switcher/spotlight drive system gestures, scroll_to scrolls until a target text becomes visible, reset_app force-quits an app via the App Switcher, set_network toggles connectivity through Settings, and measure times how long an action takes to surface a target element. Multi-target setups use list_targets/switch_target to move between window automation endpoints. Skill authoring is served by record_step and save_compiled (compiled replay) and calibrate_component (test a component definition against the live screen). See the Tools Reference for all 38 tools.

Describe Your App

mirroir can explore any iOS app blindly, but it works better when you tell it what to expect. Write an APP.md file and mirroir reads it before exploration starts:

---
app: Santé
archetype: dashboard
obstacle_mode: auto
---

## Structure
Dashboard with 4 tabs: Résumé, Partage, Parcourir, Profil.

## Résumé Tab
- Summary cards for health metrics that drill down to charts
- Cards often show "Aucune donnée" on test devices

## Obstacles
- Health Access permission → tap "Autoriser"
- Notification permission → tap "Ne pas autoriser"

## Skip
- Supprimer les données de Santé
- Réinitialiser

What the code actually uses today: archetype overrides recipe auto-detection; obstacles are auto-dismissed when obstacle_mode: auto; skip merges with permissions.json.skipElements; tabs (inline or as a section) are injected as high-priority targets; Structure + tab body + Tips become AI context in generated skills.

See the APP.md specification for the complete field list, loader resolution rules, and the permission-system bridge. Three levels of patterns work together — elements (what rows look like), screens (what the page layout means), and apps (what the developer knows). Patterns & Skills covers the full system.

Examples

Paste any of these into Claude Code, Claude Desktop, ChatGPT, Cursor, or any MCP client:

Open Messages, find my conversation with Alice, and send "running 10 min late".

Open Calendar, create a new event called "Dentist" next Tuesday at 2pm.

Open my Expo Go app, tap "LoginDemo", test the login screen with
test@example.com / password123. Screenshot after each step.

Start recording, open Settings, scroll to General > About, stop recording.

Games

Games get three controls through iPhone Mirroring: touch holds a finger across calls for virtual joysticks and press-and-hold (begin, move, end); hold_keys holds keys such as w while dragging the right mouse button, for games that switch to keyboard and mouse controls like Roblox; pinch and rotate drive two-finger zoom and rotation in gesture-recognizer UIs. Mirroring exposes a single pointer touch to apps, so a touch-only game cannot receive two independent fingers — see Games and Multi-Touch for what was measured.

Screen Intelligence

describe_screen is the AI's eyes. Three backends work together to give the agent a complete picture of what's on screen — text, icons, and semantic UI structure.

Apple Vision OCR (default)

The default backend uses Apple's Vision framework to detect every text element on screen and return exact tap coordinates. This is fast, local, and requires no API keys or external services.

Icon Detection (YOLO CoreML)

Text-only OCR misses non-text UI elements — buttons, toggles, tab bar icons, activity rings. Drop a YOLO CoreML model (.mlmodelc) in ~/.mirroir-mcp/models/ and the server auto-detects it at startup, merging icon detection results with OCR text. The AI gets tap targets for elements that text-only OCR cannot see.

AI Vision Mode (embacle)

Instead of local OCR, describe_screen can send the screenshot to an AI vision model that identifies UI elements semantically — cards, tabs, buttons, icons, navigation structure — not just raw text. This produces richer context for the agent, especially on screens with complex layouts.

The embacle runtime is embedded directly into the mirroir-mcp binary via Rust FFI. describe_screen calls the embedded runtime in-process — no separate server, no network round-trip, no additional setup. The FFI layer (EmbacleFFI.swift → libembacle.a) handles initialization, chat completion requests, and memory management across the Swift/Rust boundary.

embacle routes vision requests through already-authenticated CLI tools (GitHub Copilot, Claude Code) so there is no separate API key to manage. If you have a Copilot or Claude Code subscription, you already have access.

Install

brew tap dravr-ai/tap
brew trust --tap dravr-ai/tap # recent Homebrew refuses to load untrusted taps
brew install embacle          # CLI tools (embacle-server, embacle-mcp)
brew install embacle-ffi