ESP32 iPhone Agent
Let an AI model use a real iPhone — with a $7 ESP32. No Mac, no developer account, no jailbreak.
English · 简体中文
Example 1 — draw on a canvas. "Open Whiteboard and draw a smiley face": the agent searches for the app, opens it and draws the face with continuous strokes. Every step's screenshot, with its tap or stroke drawn on top, stays on the step wall.
Example 2 — set an alarm. "Set an alarm for 7:00 AM that repeats every day, then save it": the agent opens Clock, types 0700 into the time wheel, ticks the days under Repeat several taps at a time, and saves. Left: the real iPhone with the ESP32 plugged in; right: the web UI.
The agent sees the phone through AirPlay screen mirroring and touches it through a tiny ESP32-S3 board that plugs into the phone's USB port as a mouse and keyboard. Any OpenAI-compatible vision model can be the brain — a cloud API, or a model you run yourself.
Why
Most iPhone automation drives the phone through XCTest / WebDriverAgent. That needs a Mac, Xcode, a signed helper app and an Apple developer account (a free account's signature expires every 7 days). This project uses the same paths a person uses — the screen and a pointer — so the phone needs nothing installed.
| ESP32 iPhone Agent | XCTest / WebDriverAgent tools | |
|---|---|---|
| Apple developer account | Not needed | Needed (to sign the runner) |
| Xcode | Not needed | Needed |
| Computer | Mac or Linux | Mac |
| App installed on the phone | None | Test runner app |
| Jailbreak | Not needed | Not needed |
| Extra hardware | ESP32-S3 board (~$7) | USB cable |
| Works in any app, incl. games | Yes (it only sees pixels) | Mostly (depends on the accessibility tree) |
How it works
- Eyes — AirPlay mirroring. The phone mirrors its screen to UxPlay running on your computer; a GStreamer pipeline turns the stream into JPEG frames for the agent and the web page.
- Hands — a USB HID bridge. An ESP32-S3 plugged into the phone shows up as a standard USB mouse and keyboard.
With iOS AssistiveTouch on, the mouse moves an on-screen pointer, so a click is a tap and a drag is a swipe.
The board connects to the computer over Wi-Fi and executes simple line commands (
move,click,type, …); all logic stays on the computer. - Brain — a planner and an actor. A planner model writes a short plan from the goal and the current screen. An actor model then looks at each new screenshot and picks the next action: tap, swipe, draw a path, type, press a key, go home, or finish. Both roles can be the same model.
- Closing the loop. The pointer is relative, so the server tracks where it is. Before each tap it wiggles the pointer, finds it on the mirrored screen and corrects any miss, refining the calibration as it goes. After each action it checks whether the screen actually changed and tells the model, so a missed tap is noticed and retried instead of built upon. Repeated actions trigger a re-plan; a dropped link or a bad reply is recovered from.
Hardware
| Part | Notes |
|---|---|
| ESP32-S3 board | Tested with a bare Seeed Studio XIAO ESP32S3 (~$7, no camera add-on needed); attach the antenna that comes with it, as the XIAO has no on-board antenna and Wi-Fi drops often without it. Any ESP32-S3 board whose USB-C port is wired to the S3's native USB works. |
| iPhone | USB-C models plug in directly. Lightning models need a Lightning to USB Camera Adapter. The phone powers the board. |
| Computer | macOS or Linux, on the same network as the phone; runs UxPlay and the server. |
| Vision model | Any OpenAI-compatible chat API that accepts images. Running a model locally is optional. |
Use a USB-C hub with power pass-through if the phone should keep charging while the board is plugged in.
Quick start
Set up with an AI coding agent. Paste this into Claude Code, Codex or a similar agent on the computer the board will connect to:
Clone https://github.com/onehumanbeing/esp32-iphone-agent and follow its README to set it up on this machine: flash the ESP32-S3 plugged in over USB, install UxPlay and the server, write server/.env for my model API, start the server and give me the web UI address. Ask me for the Wi-Fi name/password and the model API key.
1. Flash the board
Install arduino-cli and the ESP32 core (3.x), then:
cp hardware/hid_bridge/secrets.example.h hardware/hid_bridge/secrets.h
# edit secrets.h: 2.4 GHz Wi-Fi name/password and the IP address of the computer running the server
hardware/flash.sh # optional arguments: [serial port] [--debug]
2. Install and start the server
# UxPlay — macOS:
brew install uxplay
# UxPlay — Debian / Ubuntu:
sudo apt install uxplay gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-libav
cd server
python3 -m venv .venv && .venv/bin/pip install -e .
cp .env.example .env # set LLM_BASE_URL / LLM_MODEL / LLM_API_KEY
set -a && . ./.env && set +a && .venv/bin/python -m hidagent
Open http://<computer-ip>:8890 in a browser.
3. Set up the phone (once)
- Settings → Accessibility → Touch → AssistiveTouch: turn it on.
- Plug the board into the phone. A round pointer appears; the web page shows controller connected.
- Control Center → Screen Mirroring → iPhone Agent. The live screen appears in the web page.
- In the web page, open Manual controls → Calibrate. The server moves the pointer, finds it on the live screen and calibrates itself in a few seconds; if it cannot see the pointer, it asks you to click exactly on it.
4. Give it a task
Type a goal (for example "Open Settings and go to Wi-Fi") and press Start, or click one of the examples.
Web UI
- Live phone — click to tap, drag to swipe, scroll to scroll; manual Home, Back, typing and keys.
- Live log — every plan, thought and action as it happens, with timings.
- Step wall — the screenshot the model decided on for every step, with its tap or swipe drawn on top, plus the model's reasoning and how long thinking and acting took. Click a card to zoom in.
- Taps are marked with a red ring and a spreading ripple, swipes and strokes with a yellow path; the Motion switch
turns the animation off. Taps the model marked
precise(pointer checked before tapping) are labelled in the log. - Emergency stop (or Esc) stops the agent and releases the pointer. Clear history empties the log.
- English and Simplified Chinese interface.
Configuration
Set in server/.env (see .env.example):
| Variable | Default | Meaning |
|---|---|---|
LLM_BASE_URL |
http://localhost:8000/v1 |
OpenAI-compatible endpoint |
LLM_MODEL / LLM_API_KEY |
— | Model name and API key |
LLM_EXTRA_BODY |
{} |
JSON merged into every request (e.g. to turn off a model's thinking mode) |
LLM_TOKEN_PARAM |
auto |
Token-limit field: max_tokens, max_completion_tokens (newer OpenAI models), or auto to detect |
ACTOR_* / PLANNER_* |
— | Per-role overrides of any LLM_* variable |
AIRPLAY_NAME |
iPhone Agent |
Receiver name in the phone's Screen Mirroring list |
PORT / DEVICE_PORT |
8890 / 7300 |
Web UI port / port the board connects to |
HIDAGENT_DATA |
~/.hidagent |
Calibration and debug traces |
HIDAGENT_DEBUG |
0 |
Debug mode, see below |
Local model (optional)
A cloud API is enough. If you want everything on your own hardware, this is the setup the project was developed and timed with:
- Model: nvidia/Qwen3.8-Flash-Next-NVFP4
- Serving: Qwen3.8-Flash-Dual-DGX-Sparks-TensorFold — two NVIDIA DGX Sparks with tensor parallelism (TP=2), OpenAI-compatible API
- Speed: about 1.3 s per action and 2 s per plan, with thinking mode off:
LLM_BASE_URL=http://<dgx-spark-ip>:8888/v1
LLM_MODEL=Qwen3.8-Flash-Next
LLM_EXTRA_BODY='{"chat_template_kwargs": {"enable_thinking": false}}'
Any other OpenAI-compatible server (vLLM, SGLang, llama.cpp, Ollama, …) with a vision model works the same way.
Debug mode
Set HIDAGENT_DEBUG=1 to log every board command with its latency and to save one folder per agent run:
~/.hidagent/traces/20261012-005207/
├── steps.jsonl # every model call (prompt, raw reply, latency) and every