SimAgentation

Annotate a running iOS simulator in your browser, and hand the notes to your coding agent.

Features | Getting started | Design Mode | Connect an agent | SDK | Configuration

[!TIP] Let your agent set it up. Paste this into Claude Code, Codex, Cursor or any other coding agent:

Set up SimAgentation for me by following https://github.com/lcandy2/sim-agentation/raw/refs/heads/main/SETUP.md

SimAgentation is Agentation for iOS. Freeze the simulator's screen, click an element or drag a box around anything, and write what should change. Your agent gets the element, the screen it is on, strings to search the source for, and two screenshots: the whole screen with the box drawn in red, and a close-up of the box.

Your app needs no SDK. Everything comes from the simulator's accessibility tree, so it works with any app on any booted simulator, Apple's own included.

[!NOTE] The native side uses Xcode's private CoreSimulator, SimulatorKit and AccessibilityPlatformTranslation frameworks, so a new Xcode release can break it until it's updated.

Features

• Design Mode: hover to see elements, then click one or drag a box over any area. Shift adds more to the selection, and one note covers them all; ↑ and ↓ walk to the parent element and back.
• Straight to your agent: an MCP server with seven tools. In Claude Code, each annotation can land in a running session as you send it, an idle one included. Or copy every pending annotation as Markdown and paste it anywhere.
• A page laid out like Xcode's Device Hub: the device drawn with Xcode's own DeviceKit artwork, hardware buttons that slide out on hover, and Simulator.app's shortcuts. Simulators can be created, started, renamed, reset and removed from the page.
• A hardware-encoded stream: H.265 or H.264 from VideoToolbox, decoded by WebCodecs, at about 60 fps and 3–8 Mbit/s while scrolling. On Auto, the codec, resolution, bitrate and frame rate adapt to the browser and to what the connection carries.
• iPhone Duo in 3D: Xcode 27.1's foldable, drawn with Apple's own model and posed by its hinge, with Device Hub's pose picker and a hinge slider. Design Mode works on the bent screen too.
• An optional SDK for apps you build: exact selections, and the file and line each tagged view comes from.

Getting started

You need macOS on Apple Silicon with Xcode 26 (27.1 for iPhone Duo). SETUP.md walks an agent through everything below; the prompt at the top of this page hands it over.

With an agent, the plugin is all you need: it brings SimAgentation along. On its own, install it with Homebrew:

brew install lcandy2/tap/sim-agentation
sim-agentation serve --open

brew services start sim-agentation keeps it running in the background instead.

To build from source you also need Node with pnpm. From the repo:

pnpm install
pnpm build    # the UI into app/web/dist, then the Swift binary
pnpm start    # app/host/.build/debug/sim-agentation serve --open

app/scripts/package.sh builds the release tarball the formula and the plugins install. To release, run the Release workflow in the Actions tab with the new version: it sets the version everywhere, publishes the release, and points the Homebrew formula at it; the plugins follow from main.

The page opens on http://localhost:38470. Pick a simulator on the left; one that isn't running shows a Start button, and nothing boots until you press it.

[!TIP] While pnpm start runs, pnpm dev adds hot reload: the page on 38470 loads the UI from the Vite dev server on 38472, so edits show up without a reload.

Using it

The page follows Xcode's Device Hub. The simulators sit on the left, with + to create one and a filter for all, running, iPhone and iPad. The device is in the middle, with Home, screenshot, screen recording (MP4) and rotate below it. The inspector on the right has two tabs: device info with the stream's settings, and your annotations. … holds everything else, from shutting down, renaming or resetting the simulator to Simulator.app's Device and Features menus.

In Interact mode you use the app: click and drag to touch, scroll with the wheel, type with the keyboard. A narrow window puts the device list away first, then the inspector, and brings them back when it widens.

Design Mode

1. Press ⇧⌘D. The screen freezes and the accessibility tree loads.
2. Hover to see elements. ↑ selects the parent, ↓ goes back.
3. Click an element, or drag a box over any area. Hold Shift to add to the selection.
4. Write what should change and press Return (⇧Return for a new line). The annotation shows up in the inspector, numbered on the screen. Click its number to open it again and change the note, or pick something else for it.
5. Press Esc to drop the selection, and again to go back to the live screen.

Rotated devices work too, and the screenshots go to the agent upright. Click Copy in the annotations tab to put every pending annotation on the clipboard as Markdown, for an agent without MCP.

Shortcuts

The rest of Simulator.app's shortcuts (shake, appearance, text size, Face ID) work too; … lists them all.

The stream

The stream picks its codec on its own: H.265, else H.264, else JPEG, taking the first this browser decodes in hardware, and moving on if the host can't encode it or nothing decodes within a few seconds. Video runs at about 60 fps, the simulator's own rate, at 3–8 Mbit/s while scrolling, against about 85 Mbit/s for JPEG.

On Auto, resolution, bitrate and frame rate follow too (auto.js). Once a second the host reports how far behind the viewer is, and the page how late that report arrived, which also catches a backlog in a tunnel or proxy. When either grows, the bitrate drops to what got through, then the resolution halves, then the frame rate drops to 30 fps. While anything is held back, the host pads the stream briefly to test for room, so a connection that recovers is used again within seconds. A hidden tab gets 5 fps. The info tab shows what's running and why, and lets you pin a codec, turn on 4:2:2 chroma for H.265, halve the resolution or set the bitrate.

iPhone Duo

Xcode 27.1's foldable is drawn as Device Hub draws it: Apple's own model from Xcode, rendered on the Mac with RealityKit and streamed like the screen. The hinge poses the book. The stream follows whichever of its two panels is lit, and touches and the chrome go with it. Under the device sit Device Hub's pose picker (closed, open, flat) and a hinge slider; the cube button shows the lit panel flat in its chrome instead.

Touches land on whichever half of the bent screen they hit. In Design Mode, boxes and labels lie on the screen in perspective, cut along the hinge, and a half turned more than 60° away from you is left out. Fold it while annotating and the screen goes live, then freezes again once the hinge settles. The screenshots that go to the agent are of the screen itself, flat and upright.

Folding and turning go through a small helper that runs inside the simulator and sends the HID events Device Hub sends, and so do the hardware keys. The host builds it from app/host/Guest/HingeControl the first time it's needed.

Connect an agent

An agent takes two things from SimAgentation: the MCP server, which hands it your annotations, and two skills that teach it to work through them. The plugin brings both, and SimAgentation itself, to Claude Code, Codex and Cursor; any other agent takes the skills and the server on their own.

Claude Code, Codex and Cursor: the plugin

This repository is a plugin marketplace for all three.

# Claude Code
/plugin marketplace add lcandy2/sim-agentation
/plugin install sim-agentation@sim-agentation

# Codex
codex plugin marketplace add https://github.com/lcandy2/sim-agentation.git
codex plugin add sim-agentation@sim-agentation

In Cursor, add the plugin from this repository as [Cursor's guide](https://cursor.com/docs/plugins#in