name: gui-agent description: "Visually operate any application on macOS — native apps, browsers, or anything with a GUI window. Open apps, click buttons, type text, send messages, fill forms, navigate menus, all driven by visual UI detection. Use when the task requires interacting with what's on screen."

GUI Agent Skill

You ARE the agent loop. Every GUI task follows this flow, in order:

INTENT MATCH → OBSERVE → ENSURE APP READY → ACT → VERIFY → SAVE WORKFLOW → REPORT

Sub-Skills

Each step has detailed instructions in its own skill file:

Read the relevant sub-skill when you reach that step. You don't need to read all of them upfront.

agent.py — Unified Entry Point

All GUI operations go through scripts/agent.py. Do not call app_memory.py or gui_agent.py directly.

source ~/gui-agent-env/bin/activate

# Core operations
python3 scripts/agent.py open --app AppName
python3 scripts/agent.py learn --app AppName
python3 scripts/agent.py detect --app AppName
python3 scripts/agent.py click --app AppName --component ButtonName
python3 scripts/agent.py list --app AppName
python3 scripts/agent.py read_screen --app AppName
python3 scripts/agent.py wait_for --app AppName --component X
python3 scripts/agent.py cleanup --app AppName
python3 scripts/agent.py navigate --url "https://example.com"
python3 scripts/agent.py workflows --app AppName
python3 scripts/agent.py all_workflows

# Messaging
python3 scripts/agent.py send_message --app WeChat --contact "小明" --message "明天见"
python3 scripts/agent.py read_messages --app WeChat --contact "小明"

agent.py automatically handles:

• New app → learn → plan
• Known app → eval (template match; ≥80% proceed, <80% re-learn)
• Error → re-learn + new plan
• Chinese aliases: 微信→WeChat, 浏览器→Chrome

Execution Flow

STEP -1: INTENT MATCHING

→ Details: skills/gui-workflow/SKILL.md

Match user request to saved workflows before doing anything. If matched, use workflow steps as plan. If not, proceed and save after success.

STEP 0: OBSERVE

→ Details: skills/gui-observe/SKILL.md

Screenshot, identify current state. Record session_status for token reporting.

STEP 1: ENSURE APP READY

→ Details: skills/gui-learn/SKILL.md

Check if app is in memory. If not → learn. If match rate < 80% → re-learn. This is YOUR responsibility — do not wait for the user.

STEP 2: LEARN (when needed)

→ Details: skills/gui-learn/SKILL.md

Detect all components (YOLO + OCR), identify them, filter, save to memory. Privacy check: delete personal info.

STEP 3: ACT

→ Details: skills/gui-act/SKILL.md

Execute clicks, typing, sending. Pre-verify before every click. Pre-verify contact before every message send.

STEP 4: POST-ACTION VERIFY

→ Details: skills/gui-act/SKILL.md

Screenshot after every action. Did the expected change happen? If not → re-observe.

STEP 5: SAVE WORKFLOW

→ Details: skills/gui-workflow/SKILL.md

Save successful multi-step sequences for future replay.

STEP 6: REPORT

Every GUI task ends with a report:

⏱ 45.2s | 📊 +10k tokens (85k→95k) | 🔧 3 screenshots, 2 clicks, 1 learn

Compare session_status from STEP 0 vs now.

Key Principles

1. Vision-driven, no shortcuts — every GUI interaction goes through the visual pipeline (screenshot → detect → match → click). Do not use system commands (open <url>, osascript tell app to set URL, CLI tools) to manipulate app state. Only allowed system calls: activate (bring window to front), screencapture (take screenshot), cliclick (click/type after visual detection provides coordinates).
2. Memory first, detect second — template match before YOLO+OCR
3. Template > OCR > YOLO > LLM — cheapest method first
4. Relative coordinates — all positions relative to window top-left
5. Window-based, not screen-based — capture and operate within target window only
6. Paste > Type for CJK text and special chars
7. Learn incrementally — save new components after each interaction
8. Integer coordinates only — cliclick requires integers
9. Learn once, match forever — UI positions are stable unless app updates

Safety Rules

These exist because of real bugs:

1. Verify before sending — always OCR the chat header to confirm correct contact
2. Stay within window bounds — get bounds first, filter all OCR/clicks
3. No wrong-app learning — validate window bounds before auto_learn
4. Reject tiny templates — <30×30 pixels produce false matches
5. LLM never provides coordinates — detection tools provide coordinates, you decide what to click by name
6. Never send screenshots to conversation — internal detection only

Memory System

→ Details: skills/gui-memory/SKILL.md

Visual memory stores app profiles, components, page fingerprints, workflows. See gui-memory for directory structure, profile schema, CRUD operations, and cleanup rules.

File Structure

gui-agent/
├── SKILL.md              # This file (main orchestrator)
├── skills/               # Sub-skills (read on demand)
│   ├── gui-observe/SKILL.md
│   ├── gui-learn/SKILL.md
│   ├── gui-act/SKILL.md
│   ├── gui-memory/SKILL.md
│   ├── gui-workflow/SKILL.md
│   └── gui-setup/SKILL.md
├── scripts/              # Core scripts
│   ├── agent.py, ui_detector.py, app_memory.py, gui_agent.py, template_match.py
├── memory/               # Visual memory (gitignored)
│   ├── apps/<appname>/
│   └── meta_workflows/
├── actions/              # Atomic operations
├── docs/
└── README.md

🔥 News

• [03/19/2026] v0.4.0 — Workflow memory + async polling: Saved workflows auto-matched by LLM intent; wait_for command (template-match polling, no blind clicks); mandatory timing & token delta reporting; multi-window fix (selects largest window).
• [03/19/2026] v0.3.0 — Click-graph state architecture: UI modeled as a graph of states; each click creates a new state entry; state identification via OCR text matching. Removed pages/regions/overlays complexity.
• [03/17/2026] v0.2.0 — Workflow-based revise, event-driven polling, mandatory operation protocol (observe→verify→act→confirm), per-app visual memory with auto-cleanup.
• [03/16/2026] v0.1.0 — GPA-GUI-Detector integration, Apple Vision OCR, template matching, browser automation, per-site memory.
• [03/10/2026] v0.0.1 — Initial release: WeChat/Discord/Telegram automation, app profiles, fuzzy app matching.

💬 What It Looks Like

You: "Send a message to John in WeChat saying see you tomorrow"

OBSERVE  → Screenshot, identify current state
           ├── Current app: Finder (not WeChat)
           └── Action: need to switch to WeChat

STATE    → Check WeChat memory
           ├── Learned before? Yes (24 components)
           ├── OCR visible text: ["Chat", "Cowork", "Code", "Search", ...]
           ├── State identified: "initial" (89% match)
           └── Components for this state: 18 → use these for matching

NAVIGATE → Find contact "John"
           ├── Template match search_bar → found (conf=0.96) → click
           ├── Paste "John" into search field (clipboard → Cmd+V)
           ├── OCR search results → found → click
           └── New state: "click:John" (chat opened)

VERIFY   → Confirm correct chat opened
           ├── OCR chat header → "John" ✅
           └── Wrong contact? → ABORT

ACT      → Send message
           ├── Click input field (template match)
           ├── Paste "see you tomorrow" (clipboard → Cmd+V)
           └── Press Enter

CONFIRM  → Verify message sent
           ├── OCR chat area → "see you tomorrow" visible ✅
           └── Done

"Scan my Mac for malware"

OBSERVE  → Screenshot → CleanMyMac X not in foreground → activate
           ├── Get main window bounds (largest window, skip status bar panels)
           └── OCR window content → identify current state

STATE    → Check memory for CleanMyMac X
           ├── OCR visible text: ["Smart Scan", "Malware Removal", "Privacy", ...]
           ├── State identified: "initial" (92% match)
           └── Know which components to match: 21 components

NAVIGATE → Click "Malware Removal" in sidebar
           ├── Find element in window (exact match, filter by window bounds)
           ├── Click → new state: "click:Malware_Removal"
           └── OCR confirms new state (87% match)

ACT      → Click "Scan" button
           ├── Find "Scan" (exact match, bottom position — prevents matching "Deep Scan")
           └── Click → scan starts

POLL     → Wait for completion (event-driven, no fixed sleep)
           ├── Every 2s: screenshot → OCR check for "No threats"
           └── Target found → proceed immediately

CONFIRM  → "No threats found" ✅

"Check if my GPU training is still running"

OBSERVE  → Screenshot → Chrome is open
           └── Identify target: JupyterLab tab

NAVIGATE → Find JupyterLab tab in browser
           ├── OCR tab bar or use bookmarks
           └── Click to switch

EXPLORE  → Multiple terminal tabs visible
           ├── Screenshot terminal area
           ├── LLM vision analysis → identify which tab has nvitop
           └── Click the correct tab

READ     → Screenshot terminal content
           ├── LLM reads GPU utilization table
           └── Report: "8 GPUs, 7 at 100% — experiment running" ✅

"Kill GlobalProtect via Activity Monitor"

OBSERVE  → Screenshot current state
           └── Neither GlobalProtect nor Activity Monitor in foreground

ACT      → Launch both apps
           ├── open -a "GlobalProtect"
           └── open -a "Activity Monitor"

EXPLORE  → Screenshot Activity Monitor window
           ├── LLM vision → "Network tab active, search field empty at top-right"
           └── Decide: click search field first

ACT      → Search for process
           ├── Click search field (identified by explore)
           ├── Paste "GlobalProtect" (clipboard → Cmd+V, never cliclick type)
           └── Wait for filter results

VERIFY   → Process found in list → select it

ACT      → Kill process
           ├── Click stop button (X) in toolbar
           └── Confirmation dialog appears

VERIFY   → Click "Force Quit"

CONFIRM  → Screenshot → process list empty → terminated ✅

🚀 Quick Start

1. Clone & install

git clone https://github.com/Fzkuji/GUIClaw.git
cd GUIClaw
bash scripts/setup.sh

2. Grant accessibility permissions

System Settings → Privacy & Security → Accessibility → Add Terminal / OpenClaw

3. Enable in OpenClaw (recommended)

Add to ~/.openclaw/openclaw.json:

{ "skills": { "entries": { "gui-agent": { "enabled": true } } } }

Then just chat with your agent — it reads SKILL.md and handles everything automatically.

🧠 How It Works

Learn Once, Match Forever

First time — YOLO detects everything (~4 seconds):

🔍 YOLO: 43 icons    📝 OCR: 34 text elements    🔗 → 24 fixed UI components saved

Every time after — instant template match (~0.3 seconds):

✅ search_bar_icon (202,70) conf=1.0
✅ emoji_button (354,530) conf=1.0
✅ sidebar_contacts (85,214) conf=1.0

🔍 Detection Stack

📁 App Visual Memory

Each app gets its own visual memory with a click-graph state model.

memory/apps/
├── wechat/
│   ├── profile.json              # Components + click-graph states
│   ├── components/               # Cropped UI element images
│   │   ├── search_bar.png
│   │   ├── emoji_button.png
│   │   └── ...
│   ├── workflows/                # Saved task sequences
│   │   └── send_message.json
│   └── pages/
│       └── main_annotated.jpg
├── cleanmymac_x/
│   ├── profile.json
│   ├── components/
│   ├── workflows/
│   │   └── smart_scan_cleanup.json
│   └── pages/
├── claude/
│   ├── profile.json
│   ├── components/
│   ├── workflows/
│   │   └── check_usage.json
│   └── pages/
└── google_chrome/
    ├── profile.json
    ├── components/
    └── sites/                    # Per-website memory
        ├── 12306_cn/
        └── github_com/

Click Graph

The UI is modeled as a graph of states. Each state is defined by which components are visible on screen.

profile.json structure:

{
  "app": "Claude",
  "window_size": [1512, 828],
  "components": {
    "Search": { "type": "icon", "rel_x": 115, "rel_y": 143, "icon_file": "components/Search.png", ... },
    "Settings": { ... }
  },
  "states": {
    "initial": {
      "visible": ["Chat_tab", "Cowork_tab", "Code_tab", "Search", "Ideas", ...],
      "description": "Main app view when first opened"
    },
    "click:Settings": {
      "trigger": "Settings",
      "trigger_pos": [63, 523],
      "visible": ["Chat_tab", "Account", "Billing", "Usage", "General", ...],
      "disappeared": ["Ideas", "Customize", ...],
      "description": "Settings page"
    },
    "click:Usage": {
      "trigger": "Usage",
      "visible": ["Chat_tab", "Account", "Billing", "Usage", "Developer", ...],
      "description": "Settings > Usage tab"
    }
  }
}

How it works:

1. Initial state = what's visible when the app first opens (captured during first learn)
2. Click creates state = every click that changes the screen creates a new click:ComponentName state
3. State identification = OCR screen → match visible text against each state's visible list → highest match ratio wins
4. Components belong to states = a component can appear in multiple states (e.g., Chat_tab is visible in initial, click:Settings, click:Usage)
5. Matching is state-specific = only match components that belong to the identified state

Why this works:

• No need to predefine "pages" or "regions" — states are discovered through interaction
• State identification is fast (OCR text matching, no vision model needed)
• Handles overlays, popups, nested