agent.py automatically handles:

• New app → learn → plan
• Known app → eval (template match; ≥80% proceed, <80% re-learn)
• Error → re-learn + new plan
• Chinese aliases: 微信→WeChat, 浏览器→Chrome

Execution Flow

STEP -1: INTENT MATCHING

→ Details: skills/gui-workflow/SKILL.md

Match user request to saved workflows before doing anything. If matched, use workflow steps as plan. If not, proceed and save after success.

STEP 0: OBSERVE

→ Details: skills/gui-observe/SKILL.md

Screenshot, identify current state. Record session_status for token reporting.

STEP 1: ENSURE APP READY

→ Details: skills/gui-learn/SKILL.md

Check if app is in memory. If not → learn. If match rate < 80% → re-learn. This is YOUR responsibility — do not wait for the user.

STEP 2: LEARN (when needed)

→ Details: skills/gui-learn/SKILL.md

Detect all components (YOLO + OCR), identify them, filter, save to memory. Privacy check: delete personal info.

STEP 3: ACT

→ Details: skills/gui-act/SKILL.md

Execute clicks, typing, sending. Pre-verify before every click. Pre-verify contact before every message send.

STEP 4: POST-ACTION VERIFY

→ Details: skills/gui-act/SKILL.md

Screenshot after every action. Did the expected change happen? If not → re-observe.

STEP 5: SAVE WORKFLOW

→ Details: skills/gui-workflow/SKILL.md

Save successful multi-step sequences for future replay.

STEP 6: REPORT

Every GUI task ends with a report:

⏱ 45.2s | 📊 +10k tokens (85k→95k) | 🔧 3 screenshots, 2 clicks, 1 learn

Compare session_status from STEP 0 vs now.

Key Principles

1. Vision-driven, no shortcuts — every GUI interaction goes through the visual pipeline (screenshot → detect → match → click). Do not use system commands (open <url>, osascript tell app to set URL, CLI tools) to manipulate app state. Only allowed system calls: activate (bring window to front), screencapture (take screenshot), cliclick (click/type after visual detection provides coordinates).
2. Memory first, detect second — template match before YOLO+OCR
3. Template > OCR > YOLO > LLM — cheapest method first
4. Relative coordinates — all positions relative to window top-left
5. Window-based, not screen-based — capture and operate within target window only
6. Paste > Type for CJK text and special chars
7. Learn incrementally — save new components after each interaction
8. Integer coordinates only — cliclick requires integers
9. Learn once, match forever — UI positions are stable unless app updates

Safety Rules

These exist because of real bugs:

1. Verify before sending — always OCR the chat header to confirm correct contact
2. Stay within window bounds — get bounds first, filter all OCR/clicks
3. No wrong-app learning — validate window bounds before auto_learn
4. Reject tiny templates — <30×30 pixels produce false matches
5. LLM never provides coordinates — detection tools provide coordinates, you decide what to click by name
6. Never send screenshots to conversation — internal detection only

Memory System

→ Details: skills/gui-memory/SKILL.md

Visual memory stores app profiles, components, page fingerprints, workflows. See gui-memory for directory structure, profile schema, CRUD operations, and cleanup rules.

File Structure

gui-agent/
├── SKILL.md              # This file (main orchestrator)
├── skills/               # Sub-skills (read on demand)
│   ├── gui-observe/SKILL.md
│   ├── gui-learn/SKILL.md
│   ├── gui-act/SKILL.md
│   ├── gui-memory/SKILL.md
│   ├── gui-workflow/SKILL.md
│   └── gui-setup/SKILL.md
├── scripts/              # Core scripts
│   ├── agent.py, ui_detector.py, app_memory.py, gui_agent.py, template_match.py
├── memory/               # Visual memory (gitignored)
│   ├── apps/<appname>/
│   └── meta_workflows/
├── actions/              # Atomic operations
├── docs/
└── README.md

"Scan my Mac for malware"

OBSERVE  → Screenshot → CleanMyMac X not in foreground → activate
           ├── Get main window bounds (largest window, skip status bar panels)
           └── OCR window content → identify current state

STATE    → Check memory for CleanMyMac X
           ├── OCR visible text: ["Smart Scan", "Malware Removal", "Privacy", ...]
           ├── State identified: "initial" (92% match)
           └── Know which components to match: 21 components

NAVIGATE → Click "Malware Removal" in sidebar
           ├── Find element in window (exact match, filter by window bounds)
           ├── Click → new state: "click:Malware_Removal"
           └── OCR confirms new state (87% match)

ACT      → Click "Scan" button
           ├── Find "Scan" (exact match, bottom position — prevents matching "Deep Scan")
           └── Click → scan starts

POLL     → Wait for completion (event-driven, no fixed sleep)
           ├── Every 2s: screenshot → OCR check for "No threats"
           └── Target found → proceed immediately

CONFIRM  → "No threats found" ✅

"Check if my GPU training is still running"

OBSERVE  → Screenshot → Chrome is open
           └── Identify target: JupyterLab tab

NAVIGATE → Find JupyterLab tab in browser
           ├── OCR tab bar or use bookmarks
           └── Click to switch

EXPLORE  → Multiple terminal tabs visible
           ├── Screenshot terminal area
           ├── LLM vision analysis → identify which tab has nvitop
           └── Click the correct tab

READ     → Screenshot terminal content
           ├── LLM reads GPU utilization table
           └── Report: "8 GPUs, 7 at 100% — experiment running" ✅

"Kill GlobalProtect via Activity Monitor"

OBSERVE  → Screenshot current state
           └── Neither GlobalProtect nor Activity Monitor in foreground

ACT      → Launch both apps
           ├── open -a "GlobalProtect"
           └── open -a "Activity Monitor"

EXPLORE  → Screenshot Activity Monitor window
           ├── LLM vision → "Network tab active, search field empty at top-right"
           └── Decide: click search field first

ACT      → Search for process
           ├── Click search field (identified by explore)
           ├── Paste "GlobalProtect" (clipboard → Cmd+V, never cliclick type)
           └── Wait for filter results

VERIFY   → Process found in list → select it

ACT      → Kill process
           ├── Click stop button (X) in toolbar
           └── Confirmation dialog appears

VERIFY   → Click "Force Quit"

CONFIRM  → Screenshot → process list empty → terminated ✅

🚀 Quick Start

1. Clone & install

git clone https://github.com/Fzkuji/GUIClaw.git
cd GUIClaw
bash scripts/setup.sh

2. Grant accessibility permissions

System Settings → Privacy & Security → Accessibility → Add Terminal / OpenClaw

3. Enable in OpenClaw (recommended)

Add to ~/.openclaw/openclaw.json:

{ "skills": { "entries": { "gui-agent": { "enabled": true } } } }

Then just chat with your agent — it reads SKILL.md and handles everything automatically.

🧠 How It Works

Learn Once, Match Forever

First time — YOLO detects everything (~4 seconds):

🔍 YOLO: 43 icons    📝 OCR: 34 text elements    🔗 → 24 fixed UI components saved

Every time after — instant template match (~0.3 seconds):

✅ search_bar_icon (202,70) conf=1.0
✅ emoji_button (354,530) conf=1.0
✅ sidebar_contacts (85,214) conf=1.0

🔍 Detection Stack

| Detector | Speed | Finds | Why | |----------|-------|-------|-----| | GPA-GUI-Detector | 0.3s | Icons, buttons | Finds gray-on-gray icons others miss | | Apple Vision OCR | 1.6s | Text (CN + EN) | Best Chinese OCR, pixel-accurate | | Template Match | 0.3s | Known components | 100% accuracy after first learn |

📁 App Visual Memory

Each app gets its own visual memory with a click-graph state model.

memory/apps/
├── wechat/
│   ├── profile.json              # Components + click-graph states
│   ├── components/               # Cropped UI element images
│   │   ├── search_bar.png
│   │   ├── emoji_button.png
│   │   └── ...
│   ├── workflows/                # Saved task sequences
│   │   └── send_message.json
│   └── pages/
│       └── main_annotated.jpg
├── cleanmymac_x/
│   ├── profile.json
│   ├── components/
│   ├── workflows/
│   │   └── smart_scan_cleanup.json
│   └── pages/
├── claude/
│   ├── profile.json
│   ├── components/
│   ├── workflows/
│   │   └── check_usage.json
│   └── pages/
└── google_chrome/
    ├── profile.json
    ├── components/
    └── sites/                    # Per-website memory
        ├── 12306_cn/
        └── github_com/

Click Graph

The UI is modeled as a graph of states. Each state is defined by which components are visible on screen.

profile.json structure:

{
  "app": "Claude",
  "window_size": [1512, 828],
  "components": {
    "Search": { "type": "icon", "rel_x": 115, "rel_y": 143, "icon_file": "components/Search.png", ... },
    "Settings": { ... }
  },
  "states": {
    "initial": {
      "visible": ["Chat_tab", "Cowork_tab", "Code_tab", "Search", "Ideas", ...],
      "description": "Main app view when first opened"
    },
    "click:Settings": {
      "trigger": "Settings",
      "trigger_pos": [63, 523],
      "visible": ["Chat_tab", "Account", "Billing", "Usage", "General", ...],
      "disappeared": ["Ideas", "Customize", ...],
      "description": "Settings page"
    },
    "click:Usage": {
      "trigger": "Usage",
      "visible": ["Chat_tab", "Account", "Billing", "Usage", "Developer", ...],
      "description": "Settings > Usage tab"
    }
  }
}

How it works:

1. Initial state = what's visible when the app first opens (captured during first learn)
2. Click creates state = every click that changes the screen creates a new click:ComponentName state
3. State identification = OCR screen → match visible text against each state's visible list → highest match ratio wins
4. Components belong to states = a component can appear in multiple states (e.g., Chat_tab is visible in initial, click:Settings, click:Usage)
5. Matching is state-specific = only match components that belong to the identified state

Why this works:

• No need to predefine "pages" or "regions" — states are discovered through interaction
• State identification is fast (OCR text matching, no vision model needed)
• Handles overlays, popups, nested