agent.py automatically handles:

• New app → learn → plan
• Known app → eval (template match; ≥80% proceed, <80% re-learn)
• Error → re-learn + new plan
• Chinese aliases: 微信→WeChat, 浏览器→Chrome

Execution Flow

STEP -1: INTENT MATCHING

→ Details: skills/gui-workflow/SKILL.md

Match user request to saved workflows before doing anything. If matched, use workflow steps as plan. If not, proceed and save after success.

STEP 0: OBSERVE

→ Details: skills/gui-observe/SKILL.md

Screenshot, identify current state. Record session_status for token reporting.

STEP 1: ENSURE APP READY

→ Details: skills/gui-learn/SKILL.md

Check if app is in memory. If not → learn. If match rate < 80% → re-learn. This is YOUR responsibility — do not wait for the user.

STEP 2: LEARN (when needed)

→ Details: skills/gui-learn/SKILL.md

Detect all components (YOLO + OCR), identify them, filter, save to memory. Privacy check: delete personal info.

STEP 3: ACT

→ Details: skills/gui-act/SKILL.md

Execute clicks, typing, sending. Pre-verify before every click. Pre-verify contact before every message send.

STEP 4: POST-ACTION VERIFY

→ Details: skills/gui-act/SKILL.md

Screenshot after every action. Did the expected change happen? If not → re-observe.

STEP 5: SAVE WORKFLOW

→ Details: skills/gui-workflow/SKILL.md

Save successful multi-step sequences for future replay.

STEP 6: REPORT

Every GUI task ends with a report:

⏱ 45.2s | 📊 +10k tokens (85k→95k) | 🔧 3 screenshots, 2 clicks, 1 learn

Compare session_status from STEP 0 vs now.

Key Principles

1. Vision-driven, no shortcuts — every GUI interaction goes through the visual pipeline (screenshot → detect → match → click). Do not use system commands (open <url>, osascript tell app to set URL, CLI tools) to manipulate app state. Only allowed system calls: activate (bring window to front), screencapture (take screenshot), cliclick (click/type after visual detection provides coordinates).
2. Memory first, detect second — template match before YOLO+OCR
3. Template > OCR > YOLO > LLM — cheapest method first
4. Relative coordinates — all positions relative to window top-left
5. Window-based, not screen-based — capture and operate within target window only
6. Paste > Type for CJK text and special chars
7. Learn incrementally — save new components after each interaction
8. Integer coordinates only — cliclick requires integers
9. Learn once, match forever — UI positions are stable unless app updates

Safety Rules

These exist because of real bugs:

1. Verify before sending — always OCR the chat header to confirm correct contact
2. Stay within window bounds — get bounds first, filter all OCR/clicks
3. No wrong-app learning — validate window bounds before auto_learn
4. Reject tiny templates — <30×30 pixels produce false matches
5. LLM never provides coordinates — detection tools provide coordinates, you decide what to click by name
6. Never send screenshots to conversation — internal detection only

Memory System

→ Details: skills/gui-memory/SKILL.md

Visual memory stores app profiles, components, page fingerprints, workflows. See gui-memory for directory structure, profile schema, CRUD operations, and cleanup rules.

File Structure

gui-agent/
├── SKILL.md              # This file (main orchestrator)
├── skills/               # Sub-skills (read on demand)
│   ├── gui-observe/SKILL.md
│   ├── gui-learn/SKILL.md
│   ├── gui-act/SKILL.md
│   ├── gui-memory/SKILL.md
│   ├── gui-workflow/SKILL.md
│   └── gui-setup/SKILL.md
├── scripts/              # Core scripts
│   ├── agent.py, ui_detector.py, app_memory.py, gui_agent.py, template_match.py
├── memory/               # Visual memory (gitignored)
│   ├── apps/<appname>/
│   └── meta_workflows/
├── actions/              # Atomic operations
├── docs/
└── README.md

💬 What It Looks Like

You: "Send a message to John in WeChat saying see you tomorrow"

OBSERVE  → Screenshot, identify current state
           ├── Current app: Finder (not WeChat)
           └── Action: need to switch to WeChat

STATE    → Check WeChat memory
           ├── Learned before? Yes (24 components)
           ├── OCR visible text: ["Chat", "Cowork", "Code", "Search", ...]
           ├── State identified: "initial" (89% match)
           └── Components for this state: 18 → use these for matching

NAVIGATE → Find contact "John"
           ├── Template match search_bar → found (conf=0.96) → click
           ├── Paste "John" into search field (clipboard → Cmd+V)
           ├── OCR search results → found → click
           └── New state: "click:John" (chat opened)

VERIFY   → Confirm correct chat opened
           ├── OCR chat header → "John" ✅
           └── Wrong contact? → ABORT

ACT      → Send messag...