Ghostwork

The first agent you don't prompt.

Ghostwork runs silently in the background, learns how you work from observation alone, and gradually takes repetitive tasks off your hands — without you ever writing a prompt or setting up an integration.

What it does

Ghostwork watches every interaction across every app on your Mac (via Screenpipe), builds a rich memory of your work patterns, and surfaces automations the moment they're relevant.

Memory layers

Nightly consolidation (sleep cycle)

Every night Ghostwork runs a 3-phase consolidation:

1. NREM — LLM analyses unsummarised sessions and promotes patterns to rules
2. REM — Rules with 3+ observations that have browser-recorded events are compiled into executable skills
3. GC — Power-law confidence decay, dedup, 90-day prune of raw events, behaviour.md rewrite

The living behaviour.md profile is injected into every LLM prompt, so the trigger decision has full context about who you are and what you do.

Autonomy tiers

Ghostwork earns autonomy, never assumes it:

Actions that are externally visible (send email, submit form, post) always require one-tap approval regardless of tier.

Execution stack

1. Compiled skill replay — zero-token deterministic replay of recorded step sequences
2. AX-first native control — ax_list_elements + ax_click_element via macOS Accessibility API; ~95% accuracy on native apps
3. Claude vision fallback — pixel-level screenshots + function calling for browsers and AX-empty apps

When a multi-step sequence fails mid-way, Ghostwork fires Cmd+Z for each completed reversible step in reverse order before surfacing the error.

UI

• Menu bar icon — reflects current state: observing / working
• Activity feed — chronological log of every action taken and approval waiting
• Timeline tab — drill down into every recorded session; save any session as a skill
• Behaviour tab — learned rules with autonomy progress, category badges, and one-click boost
• Approvals — staged actions waiting for your confirmation before executing externally visible steps

Setup

Prerequisites

• macOS 12+
• Screenpipe installed and running (screenpipe)
• Node.js 20+ and npm
• An OpenRouter API key (or Anthropic key)

Install

git clone https://github.com/your-org/ghostwork
cd ghostwork
npm install
npx @electron/rebuild -f -w better-sqlite3

Configure

cp .env.example .env

Edit .env:

OPENROUTER_API_KEY=sk-or-...
ANTHROPIC_API_KEY=sk-ant-...   # optional, used for native computer use

Run

npm start          # development (hot reload)
npm run build      # production build
npm run dist       # package as .dmg

Architecture

Screenpipe ──► sessionIngester (2min) ──► raw_events ──► extractor (30min) ──► rules
                                                                                   │
                                                                          NREM/REM nightly
                                                                                   │
                                                                              compiled skills
                                                                                   │
                              actionEngine (10s) ──► LLM trigger ──► AX-first executor
                                                                    └──► vision fallback

Key files:

Contributing

See CONTRIBUTING.md for setup instructions and coding guidelines.

Open issues on GitHub — look for good first issue labels. Join the Discord to discuss ideas or get help getting set up.

Privacy

• All data stays on your device. No cloud sync, no telemetry.
• Ghostwork excludes itself, Cursor, and any app you add to the exclusion list.
• Raw events are pruned after 90 days.
• PII (emails, phone numbers, card numbers) is stripped before any LLM call.
• The behaviour.md profile never leaves your machine.

Roadmap

• Full autopilot mode: skill execution without any prompt
• Cross-session pattern detection in REM phase
• Browser extension for richer DOM locators in user's main Chrome profile
• Windows support (via Screenpipe Windows builds)
• Team profiles (opt-in, anonymised)

Licence

GPL-3.0 — see LICENSE for details.