GhostDesk

Table of contents

• Why GhostDesk?
• How it works
• Quick start
• Secure local run (TLS + auth)
• Tools
• Model requirements
• From one agent to a workforce
• Configuration
• Security
• Troubleshooting
• Custom image
• License

Why GhostDesk?

Browser automation tools (Playwright, Puppeteer, Selenium…) were built for human test engineers driving a browser with selectors. They do one thing, and they do it well — inside the browser.

GhostDesk is built from the other end: for AI agents, driving everything a desktop runs. Browsers, native apps, IDEs, terminals, office suites, legacy software, internal tools. If it renders pixels on screen, your agent can see it and use it — in one conversation, across many applications, without a line of glue code.

You don't write selectors. You write a prompt:

"Open the CRM, export last month's leads as CSV, open LibreOffice Calc, build a pivot table, screenshot the chart, and email it to the team."

The agent opens the browser, logs in, downloads the file, switches to LibreOffice, processes the data, captures the result, composes the email, sends it. One prompt, multiple apps, fully autonomous — no glue code, no per-site scraper, no brittle selector chain.

That is what agents using a desktop looks like.

Runs on models you can actually host

Desktop control needs to be fast — an agent that takes twelve seconds to decide where to click is unusable. GhostDesk is tuned so that vision-language models from the Qwen family running on a single workstation GPU are a first-class target, not an afterthought. No API bill, no screenshots of your desktop leaving your network.

Frontier models (Claude, GPT-4o, Gemini) work too and remain the smoothest path — but they are not the bar. See Model requirements for the supported stacks and the one coordinate-space setting that matters.

How it works

GhostDesk runs a virtual Linux desktop inside Docker and exposes it as an MCP server. Your agent gets a sandboxed desktop with a taskbar, clock, and pre-installed applications — equivalent to what a human sees on their screen.

The agent perceives the screen by calling screen_shot(), which captures the full desktop at native resolution and returns it as WebP (or PNG). An optional region= argument can crop to a sub-rectangle when the agent explicitly wants to narrow its focus.

This works with any application — web apps, native apps, legacy software, Canvas, WebGL.

Quick start

1. Run the container

One command, plain HTTP, no password. Fine for kicking the tires on a laptop you trust — not fit for anything beyond that. Ready to harden it? Jump to Secure local run.

docker run -d --name ghostdesk-demo \
  --shm-size 2g \
  -p 3000:3000 \
  -p 6080:6080 \
  ghcr.io/yv17labs/ghostdesk:latest

The latest image ships with Firefox, the foot terminal, mousepad (text editor), galculator, and passwordless sudo for the agent user — enough to demo a browsing + note-taking workflow out of the box. Need a different app set? Build your own on top of base — see Custom image.

The container boots in the dev posture: plain HTTP on both ports, every auth gate disarmed on purpose. You'll see warnings in the logs reminding you of that — they go away once you follow the secured path below.

2. Connect your AI

GhostDesk speaks MCP over the Streamable HTTP transport — any MCP-compatible client can drive it. Point your client at http://localhost:3000/mcp:

Claude Desktop / Claude Code

{
  "mcpServers": {
    "ghostdesk": {
      "type": "http",
      "url": "http://localhost:3000/mcp"
    }
  }
}

Any other MCP-compatible client — same URL, no headers, no auth. That's the whole demo posture.

3. Watch your agent work

Open http://localhost:6080/ in your browser to see the virtual desktop in real time. No password prompt — the dev posture skips it.

| Service | URL | |---------|-----| | MCP server | http://localhost:3000/mcp | | noVNC (browser) | http://localhost:6080/ |

Give your agent a first prompt to confirm the wiring is right:

"Take a screenshot of the desktop, list the installed applications, then open Firefox and go to wikipedia.org."

You should see Firefox launch in the noVNC tab, the URL bar fill in, and the page load — all under your agent's control.

4. When you're done

docker stop ghostdesk-demo && docker rm ghostdesk-demo

The demo run creates no named volume, so this leaves nothing behind.

Secure local run (TLS + auth)

The Quick start above drops every gate so you can kick the tires in thirty seconds. The moment you want to expose this to anything beyond your own laptop — another machine on your LAN, a devcontainer port-forward on an untrusted network, a teammate's browser — flip to the secured posture: real TLS + bearer-token auth on MCP + password prompt on noVNC.

GhostDesk couples TLS and auth: mount a cert and you get wss:// + bearer-token on MCP + a single-password prompt on noVNC (see Security → Auth ≡ TLS). mkcert issues a browser-trusted cert for localhost in two commands:

# Issue a locally-trusted cert (first time only — installs a local CA in your trust store)
mkcert -install
mkdir -p tls
mkcert -cert-file tls/server.crt -key-file tls/server.key localhost 127.0.0.1 ::1

# Generate the MCP and VNC secrets
export GHOSTDESK_AUTH_TOKEN=$(openssl rand -hex 32)
export GHOSTDESK_VNC_PASSWORD=$(openssl rand -hex 16)

Pick a container name that matches the agent's role — sales-agent, research-agent, accounting-agent… Below we use my-agent as a placeholder; replace it everywhere in the command.

# Run the container — cert mounted, TLS + auth enabled everywhere
docker run -d --name ghostdesk-my-agent \
  --restart unless-stopped \
  --cap-add SYS_ADMIN \
  --shm-size 2g \
  -p 3000:3000 \
  -p 6080:6080 \
  -v ghostdesk-my-agent-home:/home/agent \
  -v "$PWD/tls/server.crt:/etc/ghostdesk/tls/server.crt:ro" \
  -v "$PWD/tls/server.key:/etc/ghostdesk/tls/server.key:ro" \
  -e GHOSTDESK_AUTH_TOKEN \
  -e GHOSTDESK_VNC_PASSWORD \
  -e TZ=America/New_York \
  -e LANG=en_US.UTF-8 \
  ghcr.io/yv17labs/ghostdesk:latest

echo "MCP token:    $GHOSTDESK_AUTH_TOKEN"
echo "VNC password: $GHOSTDESK_VNC_PASSWORD"

Once the container is up, update your MCP client config — same shape as the demo, now over https:// with a bearer token:

Claude Desktop / Claude Code

{
  "mcpServers": {
    "ghostdesk": {
      "type": "http",
      "url": "https://localhost:3000/mcp",
      "headers": {
        "Authorization": "Bearer <paste $GHOSTDESK_AUTH_TOKEN here>"
      }
    }
  }
}

Any other MCP-compatible client — same URL, plus an Authorization: Bearer <token> header in whatever form your client accepts.

Then open https://localhost:6080/ in your browser — the mkcert CA installed by mkcert -install is already in your trust store, so the browser accepts the cert with no warning. noVNC will prompt for $GHOSTDESK_VNC_PASSWORD.

Going to production? Swap the mkcert leaf for a real cert, source both secrets from your secret manager, and front port 6080 with an identity-aware proxy — SECURITY.md has the full contract.

--cap-add SYS_ADMIN — Required by Electron apps (VS Code, Slack, etc.) and other applications that need Linux user namespaces to run their sandbox. Safe to remove if you don't need them.

The named volume persists the agent's home directory across restarts — browser passwords, bookmarks, cookies, downloads, and desktop preferences are all preserved. On the first run, Docker automatically seeds the volume with the default configuration from the image.

Tools

13 tools at your agent's fingertips, grouped by concern (verb_noun naming):

Screen

| Tool | Description | |------|-------------| | screen_shot | Capture the screen as a WebP image (pass format="png" for lossless). Pass region= to crop to a sub-rectangle at native resoluti