Open Computer Use

AI agents that control computers like humans do.

Browser automation · Terminal access · Desktop control · Multi-agent orchestration

Website · Discord · Twitter

See it in action

What is this?

Open Computer Use is an open-source platform that gives AI agents real computer control. Unlike chatbots that only talk about tasks, agents here actually perform them — browsing the web, running commands, clicking through UIs, and orchestrating multi-step workflows in isolated containers.

Computer use capabilities similar to Anthropic's Claude Computer Use, but fully open-source and extensible.

Agents

Browser — Search-first web navigation, form filling, element interaction, multi-tab management, screenshot capture.

Terminal — Command execution, file operations, script running, package management, output streaming.

Desktop — Mouse & keyboard control, window management, screenshot analysis, UI element detection via computer vision.

Planner — Decomposes complex requests into subtasks, assigns to specialized agents, passes context between steps.

Architecture

Frontend (Next.js 15)         Backend (FastAPI)              VM (Docker)
┌──────────────────┐     ┌─────────────────────────┐     ┌──────────────────┐
│  Chat UI         │────▶│  Multi-Agent Executor    │────▶│  Chrome Browser  │
│  Model Selector  │ SSE │  ├─ Planner Agent        │ WS  │  Terminal        │
│  VM Management   │◀────│  ├─ Browser Agent        │◀────│  Desktop (XFCE)  │
│  Zustand Stores  │     │  ├─ Terminal Agent       │     │  Agent Server    │
└──────────────────┘     │  └─ Desktop Agent        │     │  VNC :5900       │
                         │  WebSocket · DB · Billing│     └──────────────────┘
                         └─────────────────────────┘

Quick Start

Prerequisites

Node.js 20+ · Python 3.10+ · Docker · Supabase account · AI provider API key

1. Clone & install

git clone https://github.com/coasty-ai/open-computer-use.git
cd open-computer-use

# Frontend
npm install

# Backend
cd backend
python -m venv venv
source venv/bin/activate   # Windows: venv\Scripts\activate
pip install -r requirements.txt
cd ..

2. Configure environment

cp .env.example .env
cp backend/.env.example backend/.env

Set these in both .env files:

# Supabase (required)
NEXT_PUBLIC_SUPABASE_URL=https://your-project.supabase.co
NEXT_PUBLIC_SUPABASE_ANON_KEY=your-anon-key
SUPABASE_SERVICE_ROLE=your-service-role-key

# Security (required — generate with: openssl rand -hex 32)
ENCRYPTION_KEY=...
CSRF_SECRET=...

# AI provider (at least one)
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...

# Google Search (required for web search)
GOOGLE_SEARCH_KEY=...
GOOGLE_SEARCH_CX=...

3. Set up database

# Via Supabase CLI
npm install -g supabase
supabase login
supabase link --project-ref your-project-ref
supabase db push

# Or paste supabase/schema.sql into the Supabase SQL Editor

4. Run

Docker (recommended):

docker-compose up --build

Manual:

# Terminal 1 — Frontend
npm run dev

# Terminal 2 — Backend
cd backend && python main.py

# Terminal 3 — AI Desktop VM (optional)
docker-compose -f docker-compose.ai-desktop.yml up --build

Open http://localhost:3000, sign in, start a chat, and give your agent a task.

Tech Stack

| Layer | Technologies | | --- | --- | | Frontend | Next.js 15, React 19, TypeScript, Tailwind CSS, Radix UI, Zustand, Vercel AI SDK | | Backend | FastAPI, Python 3.10+, WebSockets, asyncio, uvicorn | | AI Providers | OpenAI, Anthropic, Google, Azure, xAI, Mistral, Perplexity, OpenRouter | | Infrastructure | Docker, Ubuntu 22.04 + XFCE, Chrome, Selenium, Supabase, Stripe | | Desktop App | Electron 40, Puppeteer-core, platform-native automation (Win32 / CoreGraphics / xdotool) |

Electron Desktop App

A lightweight overlay that runs AI agent commands directly on your local machine instead of a remote VM.

• Floating always-on-top pill UI with expanded chat panel
• Platform-native automation (PowerShell/Win32 on Windows, CoreGraphics/osascript on macOS, xdotool on Linux)
• Browser control via Puppeteer-core, shell execution, file operations
• WebSocket bridge to backend with auto-reconnect

cd electron
npm install
npm run dev        # Development with hot reload
npm run package    # Build for current platform

Project Structure

├── app/                    # Next.js routes & pages
├── components/             # React components (UI, chat, prompts)
├── lib/                    # Stores, providers, services, utilities
├── backend/
│   └── app/
│       ├── api/routes/     # FastAPI endpoints
│       ├── services/       # Multi-agent executor, VM control, billing
│       ├── providers/      # AI provider integrations
│       └── core/           # Config, middleware, logging
├── electron/
│   └── src/
│       ├── main/           # App lifecycle, IPC, automation modules
│       ├── preload/        # Context bridge API
│       └── renderer/       # React UI, stores, components
├── docker/ai-desktop/      # Ubuntu VM container
└── supabase/               # Database schema

Contributing

1. Fork the repo
2. Create a branch: git checkout -b feature/your-feature
3. Commit your changes
4. Open a pull request

Bug reports and feature requests welcome in Issues.

Roadmap

• [ ] Multi-VM parallel orchestration
• [ ] Visual workflow builder
• [ ] Agent marketplace & templates
• [ ] Windows / macOS VM support
• [ ] Plugin system for custom tools
• [ ] Collaborative sessions
• [ ] Voice control & video understanding

Responsible Use

This platform gives AI agents significant autonomy. Use it to automate repetitive tasks, testing, research, and content creation — not to violate terms of service, spam, or scrape without permission. Always use isolated environments, respect robots.txt, and follow data protection laws.

License

Apache License 2.0 — Copyright (c) 2025 Open Computer Use Contributors

Star on GitHub · Join Discord · Follow on X