"Computer use" capabilities similar to Anthropic's Claude Computer Use, but fully open-source and extensible.

🎬 See It In Action

Browser Automation

AI agent searching, navigating, and interacting with websites autonomously

▶️ Watch: AI Agent Browsing and Playing

Terminal Operations & Development

Executing commands, managing files, and running complex workflows

▶️ Watch: Quant Trading & Research on QuantConnect

Multi-Agent Orchestration

Complex tasks broken down and executed by specialized agents

▶️ Watch: Building Nvidia Options Dashboard

Advanced Features

Human-in-the-loop control and intelligent collaboration

▶️ Watch: AI Agent with Human Intervention

🎯 Core Capabilities

🌐 Browser Agent

• Search-first strategy using Google Search API
• Smart web navigation with automatic form filling
• Element detection and intelligent clicking
• Multi-tab management for parallel workflows
• Page context extraction for AI understanding
• Screenshot capture for visual verification

💻 Terminal Agent

• Command execution in isolated environments
• File operations (read, write, edit, delete)
• Directory management with full control
• Script execution (Python, Node.js, bash)
• Package installation and environment setup
• Output streaming with real-time feedback

🖱️ Desktop Agent

• UI element detection using computer vision
• Mouse and keyboard control for any application
• Window management (focus, resize, arrange)
• Screenshot analysis for context awareness
• OCR capabilities for text extraction
• Cross-platform support (Linux desktop)

🤖 Multi-Agent System

• Task decomposition by AI planner
• Sequential execution with context passing
• Specialized agents for different capabilities
• Error handling with automatic retries
• User interaction when clarification needed
• Execution reports with detailed summaries

🏗️ Architecture

┌─────────────────────────────────────────────────────────────────┐
│                         Frontend (Next.js 15)                   │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐           │
│  │  Chat UI     │  │  Model       │  │  VM          │           │
│  │  Components  │  │  Selection   │  │  Management  │           │
│  └──────────────┘  └──────────────┘  └──────────────┘           │
└─────────────────────────────────────────────────────────────────┘
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│                      Backend API (FastAPI)                      │
│  ┌──────────────────────────────────────────────────────────┐   │
│  │           Multi-Agent Executor Service                   │   │
│  │  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐       │   │
│  │  │   Planner   │→ │   Browser   │→ │   Terminal  │       │   │
│  │  │    Agent    │  │    Agent    │  │    Agent    │       │   │
│  │  └─────────────┘  └─────────────┘  └─────────────┘       │   │
│  └──────────────────────────────────────────────────────────┘   │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐           │
│  │   WebSocket  │  │   Database   │  │   Billing    │           │
│  │   VM Control │  │   Service    │  │   Service    │           │
│  └──────────────┘  └──────────────┘  └──────────────┘           │
└─────────────────────────────────────────────────────────────────┘
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│               Docker VM (Ubuntu 22.04 + XFCE)                   │
│  ┌──────────────────────────────────────────────────────────┐   │
│  │  Chrome Browser  │  Terminal  │  Desktop Apps  │  Tools  │   │
│  └──────────────────────────────────────────────────────────┘   │
│  ┌──────────────────────────────────────────────────────────┐   │
│  │         WebSocket Agent Server (Port 8080)               │   │
│  │         VNC Server (Port 5900)                           │   │
│  └──────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────┘

🚀 Quick Start

Prerequisites

• Node.js 20+ and npm
• Python 3.10+ and pip
• Docker and Docker Compose
• Supabase account (free tier works)
• API keys for AI providers (OpenAI, Anthropic, etc.)

1. Clone the Repository

git clone https://github.com/LLmHub-dev/open-computer-use.git
cd open-computer-use

2. Set Up Supabase Database

Create Supabase Project

1. Go to Supabase and create a new project
2. Wait for the project to finish setting up
3. Go to Project Settings → API to get your keys

Run Database Schema

Execute the schema to create all required tables:

# Option A: Using Supabase Dashboard
# 1. Go to SQL Editor in your Supabase dashboard
# 2. Copy contents of supabase/schema.sql
# 3. Paste and run the SQL

# Option B: Using Supabase CLI (recommended)
npm install -g supabase
supabase login
supabase link --project-ref your-project-ref
supabase db push

Or manually run the schema file:

psql -h db.your-project.supabase.co -U postgres -d postgres -f supabase/schema.sql

This creates all necessary tables:

• 👤 Users & Auth: users, user_preferences, user_keys
• 💬 Chat System: chats, messages, chat_participants, chat_attachments
• 🤖 AI Agents: machine_sessions, machine_usage, machine_ai_actions
• 💳 Billing: user_credits, credit_transactions, stripe_customers, subscription_plans
• 📊 Projects: projects, user_machines, machine_snapshots

3. Set Up Environment Variables

# Frontend
cp .env.example .env
# Edit .env with your configuration

# Backend
cp backend/.env.example backend/.env
# Edit backend/.env with your configuration

Required Variables

Supabase (Required)

NEXT_PUBLIC_SUPABASE_URL=https://your-project.supabase.co
NEXT_PUBLIC_SUPABASE_ANON_KEY=your-anon-key-from-supabase-dashboard
SUPABASE_SERVICE_ROLE=your-service-role-key-from-supabase-dashboard

Security Keys (Required)

# Generate with: openssl rand -hex 32
ENCRYPTION_KEY=your-generated-32-byte-hex-string
CSRF_SECRET=your-generated-32-byte-hex-string

Google Search API (Required for web search)

GOOGLE_SEARCH_KEY=your-google-api-key
GOOGLE_SEARCH_CX=your-custom-search-engine-id

Get these from Google Cloud Console:

1. Enable Custom Search API
2. Create API key
3. Create Custom Search Engine at programmablesearchengine.google.com

AI Provider Keys (Choose at least one)

# OpenAI
OPENAI_API_KEY=sk-...

# Anthropic
ANTHROPIC_API_KEY=sk-ant-...

# Azure OpenAI (Optional)
AZURE_OPENAI_ENDPOINT=https://your-endpoint.openai.azure.com/
AZURE_OPENAI_API_KEY=your-key
AZURE_OPENAI_DEPLOYMENT=your-deployment-name
AZURE_OPENAI_API_VERSION=2024-02-15-preview

Azure Container Instances (Optional - for cloud VM deployment)

AZURE_SUBSCRIPTION_ID=your