🎨 Visual Design

• Glass Morphism: Beautiful blur effects and transparency
• Adaptive Layout: UI adjusts based on available services
• Smart Resizing: Windows resize automatically to fit content
• Professional Look: Mimics system applications for perfect stealth

🎯 Functional Overview

📋 Core Components

🖱️ Main Overlay

• Floating command bar
• Screenshot capture (⌘⇧S)
• Microphone toggle (Optional)
• Skill selector (DSA)
• Language picker
• Status indicator

💬 Interactive Chat

• Real-time transcription
• AI conversation
• Markdown formatting
• Session memory
• Listening animations
• Auto-scroll messages

📊 Answer Window

• Draggable interface
• Close button (×)
• Split layout for code
• Full markdown support
• Syntax highlighting
• Smart content sizing

✅ To-Do List & Development Status

🎯 Core Features (Completed)

• [x] Stealth overlay with draggable command bar and click‑through toggle
• [x] Screenshot capture with direct Gemini analysis (no OCR step)
• [x] AI response window with markdown and code highlighting
• [x] Global shortcuts (capture, visibility, interaction, chat, settings)
• [x] Session memory and chat UI
• [x] Language picker and DSA skill prompt
• [x] Optional Azure Speech / local Whisper integration with auto‑hide mic
• [x] Multi‑monitor and area capture APIs
• [x] Window binding and positioning system
• [x] Settings management with app icon/stealth modes

🚧 Planned Features (In Development)

• [ ] Hidden during screen share (auto‑hide all windows while screen is being shared)
• [ ] Multi‑model support (OpenAI/Anthropic/Local backends alongside Gemini)
• [ ] Auto‑typer for code snippets (paste or simulate typing into editors/IDEs)
• [ ] Export conversation history (save sessions as markdown/PDF)
• [ ] Performance optimizations (faster startup, reduced memory usage)
• [ ] Enhanced stealth modes (process name randomization, deeper OS integration)

⚙️ Configuration

The setup script automatically handles configuration. You only need:

# Required: Google Gemini API Key (setup script will ask for this)
GEMINI_API_KEY=your_gemini_api_key_here

# Optional: Speech Recognition (pick one provider)
SPEECH_PROVIDER=whisper

# Azure option
AZURE_SPEECH_KEY=your_azure_speech_key
AZURE_SPEECH_REGION=your_region

# Local Whisper option
WHISPER_COMMAND=whisper
WHISPER_MODEL_DIR=.whisper-models
WHISPER_MODEL=base
WHISPER_LANGUAGE=en
WHISPER_SEGMENT_MS=4000

Note: Speech recognition is completely optional. If no configured provider is available, the microphone button will be automatically hidden from all interfaces.

🚀 Quick Start & Installation

⚡ Three Simple Steps (All Operating Systems)

1. Clone the repository

   git clone https://github.com/TechyCSR/OpenCluely.git
   cd OpenCluely
   

2. Get your Gemini API key (Required)

   • Visit Google AI Studio
   • Click "Create API Key"
   • Copy the key (you'll need it in step 3)

3. Run the setup script (One command does everything!)

   ./setup.sh
   

That's it! The setup script will:

• Install all dependencies automatically
• Create your .env file from env.example if needed
• Set up a local Whisper virtualenv in .venv-whisper
• Configure .env to use local Whisper by default
• Build the app (if needed)
• Launch OpenCluely ready to use (if not works use npm install & then npm start)

💻 Platform-Specific Notes

• Windows: Use Git Bash (comes with Git for Windows), WSL, or any bash environment
• macOS/Linux: Use your regular terminal
• All platforms: No manual npm commands needed - the setup script handles everything
• Windows Whisper path: setup.sh now writes WHISPER_COMMAND=.venv-whisper/Scripts/whisper.exe
• macOS/Linux Whisper path: setup.sh writes WHISPER_COMMAND=.venv-whisper/bin/whisper

🎛️ Setup Script Options

./setup.sh --build          # Build distributable for your OS
./setup.sh --ci             # Use npm ci instead of npm install
./setup.sh --no-run         # Setup only, don't launch the app
./setup.sh --install-system-deps  # Install sox for microphone (optional)
./setup.sh --skip-whisper  # Skip the local Whisper bootstrap

🔧 Optional: Speech Setup (For Voice Features)

Voice recognition is optional. You can use either Azure Speech or local OpenAI Whisper.

For the local Whisper path, ./setup.sh now handles the full repo-local setup:

1. Creates .venv-whisper

2. Installs openai-whisper

3. Points .env at .venv-whisper/bin/whisper

4. Creates .whisper-models

5. Runs npm run test-speech

6. For Azure Speech:

   • Visit Azure Portal
   • Create a Speech Service
   • Copy your key and region

7. For local Whisper:

   • Run ./setup.sh --install-system-deps
   • Or install required audio tools such as ffmpeg and sox yourself
   • On Windows, install audio tooling separately and prefer Git Bash or WSL for setup.sh

8. Add one provider to your .env file:

   GEMINI_API_KEY=your_gemini_api_key_here
   SPEECH_PROVIDER=azure
   AZURE_SPEECH_KEY=your_azure_speech_key
   AZURE_SPEECH_REGION=your_region
   

   GEMINI_API_KEY=your_gemini_api_key_here
   SPEECH_PROVIDER=whisper
   WHISPER_COMMAND=whisper
   WHISPER_MODEL_DIR=.whisper-models
   WHISPER_MODEL=base
   WHISPER_LANGUAGE=en
   WHISPER_SEGMENT_MS=4000
   

9. Restart the app - microphone buttons will now appear automatically

🎮 How to Use

🖱️ Main Controls

| Action | Shortcut | Description | |--------|----------|-------------| | Screenshot Capture | ⌘⇧S | Capture screen and analyze via Gemini (image understanding) | | Toggle Speech | Alt+R | Start/stop voice recognition (if configured) | | Toggle Visibility | ⌘⇧V | Show/hide all windows | | Toggle Interaction | ⌘⇧I or Alt+A | Enable/disable window interaction | | Switch to Chat | ⌘⇧C | Open interactive chat window | | Settings | ⌘, | Open settings panel |

🎯 Workflow

1. Start OpenCluely → App appears as system process (Terminal/Activity Monitor)
2. Position Windows → Drag overlay and answer windows to preferred locations
3. Capture Questions → Use screenshot (⌘⇧S) or voice commands
4. Get AI Answers → Instant responses in draggable answer window
5. Interactive Chat → Type or speak for detailed conversations
6. Stay Stealth → All operations invisible to screen recording

🔧 Advanced Features

🎨 Window Management

• Draggable Interface: Click and drag any window to reposition
• Auto-resize: Windows automatically adjust to content
• Close Button: Click × to close answer window
• Always on Top: Windows stay above all applications

🧠 AI Intelligence

• Context Awareness: Remembers entire conversation
• **Code Dete