Crayotter

Crayotter is a multimodal, agent-driven video editing system that turns a single text request into a complete edited video. It combines planning, deep editing research, and tool-based execution into a three-phase workflow, with full logs and visual trace analysis for debugging and iteration.

📽️ Demo

A full end-to-end run: from a one-line text request, Crayotter prepares materials, researches an editing blueprint, and produces the final edited video.

Table of Contents

• News
• Highlights
• Overview
• Workflow
• Quick Start
  • Windows Standalone Release
  • Prerequisites
  • 1) Environment
  • 2) Install Dependencies
  • 3) Configure API Endpoints and Runtime Options
  • 4) Run the Agent
  • 5) Run the Workbench GUI
• Log Trace Visualization
• Citation
• Repository Layout

News

• 2026.6.27 — Crayotter 1.0.0 adds multi-source material import, unified download cleanup, and refreshed Windows release packaging.
• 2026.6.15 — Asynchronous scheduling upgrade accelerated the overall video generation workflow by ~1.6×.
• 2026.5.31 — Our paper is now available: Crayotter: Traceable Multi-Agent Workflows for Long-Form Video Editing.
• 2026.5.23 — 100 Stars achieved!
• 2026.5.11 — The paper page is now live at Crayotter Paper Page.
• 2026.4.10 — The release has been updated.
• 2026.3.30 — First release version available. See v0.1.0-demo.

Highlights

• One request to one video — A single natural-language prompt drives the whole pipeline from sourcing to export.
• Three-phase, traceable workflow — Explicit planning, pure-reasoning editing research, and controlled tool execution, all logged for replay.
• Multimodal material understanding — Each source video is analyzed by a multimodal model to inform narrative and pacing.
• Resource-aware DAG scheduling — Bounded concurrency for search, download, analysis, LLM, FFmpeg, TTS, and export, with retries, conflict keys, and checkpoint/resume.
• Multi-source material import — Bilibili, Douyin, Xiaohongshu/Rednote, Kuaishou, YouTube, or generic URLs (via yt-dlp).
• Built-in narration, subtitles, and audio mixing — Segmented TTS, loudness normalization, background-ducking, and subtitle burn-in.
• Local workbench + desktop mode — Web UI for task management, .env-synced config, structured logs, artifact preview, and interrupted-job resume.
• Visual trace analysis — Local server + static HTML export for inspecting phase progress and tool-call trajectories.

Overview

This repository centers around four core components:

• script/agent.py — Main entrypoint. Initializes runtime, runs tasks (interactive or single request), performs workspace cleanup, and writes logs/experience memory.
• script/graph.py — Orchestration layer (LangGraph StateGraph). Defines the three-phase workflow and routing.
• script/tools/ — Modular toolset for search, download, analysis, cutting, transitions, narration, subtitles, and export.
• script/visualize.py — Log parser + local trace server for inspecting phase progress and tool calls.

Supporting folders:

• temp/ — Intermediate and output artifacts during execution.
• user_temp/ — User-provided local source assets.
• logs/ — Runtime logs (video_agent_*.log).
• memory_experience/ — Concise historical-case notes kept for reference only; they must not override the current task goal.
• website/ — Static launch site and GitHub Pages assets.

Workflow

Crayotter uses a three-phase architecture:

START -> planner -> phase1_scheduler -> material_gap_evaluator
material_gap_evaluator -> planner (supplement) | editing_research | react_editor
editing_research -> react_editor -> END

1. Phase 1 — Material Preparation (Planner + Executor)

   • Planner emits an explicit dependency DAG.
   • A deterministic scheduler validates dependencies, resource pools, retries, and write conflicts.
   • Search, per-video download, and per-video analysis tasks execute concurrently when resources are available.
   • A Material Gap Evaluator decides whether to proceed or run an incremental sourcing round (at most two supplement rounds, reusing already-successful tasks).
   • Search candidate videos through the platform-agnostic material-source layer.
   • Import user-provided URLs from Bilibili, Douyin, Xiaohongshu/Rednote, Kuaishou, YouTube, or generic URLs when supported by yt-dlp.
   • Rank/select high-quality candidates (target orientation is a scoring factor: landscape by default, portrait when requested).
   • Download selected videos and standardize them into editing-compatible MP4/H.264/AAC assets.
   • Analyze each source video multimodally.

2. Phase 2 — Editing Research

   • Research source analyses concurrently.
   • Build narrative, visual, pacing, and narration strategies concurrently.
   • Integrate them into one structured editing blueprint (JSON + compatibility Markdown).
   • No editing tools are called in this phase — it is pure reasoning.

   This phase can be disabled with CRAYOTTER_ENABLE_PHASE2_RESEARCH=false in the runtime .env to save tokens. When disabled, the workflow becomes: Phase 1 → Phase 3.

3. Phase 3 — ReAct Editing Execution

   • Prefer a controlled editing DAG with parallel clip cutting and segmented TTS.
   • Keep timeline merge, mixing, subtitles, quality evaluation, and export serial.
   • Fall back to the existing ReAct editor when structured planning or validation fails.
   • Log full tool-call trajectory for later trace visualization.

Quick Start

Windows Standalone Release

Windows 10/11 x64 users can download Crayotter-Windows-x64.zip from the release page:

1. Extract the complete archive.
2. Double-click Crayotter.exe.
3. Enter the API key and model settings in the workbench.

The release includes Python, FFmpeg, and yt-dlp, so users do not need to install Python. It opens in a native desktop window when Microsoft Edge WebView2 is available and falls back to the default browser otherwise.

Runtime data is stored next to the executable when that directory is writable, or under %LOCALAPPDATA%\Crayotter otherwise. The release never includes .env, API keys, uploaded media, logs, or generated videos.

To build the Windows x64 release with Python 3.12:

powershell -ExecutionPolicy Bypass -File packaging\build_windows.ps1

The build produces:

• dist/Crayotter/
• Crayotter-Windows-x64.zip

Prerequisites

Ensure ffmpeg is reachable from your PATH so it can be called directly in a terminal. Download it from https://ffmpeg.org/download.html and verify with ffmpeg -version. On Windows, the bundled binaries under script/dep/windows (ffmpeg.exe, ffprobe.cmd, yt-dlp.exe) are also prepended to PATH automatically.

1) Environment

Use Python 3.10+.

python -m venv .venv
.venv\Scripts\activate

2) Install Dependencies

pip install -r requirements.txt

3) Configure API Endpoints and Runtime Options

Copy .env.example to .env, then edit the values there:

copy .env.example .env

Common options:

CRAYOTTER_API_KEY=your-key
CRAYOTTER_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1
CRAYOTTER_MODEL_NAME=qwen-plus
CRAYOTTER_VIDEO_MODEL_NAME=qwen-vl-max-latest
CRAYOTTER_TTS_MODEL_NAME=qwen-tts-latest

CRAYOTTER_ENABLE_PHASE2_RESEARCH=true
CRAYOTTER_ENABLE_PLAN_REVIEW=true
CRAYOTTER_DIRECT_PHASE3_EXECUTION=false
CRAYOTTER_PREFER_LOCAL_MATERIALS=false
CRAYOTTER_SEARCH_POOL_SIZE=4
CRAYOTTER_DOWNLOAD_POOL_SIZE=2
CRAYOTTER_VIDEO_ANALYSIS_POOL_SIZE=2
CRAYOTTER_LLM_POOL_SIZE=2
CRAYOTTER_FFMPEG_POOL_SIZE=2
CRAYOTTER_TTS_POOL_SIZE=2
CRAYOTTER_EXPORT_POOL_SIZE=1
CRAYOTTER_ST