by pifferologo
ai agent video editor use with ElevenLabs Scribe, pack phrase-level transcripts, reason over an EDL, render with ffmpeg, video editor ai agent
# Add to your Claude Code skills
git clone https://github.com/pifferologo/ai-agent-video-editorGuides for using ai agents skills like ai-agent-video-editor.
ai-agent-video-editor is an open-source ai agents skill for AI coding assistants such as Claude Code, Codex CLI, and ChatGPT, built by pifferologo. ai agent video editor use with ElevenLabs Scribe, pack phrase-level transcripts, reason over an EDL, render with ffmpeg, video editor ai agent. It has 151 GitHub stars.
ai-agent-video-editor's catalog security scan is still queued. You can run an instant dependency and prompt-injection check now with the "Scan for vulnerabilities" button above.
Clone the repository with "git clone https://github.com/pifferologo/ai-agent-video-editor" and add it to your Claude Code skills directory (see the Installation section above). ai-agent-video-editor ships a SKILL.md manifest, so compatible agents can discover and load it automatically.
ai-agent-video-editor is primarily written in TypeScript. It is open-source under pifferologo on GitHub, so you can review or fork the full source.
Yes. SkillsLLM lists many other AI Agents skills you can browse and compare side by side. Open the AI Agents category from the badge at the top of this page, or use the Related Skills and comparison links further down to weigh ai-agent-video-editor against similar tools.
No comments yet. Be the first to share your thoughts!
Unlocks once the catalog security scan passes (runs nightly).
The deep catalog scan for this skill is still queued. Run an instant dependency check now instead.
takes_packed.md). Everything else — filler tagging, retake detection, shot classification, emphasis scoring — you derive at decision time.These are the things where deviation produces silent failures or broken output. They are not taste, they are correctness. Memorize them.
-c copy concat, not single-pass filtergraph. Otherwise you double-encode every segment when overlays are added.afade=t=in:st=0:d=0.03,afade=t=out:st={dur-0.03}:d=0.03). Otherwise audible pops at every cut.setpts=PTS-STARTPTS+T/TB to shift the overlay's frame 0 to its window start. Otherwise you see the middle of the animation during the overlay window.output_time = word.start - segment_start + segment_offset. Otherwise captions misalign after segment concat.Agent tool; total wall time ≈ slowest one.<videos_dir>/edit/. Never write inside the video-use/ project directory.Everything else in this document is a worked example. Deviate whenever the material calls for it.
The skill lives in video-use/. User footage lives wherever they put it. All session outputs go into <videos_dir>/edit/.
<videos_dir>/
├── <source files, untouched>
└── edit/
├── project.md ← memory; appended every session
├── takes_packed.md ← phrase-level transcripts, the LLM's primary reading view
├── edl.json ← cut decisions
├── transcripts/<name>.json ← cached raw Scribe JSON
├── animations/slot_<id>/ ← per-animation source + render + reasoning
├── clips_graded/ ← per-segment extracts with grade + fades
├── master.srt ← output-timeline subtitles
├── downloads/ ← yt-dlp outputs
├── verify/ ← debug frames / timeline PNGs
├── preview.mp4
└── final.mp4
First-time install lives in install.md (clone, deps, ffmpeg, skill registration, API key). Don't re-run it every session; on cold start just verify:
ELEVENLABS_API_KEY resolves — either in the environment or in .env at the video-use repo root. If missing, ask the user to paste one and write it to .env (never to the user's <videos_dir>).ffmpeg + ffprobe on PATH.npm install inside the repo).yt-dlp, HyperFrames, Remotion, Manim installed only on first use.npx --yes hyperframes ...; Remotion can be scaffolded with npx create-video@latest or installed as a project-local dependency before using its remotion render command.skills/manim-video/. Read its SKILL.md when building a Manim slot.The CLI lives in this repo. Resolve commands as npx video-use <subcommand> from the video-use root, or use a globally linked install.
npx video-use transcribe <video> — single-file Scribe call. --num-speakers N optional. Cached.npx video-use transcribe-batch <videos_dir> — 4-worker parallel transcription. Use for multi-take.npx video-use pack --edit-dir <dir> — transcripts/*.json → takes_packed.md (phrase-level, break on silence ≥ 0.5s).npx video-use timeline <video> <start> <end> — filmstrip + waveform PNG. On-demand visual drill-down. Not a scan tool — use it at decision points, not constantly.npx video-use render <edl.json> -o <out> — per-segment extract → concat → overlays (PTS-shifted) → subtitles LAST. --preview for 720p fast. --build-subtitles to generate master.srt inline.npx video-use grade <in> -o <out> — ffmpeg filter chain grade. Presets + --filter '<raw>' for custom.For animations, create <edit>/animations/slot_<id>/ with Bash and spawn a sub-agent via the Agent tool.
Inventory. ffprobe every source. npx video-use transcribe-batch <dir>. npx video-use pack --edit-dir <edit>. Sample one or two npx video-use timeline composites for a visual first impression.
Pre-scan for problems. One pass over takes_packed.md to note verbal slips, obvious mis-speaks, or phrasings to avoid. Plain list, feed into the editor brief.
Converse. Describe what you see in plain English. Ask questions shaped by the material. Collect: content type, target length/aspect, aesthetic/brand direction, pacing feel, must-preserve moments, must-cut moments, animation and grade preferences, subtitle needs. Do not use a fixed checklist — the right questions are different every time.
Propose strategy. 4–8 sentences: shape, take choices, cut direction, animation plan, grade direction, subtitle style, length estimate. Wait for confirmation.
Execute. Produce edl.json via the editor sub-agent brief. Drill into npx video-use timeline at ambiguous moments. Build animations in parallel sub-agents. Apply grade per-segment. Compose via npx video-use render.
Preview. npx video-use render ... --preview.
Self-eval (before showing the user). Run npx video-use timeline on the rendered output (not the sources) at every cut boundary (±1.5s window). Check each image for:
Also sample: first 2s, last 2s, and 2–3 mid-points — check grade consistency, subtitle readability, overall coherence. Run ffprobe on the output to verify duration matches the EDL expectation.
If anything fails: fix → re-render → re-eval. Cap at 3 self-eval passes — if issues remain after 3, flag them to the user rather than looping forever. Only present the preview once the self-eval passes.
Iterate + persist. Natural-language feedback, re-plan, re-render. Never re-transcribe. Final render on confirmation. Append to project.md.
(laughs), (sighs), (applause) mark beats. Extend past them.pack reads all transcripts/*.json and produces one markdown file where each take is a list of phrase-level lines, each prefixed with its [start-end] time range. Phrases break on any silence ≥ 0.5s OR speaker change. This is the artifact the editor sub-agent reads to pick cuts — it gives word-boundary precision from text alone at 1/10 the tokens of raw JSON.
Example line:
## C0103 (duration: 43.0s, 8 phrases)
[002.52-005.36] S0 Ninety percent of what a web agent does is completely wasted.
[006.08-006.74] S0 We fixed this.
When the task is "pick the best take of each beat across many clips," spawn a dedicated sub-agent with a brief shaped like this. The structure is load-bearing; the pitch-shape example is not.
You are editing a <type> video. Pick the best take of each beat and
assemble them chronologically by beat, not by source clip order.
INPUTS:
- takes_packed.md (time-annotated phrase-level transcripts of all takes)
- Product/narrative context: <2 sentences from the user>
- Speaker(s): <name, role, delivery style note>
- Expected structure: <pick an archetype or invent one>
- Verbal slips to avoid: <list from the pre-scan pass>
- Target runtime: <seconds>
Common structural archetypes (pick, adapt, or invent):
- Tech launch / demo: HOOK → PROBLEM → SOLUTION → BENEFIT → EXAMPLE → CTA
- Tutorial: INTRO → SETUP → STEPS → GOTCHAS → RECAP
- Interview: (QUESTION → ANSWER → FOLLOWUP) repeat
- Travel / event: ARRIVAL → HIGHLIGHTS → QUIET MOMENTS → DEPARTURE
- Documentary: THESIS → EVIDENCE → COUNTERPOINT → CONCLUSION
- Music / performance: INTRO → VERSE → CHORUS → BRIDGE → OUTRO
- Or invent your own.
RULES:
- Start/end times must fall on word boundaries from the transcript.
- Pad cut boundaries (working window 30–200ms).
- Prefer silences ≥ 400ms as cut targets.
- Unavoidable slips are kept if no better take exists. Note them in "reason".
- If over budget, revise: drop a beat or trim tails. Report total and self-correct.
OUTPUT (JSON array, no prose):
[{"source": "C0103", "start": 2.42, "end": 6.85, "beat": "HOOK",
"quote": "...", "reason": "..."}, ...]
Return the final EDL and a one-line total runtime check.
Your job is to reason about the image, not apply a preset. Look at a frame (via npx video-use timeline), decide what's wrong, adjust one thing, look again.
Mental model is ASC CDL. Per channel: out = (in * slope + offset) ** power, then global saturation. slope → highlights, offset → shadows, power → midtones.
Example filter chains (npx video-use grade --list-presets; use them as starting points or mix your own):
warm_cinematic — retro/technical, subtle teal/orange split, desaturated. Shipped in a real launch video. Safe for talking heads.neutral_punch — minimal corrective: contrast bump + gentle S-curve. No hue shifts.none — straight copy. Default when the user hasn't asked.For anything else — portraiture, nature, product, music video, documentary — invent your own chain. npx video-use grade --filter '<raw ffmpeg>' accepts any filter string.
Hard rules: apply per-segment during extraction (not post-concat, which re-encodes twice). Never go aggressive without testing skin tones.
Subtitles have three dimensions worth reasoning about: chunking (1/2/3/sentence per line), case (UPPER/Title/Natural), and placement (margin from bottom). The right combo depends on content.
Worked styles — pick, adapt, or invent:
bold-overlay — short-form tech launch, fast-paced social. 2-word chunks, UPPERCASE, break on punctuation, Helvetica 18 Bold, white-on-outline, MarginV=35. npx video-use render ships with this as SUB_FORCE_STYLE.
FontName=Helvetica,FontSize=18,Bold=1,
PrimaryColour=&H00FFFFFF,OutlineColour=&H00000000,BackColour=&H00000000,
BorderStyle=1,Outline=2,Shadow=0,
Alignment=2,MarginV=35
natural-sentence (if you invent this mode) — narrative, documentary, education. 4–7 word chunks, sentence case, break on natural pauses, MarginV=60–80, larger font for readability, slightly wider max-width. No shipped force_style — design one if you need it.
Invent a third style if neither fits. Hard rules: subtitles LAST (Rule 1), output-timeline offsets (Rule 5).
Animations match the content and the brand. Get the palette, font, and visual language from the conversation — never assume a default. If the user hasn't told you, propose a palette in the strategy phase and wait for confirmation before building anything.
Tool options:
Pick the engine per animation slot. Do not default to Remotion just because the animation is web-adjacent.
skills/manim-video/SKILL.md and its references for depth.For HyperFrames slots, scaffold the slot inside edit/animations/slot_<id>/ with npx --yes hyperframes init . --example blank --non-interactive --skip-skills, build the HTML composition there, run the HyperFrames checks that fit the slot (lint, validate, and a draft render when practical), then produce the final overlay video with npx --yes hyperframes render . -o render.mp4 or --format webm -o render.webm when alpha is required. Point the EDL overlay file at the actual rendered path.
For Remotion slots, keep the Remotion project isolated inside the same slot directory, scaffold with npx create-video@latest or install Remotion locally there, render the composition to render.mp4 with the project-local remotion render command, and verify duration and dimensions with ffprobe.
None is mandatory. Invent hybrids if useful (e.g., PIL background with a HyperFrames or Remotion layer on top).
Duration rules of thumb, context-dependent:
narration_length + 1s (universal).Animation payoff timing (rule for sync-to-narration): get the payoff word's timestamp. Start the overlay reveal_duration seconds earlier so the landing frame coincides with the spoken payoff word. Without this sync the animation feels disconnected.
Easing (universal — never linear, it looks robotic):
def ease_out_cubic(t): return 1 - (1 - t) ** 3
def ease_in_out_cubic(t):
if t < 0.5: return 4 * t ** 3
return 1 - (-2 * t + 2) ** 3 / 2
ease_out_cubic for single reveals (slow landing). ease_in_out_cubic for continuous draws.
Typing text anchor trick: center on the FULL string's width, not the partial-string width — otherwise text slides left during reveal.
Example palette (the launch video — one aesthetic among infinite):
(10, 10, 10) near-black#FF5A00 / (255, 90, 0) orange(110, 110, 110) dim gray/System/Library/Fonts/Menlo.ttc (index 1)This is one style. If the brand is warm and serif, use that. If it's colorful and playful, use that. If the user handed you a style guide, follow it. If they didn't, propose one and confirm.
Parallel sub-agent brief — each animation is one sub-agent spawned via the Agent tool. Each prompt is self-contained (sub-agents have no parent context). Include:
<edit>/animations/slot_<id>/render.mp4)One sub-agent = one file (unique filenames, parallel agents don't overwrite each other).
Match the source unless the user asked for something specific. Common targets: 1920×1080@24 cinematic, 1920×1080@30 screen content, 1080×1920@30 vertical social, 3840×2160@24 4K cinema, 1080×1080@30 square. npx video-use render defaults the scale to 1080p from any source; pass --filter or edit the extract command for other targets. Worth asking the user which delivery format matters.
{
"version": 1,
"sources": {"C0103": "/abs/path/C0103.MP4", "C0108": "/abs/path/C0108.MP4"},
"ranges": [
{"source": "C0103", "start": 2.42, "end": 6.85,
"beat": "HOOK", "quote": "...", "reason": "Cleanest delivery, stops before slip at 38.46."},
{"source": "C0108", "start": 14.30, "end": 28.90,
"beat": "SOLUTION", "quote": "...", "reason": "Only take without the false start."}
],
"grade": "warm_cinematic",
"overlays": [
{"file": "edit/animations/slot_1/render.mp4", "start_in_output": 0.0, "duration": 5.0}
],
"subtitles": "edit/master.srt",
"total_duration_s": 87.4
}
grade is a preset name or raw ffmpeg filter. overlays are rendered animation clips. subtitles is optional and applied LAST.
project.mdAppend one section per session at <edit>/project.md:
## Session N — YYYY-MM-DD
**Strategy:** one paragraph describing the approach
**Decisions:** take choices, cuts, grades, animations + why
**Reasoning log:** one-line rationale for non-obvious decisions
**Outstanding:** deferred items
On startup, read project.md if it exists and summarize the last session in one sentence before asking whether to continue.
Things that consistently fail regardless of style:
Conversation-driven video editing for AI agents. Transcribe raw footage with ElevenLabs Scribe, pack phrase-level transcripts, reason over an EDL, render with ffmpeg, and self-evaluate cut boundaries — all through a typed Node.js CLI.
video-use gives an LLM structured text (word-level transcripts) plus on-demand visual composites instead of dumping video frames. An agent reads takes_packed.md, proposes a cut strategy, writes edl.json, and renders final.mp4 into <videos_dir>/edit/.
flowchart LR
subgraph inputs [Inputs]
RAW[Raw takes]
ENV[.env + ffmpeg]
end
subgraph pipeline [Pipeline]
T[transcribe]
P[pack]
E[EDL reasoning]
R[render]
V[self-eval timeline]
end
subgraph outputs [Outputs]
MD[takes_packed.md]
MP4[final.mp4]
end
RAW --> T --> P --> MD
MD --> E --> R --> MP4
R --> V --> MP4
ENV --> T
ENV --> R
| Requirement | Notes |
|---|---|
| Node.js 18+ | Required for the CLI |
| ffmpeg / ffprobe | Hard requirement for all media operations |
| ElevenLabs API key | Scribe transcription (get a key) |
| Redis 6+ | Optional; enable with REDIS_ENABLED=true |
git clone https://github.com/browser-use/video-use ~/Developer/video-use
cd ~/Developer/video-use
npm install
npm run build
cp .env.example .env
# Edit .env — set ELEVENLABS_API_KEY
Register the skill with your agent (symlink the repo into your skills directory). See install.md for agent-specific steps.
Set up https://github.com/browser-use/video-use for me.
Read install.md first, then SKILL.md for daily usage.
Use npx video-use for all editing commands.
Copy .env.example to .env:
| Variable | Default | Description |
|---|---|---|
ELEVENLABS_API_KEY |
— | Required for transcription |
LOG_LEVEL |
info |
debug, info, warn, or error |
REDIS_ENABLED |
false |
Enable Redis transcript cache |
REDIS_URL |
redis://127.0.0.1:6379 |
Redis connection URL |
REDIS_KEY_PREFIX |
video-use: |
Key namespace prefix |
REDIS_DEFAULT_TTL_SECONDS |
86400 |
Cache TTL (24 h) |
Verify Redis connectivity:
npx video-use redis ping
npx video-use transcribe <video> [--edit-dir <dir>] [--num-speakers N]
npx video-use transcribe-batch <videos_dir> [--workers 4]
npx video-use pack --edit-dir <dir> [--silence-threshold 0.5]
npx video-use timeline <video> <start> <end> [-o out.png]
npx video-use render <edl.json> -o final.mp4 [--preview] [--build-subtitles]
npx video-use grade <input> -o output [--preset warm_cinematic]
npx video-use redis ping
Run npx video-use --help for full option lists.
video-use/
├── src/
│ ├── cli/ # Commander entry point
│ ├── config/ # env + logger
│ ├── lib/ # ffmpeg, scribe, pack, grade, render, timeline
│ ├── redis/ # connection manager + cache
│ └── types/ # shared TypeScript interfaces
├── tests/ # Vitest unit tests
├── docs/AUDIT.md # internal architecture audit
├── SKILL.md # agent editing instructions
├── install.md # first-time setup guide
└── skills/manim-video/ # vendored animation skill
flowchart TB
CLI[src/cli/index.ts]
CFG[src/config]
LIB[src/lib]
REDIS[src/redis]
TYPES[src/types]
CLI --> CFG
CLI --> LIB
LIB --> CFG
LIB --> TYPES
LIB --> REDIS
REDIS --> CFG
npm install
npm run dev -- --help # run CLI via tsx without building
npm run typecheck # tsc --noEmit (strict)
npm run lint # ESLint
npm run test # Vitest
npm run build # emit dist/
npm run validate # all of the above
sequenceDiagram
participant EDL as edl.json
participant X as extract segments
participant C as concat base
participant O as overlays
participant S as subtitles
participant L as loudnorm
EDL->>X: per-range grade + fades
X->>C: lossless -c copy
C->>O: PTS-shifted overlays
O->>S: burn captions LAST
S->>L: -14 LUFS normalize
L->>EDL: final.mp4
npm run test
Tests cover phrase packing logic and Redis manager lifecycle (disabled-mode paths). Integration tests against ffmpeg and Scribe are intentionally manual — they require external services and burn API credits.
ELEVENLABS_API_KEY not foundEnsure .env exists at the repo root with a non-empty key, or export the variable in your shell.
ffmpeg failed with exit codeConfirm ffmpeg and ffprobe are on PATH and support libx264. Run ffprobe -version.
Set REDIS_ENABLED=true and confirm Redis is running locally (redis-cli ping → PONG).
Subtitles must be applied last in the filter chain (Hard Rule 1 in SKILL.md). Use npx video-use render — it enforces this order.
The render pipeline detects PQ/HLG sources and applies tone-mapping automatically. If issues persist, probe with ffprobe -show_entries stream=color_transfer.
Does the LLM watch the video? No. It reads packed transcripts (~12 KB per hour of takes) and requests timeline PNGs only at decision points.
Where do session outputs go?
Always <videos_dir>/edit/ — never inside the video-use repo.
Can I skip Redis?
Yes. Redis is off by default. Filesystem caching in edit/transcripts/ is always used.
What animation engines are supported?
HyperFrames, Remotion, Manim, and PIL sequences — see SKILL.md. Each renders inside edit/animations/slot_<id>/.
How do I update the skill?
git pull in the clone; re-run npm install && npm run build if dependencies changed.
npm run validate before opening a PR.js extensions)See SKILL.md for production editing rules and docs/AUDIT.md for architecture notes.
MIT — see LICENSE.