vocaloid-style-mv-pipeline

What it is

Give an AI agent a song and its lyrics, and this kit lets it produce a finished, showreel-quality lyric music video in the ボカロ / V家手书 / 文字PV tradition. You get a 1920×1080 master, a natively re-laid-out 1080×1920 vertical version, and share encodes.

It is front-end animation. Every frame is drawn by HTML Canvas2D and composited by WebGL2 in a headless browser. Playwright captures the frames and ffmpeg encodes them. No After Effects, no timeline editor. A frame is a pure function of song time t, so the film is code: re-rendering always gives the same result, parallel workers can render any frame, and an agent can review, fix and re-render a 4-minute MV in minutes.

The kit has two parts:

• The skill (skills/vocaloid-style-mv/): SKILL.md plus method references that take an agent from listen → concept → assets → storyboard → build → review → deliver.
• The project template (skills/vocaloid-style-mv/template/): the engine, a scene library, the JIZURA 字面 lyric-motion layer, audio-analysis and lyric-timing tools, AI image-generation tooling, Blender toon-3D shot scripts and the renderer. Every new MV project starts as a copy of it.

The skill was written for Claude Code, but it is plain Markdown plus scripts, so other agents such as Codex can follow it too.

Showcase: 「残光」

「残光」 (Zankō, "Afterglow"). Music / Lyrics / Movie：NikusonP. A 3:46 MV with an original V家-style singer, 灯 (Akari). The sky cracks like glass, she falls into the night, keeps a small flame alive, gathers the pieces of her shattered heart, and walks a road made of her own tears toward the dawn. It has 68 beat-locked shots, JIZURA typography, two Blender toon-3D sequences, and a hard cut on the song's dead stop to a cyan afterimage. The full 1080p30 film renders in ≈4 minutes on an RTX 3080 (6,796 frames, 3 workers, ≈31 fps).

More in the gallery (click to open full size):

The full case study (logline, colour script, shot structure, signature moments and how each was built) is in examples/zanko/.

Features

• Deterministic renderer. Each frame is a pure function of t: seeded noise, closed-form particles, no wall-clock time. Parallel headless-browser workers seek arbitrary frames, and the segments join without seams.
• Layered compositing. Each frame has a scene layer (Canvas2D illustration, camera, particles), a text layer (JIZURA lyrics, custom typography, HUD), a foreground layer (the character drawn in front of the lyrics, so text can pass behind her), and a WebGL2 post pass. The post pass does line boil on 2s, OKLab palette gradient maps, bloom, chromatic aberration, glitch, zoom blur, grain, flashes and letterbox. Text stays crisp because the scene-only effects never touch it.
• Scene library. 24 parameterised scenes are included: illust, kanjiSlam, typeCard, titleRiff, ember, heart, skyShatter, fall, flame, ink, thread, wetFloor, train, memories, road, panels, burst, tunnel, converge, afterimage, ref (freeze / rewind / melt another shot), seq (image sequences), jz, solid. There are also 13 transitions: cut, xfade, flash, black, iris, irisClose, barrier, wipe, slices, push, zoom, ink, shatter. All of them are a toolbox to adapt, not a look to reuse.
• JIZURA 字面 lyric layer. The upstream engine is built unmodified into a deterministic layer, with several plans (styles) per film, curated deny-lists, centre-free layouts for character shots, and native 9:16 re-typesetting.
• A beat-locked edit. librosa analysis gives beats, downbeats, bars, sections, stops, impacts and per-frame envelopes, and it handles tempo drift. Timeline helpers snap every cut to a real bar or beat.
• Lyric timing. Official lyrics come first. faster-whisper, vocal-stem separation and forced alignment are there for timings, or for the case where no text exists. The output is an LRC file with JIZURA markup (/ cut, *emphasis*, ! キメ line).
• An original character and consistent assets. The pipeline goes character bible → master candidates → every pose generated with the master as a reference image → chroma key to RGBA. Batch generation runs through the Codex CLI, and any image model works with the same method.
• Toon 3D. The Blender 5.x shot scripts use EEVEE, Shader-to-RGB cel shading and Grease Pencil line art, at about 0.7 s per 1080p frame. Cameras are keyed to the beat grid, and the result plays back in the engine as an image sequence.
• Delivery. You get a 16:9 master (NVENC CQ16 or x264), a share encode, a 9:16 vertical re-layout and 720p previews, plus review tools: stills, contact sheets, review sheets every N seconds, and a JIZURA cut inspector.

Pipeline

flowchart LR
  A["song.wav + official lyrics"] --> B["Listen<br/>librosa beats · bars · sections<br/>lyric timing (LRC)"]
  B --> C["Concept<br/>thesis · colour script<br/>motifs · character bible"]
  C --> D["Assets<br/>Codex image gen<br/>chroma key → RGBA"]
  C --> E["Storyboard<br/>→ engine/timeline.js"]
  D --> F["Build<br/>engine scenes + JIZURA lyrics<br/>+ Blender toon 3D"]
  E --> F
  F --> G["Render<br/>Playwright + headless Edge<br/>+ ffmpeg NVENC"]
  G --> H["Review<br/>stills · contact sheets"]
  H -- "fix, 2–3 passes" --> F
  H --> I["Deliver<br/>16:9 master · share · 9:16"]

What happens inside one frame:

 song time t ─┬─► scene layer       Canvas2D 1920×1080: illustration, camera, particles ──┐
              ├─► text layer        JIZURA 字面 + custom typography + HUD (transparent) ──┼─► WebGL2 post ─► frame
              └─► foreground layer  the character, in front of the lyrics ────────────────┘   line boil · palette LUT · bloom
                                                                                              CA · glitch · grain · flash

Quick start

1. Check the machine.

python skills/vocaloid-style-mv/scripts/check_env.py

2a. Install the skill (recommended). Copy it to your Claude Code skills folder:

# macOS / Linux / Git Bash
mkdir -p ~/.claude/skills && cp -r skills/vocaloid-style-mv ~/.claude/skills/

# Windows PowerShell
New-Item -ItemType Directory -Force "$env:USERPROFILE\.claude\skills" | Out-Null
Copy-Item -Recurse -Force skills\vocaloid-style-mv "$env:USERPROFILE\.claude\skills\"

Then put your song and lyrics in a folder, open Claude Code there and ask:

make a V家手書き MV for this song (song.wav, lyrics.txt)

The skill scaffolds a project, analyses the song, proposes a concept for you to check, generates the art, builds and renders the film, and reviews it. It stops at a few checkpoints along the way: the concept, the character master, the first stills and the first full render.

2b. Or use the template directly.

python skills/vocaloid-style-mv/scripts/new_project.py ../my-mv --song path/to/song.wav --lyrics path/to/lyrics.lrc --setup
cd ../my-mv

--setup ins