Mixamo LLM Mocap

Turn any locked-camera video — filmed or AI-generated — into a clean FK animation on any Mixamo character. One performer, or two fighting each other. No mocap suit, no manual keyframing, and every stage scriptable enough that an AI agent can run the whole loop.

Left: AI-generated source video. Right: the automatic retarget on a Mixamo character in Blender — 10 seconds, nine punches, a slip under and a high side kick, straight through the pipeline.

Two performers, one plate, one pass. The left fighter throws four punches and a roundhouse; the right one blocks, folds over the body shot and ducks under the kick. Both tracks are split out of the same video by screen side, retargeted onto two Mixamo characters with different proportions — Y Bot and Ninja — and placed at the distance the performers actually stood, measured from the footage.

How it works

video plate (locked camera, T-pose bookends)
   │
   ├─ 1. estimate_pose_gvhmr.py    GVHMR (SMPL-X mesh recovery) → 33 landmarks + pelvis height
   ├─ 2. analyze_landmarks.py     numeric beat detection → you write a beat sheet from NUMBERS
   ├─ 3. action_specs/<name>.json  the motion as data: support schedule, rest blends, fists
   ├─ 4. lift_to_mixamo.py         direction-preserving retarget onto YOUR rig's proportions
   ├─ 5. apply_mixamo_fk.py        FK aim + foot planting, inside live Blender (via Blender MCP)
   ├─ 6. qa_clip.py                automated gate: no explosions, no pops, no foot skate
   ├─ 7. compare_reference.py      frame-by-frame vs the video → which windows still differ
   ├─ 8. compare_pair.py           two-character plates: separation, reach, intrusion
   ├─ 9. run_in_blender.py contact real mesh-vs-mesh collision between two characters
   └─ 10. render_preview.py        preview + side-by-side showcase video

With two performers in the plate, stages 1–7 run once per fighter (--person left|right splits the tracks), setup_duo.py builds one scene holding both characters, and compare_pair.py checks what only exists when there are two of them: whether they stand, reach and miss each other the way the performers did.

The estimator provides mesh-quality joints; the lift keeps its segment directions but rebuilds every position from your character's measured bone lengths; the apply plants feet by solving hip height (never IK — Mixamo rigs are FK-only); the spec contributes only what a video cannot know: which foot is the support in each phase (including "none" for airborne beats), when fists close, where the clip locks back to rest.

Why it's different

• Any Mixamo character. setup_rig.py builds a clean scene from your own Mixamo download and measures it into rig_profile.json (rest pose, bone lengths, hip and ground heights). Every stage reads that profile.
• Motions are data, not code. A new motion is a small JSON spec — the action_specs/ here (a kung-fu form, a combo with a jump, a fight combination, a 360° jumping spin kick and a two-fighter duel) are worked examples of the whole schema.
• Honest Mixamo FK. Hips are the only translating bone, everything else is quaternions at 30 fps — clips drop into any Mixamo-style workflow without cleanup.
• Real ground contact. Planted feet solve to ground height with zero skate (the support ankle is pinned through each stance); jumps integrate the estimator's real pelvis arc.
• A QA gate, not vibes. Exploded bones, hip pops, foot skate, drifting roots and broken rest poses are caught numerically before a human ever looks.
• A closed refinement loop. compare_reference.py measures the retarget against the source video frame by frame on what an eye actually reads — hand height relative to the face, distance between the hands, limbs inside the torso, gaze direction — and reports the exact frame windows that diverge. Notes like "his hands are too high and his arm clips his back" become numbers, and an over-correction gets caught before it ships instead of after.
• Two characters, one scene. A two-performer plate is split into tracks by which side of frame each occupies — robust where tracker ids swap on contact — retargeted onto two different Mixamo characters with their own measured proportions, and placed at the distance the performers actually stood, recovered from the footage rather than eyeballed. compare_pair.py then verifies separation, strike reach and limb intrusion against the video, frame by frame, and a Blender BVH pass checks the actual skinned meshes for collision — because two Mixamo characters are thicker than two humans, and a choreography built out of 2 cm near-misses collides when you retarget it faithfully. Clearance is bought from the stage with a declared, measured offset, which the comparator keeps reporting so the cost stays visible.
• A review pass that is part of the loop. Render the showcase, put source and retarget side by side at the same beat, name what looks wrong in one sentence, then measure it. When eye and numbers disagree it is usually the numbers — every false reading in this project came from a mismatched proxy (a nose against a skull-base joint, a capsule against a mesh). docs/PIPELINE.md section 10.
• Written for agents. Beat decisions come from analyze_landmarks.py numbers (never from eyeballing frames), every stage is a CLI or a socket call, and docs/PITFALLS.md encodes every mistake so the next operator — human or AI — doesn't repeat them.

Quickstart

1. Install — docs/INSTALL.md walks through every dependency (list below).

2. Build your rig scene:

   blender --background --python pipeline\setup_rig.py -- --fbx ybot.fbx --out ybot_rest.blend
   

3. Run a plate (Blender open on the scene; plate rules in docs/PROMPTING.md):

   tools\GVHMR\.venv\Scripts\python.exe pipeline\estimate_pose_gvhmr.py --video plates\<name>\<name>.mp4 --out plates\<name>\landmarks.json
   tools\GVHMR\.venv\Scripts\python.exe pipeline\analyze_landmarks.py --landmarks plates\<name>\landmarks.json
   # beat sheet → action_specs\<name>.json  (schema: docs/PIPELINE.md)
   tools\GVHMR\.venv\Scripts\python.exe pipeline\lift_to_mixamo.py --spec action_specs\<name>.json
   python pipeline\run_in_blender.py all action_specs\<name>.json
   tools\GVHMR\.venv\Scripts\python.exe pipeline\qa_clip.py --spec action_specs\<name>.json
   tools\GVHMR\.venv\Scripts\python.exe pipeline\compare_reference.py --spec action_specs\<name>.json
   tools\GVHMR\.venv\Scripts\python.exe pipeline\render_preview.py action_specs\<name>.json --showcase
   

   compare_reference.py tells you which frame windows still differ from the video; the last command produces preview.mp4 and the side-by-side showcase.mp4 — the same format as the demo GIF above.

   Two-performer plates add --person left|right to the estimate, one spec per fighter, and a compare_pair.py run — see docs/PIPELINE.md section 9.

4. Iterate with docs/PIPELINE.md and docs/PITFALLS.md.

What you need to bring (and where to get it)

Docs

License

MIT — see LICENSE, including third-party notes (Mixamo, GVHMR, SMPL-X, Blender MCP).