AI FILMS benchmark chart from FlowHMR paper (arXiv:2610.03691)
Share this post:
FlowHMR: Open Source 3D Human Motion Capture
Markerless motion capture from single camera video usually produces floating feet, sliding steps, and impossible joint angles that break CG character rigs. FlowHMR (fafsaf1/flowhmr-0.46B), published in October 2026 (arXiv:2610.03691), solves depth ambiguity by treating motion capture as a video conditioned flow matching generation problem refined with Group Relative Policy Optimization (GRPO) physics rewards. On the new 4,000 clip Wild-4K benchmark, FlowHMR hits an 82.47% physics tracking success rate, beating the strongest prior baseline GVHMR (62.82%) by nearly 20 percentage points, and exports directly to FBX and NPZ.
Studio availability. Try the public Hugging Face demo below; FlowHMR is not part of a paid AI FILMS Studio plan (use Kling 3.0 Pro Motion Control in the video workspace for example instead).
Try FlowHMR Live
Upload any single actor or multi actor clip (up to 900 frames / 30 seconds at 30 FPS) into the official FlowHMR Space below to recover global 3D motion, inspect synchronized video and skeleton playback, and export FBX or NPZ rig data.
Architecture and Technical Design
Earlier monocular human mesh recovery systems regressed 3D poses directly from 2D pixels using geometric losses. Because single camera footage is ambiguous in depth, direct regression averages across valid hypotheses, causing limbs to clip and feet to hover above the floor. FlowHMR instead pretrains a compact 0.46B flow matching generative model conditioned on per frame visual tracks extracted by SAM 3D Body (DINOv3) and optional camera poses estimated by VGGT-Omega.
AI FILMS pipeline diagram from fafsaf1/flowhmr-0.46B & arXiv:2610.03691
Pretraining generates diverse 3D motion candidates, but not every sample obeys gravity or matches the actor's exact silhouette. The authors therefore post train the flow model with GRPO using two joint rewards: a video fidelity reward that enforces alignment with the filmed performance, and a physics tracking reward that scores whether a simulated physics controller can actually execute the recovered motion without falling.
That dual reward shifts the output distribution toward grounded footsteps, solid momentum transfer, and clean root trajectories across dynamic camera moves.
Hardware, VRAM, and License Matrix
| Component | License | Hardware / Runtime Notes | Output Format |
|---|---|---|---|
FlowHMR Source Code (flowhmr/flowhmr) |
Apache 2.0 | PyTorch 2.8 / CUDA 12.8 recommended | Python CLI + Gradio UI |
flowhmr-0.46B Weights (flowhmr_latest) |
FlowHMR Non Commercial License | ~12 to 16 GB VRAM (with SAM 3D Body + VGGT-Omega) | SMPL-H NPZ & FBX |
| Companion Vision Stack | Third party model licenses | YOLOX-L, sam-3d-body-dinov3, VGGT-Omega |
30 FPS chunked tracks |
| SMPL-H Neutral Body Model | SMPL-H License (separate download) | Required for mesh reconstruction and FBX rigging | .npz body asset |
Take note of the split license structure. The FlowHMR codebase is Apache 2.0, while the flowhmr_base and flowhmr_latest weights carry a non commercial research, educational, and non profit license. Commercial studios must contact the authors for production weight licensing and maintain valid SMPL-H commercial rights.
Parameter Reference
| Parameter | Space Default | Production Guidance |
|---|---|---|
num_steps |
20 |
Default flow matching denoising steps for clean global trajectories |
cfg_scale |
1.0 |
Fixed CFG 1 matches the GRPO post trained distribution |
| Max frame limit | 900 frames (30s at 30 FPS) |
Split longer takes at scene cuts for cleanest camera estimation |
sam_chunk |
32 (hands 64) |
Controls SAM 3D Body tracking memory footprint |
vggt_interval |
10 |
Frame stride for VGGT-Omega moving camera estimation |
FLOWHMR_SPACE_CAMERA |
0 (static) or 1 (moving) |
Enable VGGT-Omega for handheld or dolly footage; disable for tripod shots |
Local Python and CLI Setup
Clone the Apache 2.0 repository, prepare the PyTorch 2.8 / CUDA 12.8 environment, and download the weights along with your licensed SMPL-H body model.
git clone https://github.com/flowhmr/flowhmr.git
cd flowhmr
hf download fafsaf1/flowhmr-0.46B --local-dir ./checkpoints/flowhmr_latest
# Launch the local Gradio interface (after placing SAM 3D Body, YOLOX, and SMPL-H assets)
python tools/hf_space/app.py
The repository also includes tools/hf_space/build_bundle.py for packaging self hosted Gradio instances and supports pinned FBX SDK wheels on Linux x86_64 (Python 3.10) for automated FBX animation export into Blender, Maya, or Unreal Engine.
Filmmaker Production Playbook
- Capture stunt and blocking references on a phone. Directors can shoot an actor rehearsing a fight beat or stairwell run on a standard 30 FPS smartphone, drop the clip into FlowHMR, and export an FBX skeleton with physically grounded foot contacts straight into Blender or Unreal Engine for camera previsualization.
- Switch off VGGT for locked off tripod plates. If your rehearsal camera sits on a tripod, keep static camera mode active (
FLOWHMR_SPACE_CAMERA=0). Skipping moving camera estimation eliminates subtle ground plane drift and cuts processing time. - Isolate one performer in multi actor shots. On clips with multiple people in frame, FlowHMR pauses after person detection so you can click the exact actor to track before GPU motion generation runs.
- Drive cloud video generation with motion control. When you want to transfer real actor movement directly onto stylized AI video without 3D rigging, use Kling 3.0 Pro Motion Control on our motion control page inside the video workspace.
Related Reading
- SCAIL-2 Animates Any Character From a Driving Video Without Skeletons
- Wan-Animate-2: Open Source Character Animation & Camera Control
- NVIDIA ARDY: Open Source Real-Time 3D Motion AI for Film
- GeoState & World from Motion: Open Source 4D Camera AI
Sources
Continue Reading
Video & LipSync
- Video Generator
- Text to Video
- Image to Video
- Start-End Frame to Video
- Draw to Video
- Motion Control
- Video Enhancer
- Video Upscaler
- Video Extend
- Video to Video LipSync
- Audio to Video LipSync
- Image to Video LipSync
- Video FaceSwap
- Seedance 2.5
- WAN 3.0
- Seedance 2
- FLUX 3
- Minimax H3
- Vidu Q3 Pro
- Grok Imagine Video 1.5
- Gemini Omni
- Google Veo 3.1
- Kling 3.0 Pro
- Luma Ray 3.2
- LTX 2.3
- Happy Horse 1.1
- Kling 3.0 Motion
- ByteDance Upscaler
- InfiniteTalk
- InsightFace
Image & Edit
- AI Character
- AI Actor
- Art Generator
- Text to Image
- Image to Image
- LoRA Image to Image
- Draw to Edit
- Image LoRA Training
- Remove Background
- Image Enhancer
- MidJourney 8.0
- Krea 2
- OpenAI GPT Image 2.0
- Nano Banana Pro Ultra
- Nano Banana 2 Lite
- Nano Banana Pro
- Minimax Image
- Clarity Pro
- Z-Image Turbo LoRA
- Krea 2 Turbo LoRA
- Z-Image LoRA Trainer
- Qwen Image LoRA Trainer
- Background Erase Network 2 (BEN2)


