EditorNodesPricingBlog

FlowHMR: Open Source 3D Human Motion Capture

November 21, 2026
FlowHMR: Open Source 3D Human Motion Capture

AI FILMS benchmark chart from FlowHMR paper (arXiv:2610.03691)

Share this post:

FlowHMR: Open Source 3D Human Motion Capture

Markerless motion capture from single camera video usually produces floating feet, sliding steps, and impossible joint angles that break CG character rigs. FlowHMR (fafsaf1/flowhmr-0.46B), published in October 2026 (arXiv:2610.03691), solves depth ambiguity by treating motion capture as a video conditioned flow matching generation problem refined with Group Relative Policy Optimization (GRPO) physics rewards. On the new 4,000 clip Wild-4K benchmark, FlowHMR hits an 82.47% physics tracking success rate, beating the strongest prior baseline GVHMR (62.82%) by nearly 20 percentage points, and exports directly to FBX and NPZ.

Studio availability. Try the public Hugging Face demo below; FlowHMR is not part of a paid AI FILMS Studio plan (use Kling 3.0 Pro Motion Control in the video workspace for example instead).

Try FlowHMR Live

Upload any single actor or multi actor clip (up to 900 frames / 30 seconds at 30 FPS) into the official FlowHMR Space below to recover global 3D motion, inspect synchronized video and skeleton playback, and export FBX or NPZ rig data.

Architecture and Technical Design

Earlier monocular human mesh recovery systems regressed 3D poses directly from 2D pixels using geometric losses. Because single camera footage is ambiguous in depth, direct regression averages across valid hypotheses, causing limbs to clip and feet to hover above the floor. FlowHMR instead pretrains a compact 0.46B flow matching generative model conditioned on per frame visual tracks extracted by SAM 3D Body (DINOv3) and optional camera poses estimated by VGGT-Omega.

FlowHMR 0.46B video to physics plausible 3D SMPL-H motion capture pipeline diagram

AI FILMS pipeline diagram from fafsaf1/flowhmr-0.46B & arXiv:2610.03691

Pretraining generates diverse 3D motion candidates, but not every sample obeys gravity or matches the actor's exact silhouette. The authors therefore post train the flow model with GRPO using two joint rewards: a video fidelity reward that enforces alignment with the filmed performance, and a physics tracking reward that scores whether a simulated physics controller can actually execute the recovered motion without falling.

That dual reward shifts the output distribution toward grounded footsteps, solid momentum transfer, and clean root trajectories across dynamic camera moves.

Hardware, VRAM, and License Matrix

Component License Hardware / Runtime Notes Output Format
FlowHMR Source Code (flowhmr/flowhmr) Apache 2.0 PyTorch 2.8 / CUDA 12.8 recommended Python CLI + Gradio UI
flowhmr-0.46B Weights (flowhmr_latest) FlowHMR Non Commercial License ~12 to 16 GB VRAM (with SAM 3D Body + VGGT-Omega) SMPL-H NPZ & FBX
Companion Vision Stack Third party model licenses YOLOX-L, sam-3d-body-dinov3, VGGT-Omega 30 FPS chunked tracks
SMPL-H Neutral Body Model SMPL-H License (separate download) Required for mesh reconstruction and FBX rigging .npz body asset

Take note of the split license structure. The FlowHMR codebase is Apache 2.0, while the flowhmr_base and flowhmr_latest weights carry a non commercial research, educational, and non profit license. Commercial studios must contact the authors for production weight licensing and maintain valid SMPL-H commercial rights.

Parameter Reference

Parameter Space Default Production Guidance
num_steps 20 Default flow matching denoising steps for clean global trajectories
cfg_scale 1.0 Fixed CFG 1 matches the GRPO post trained distribution
Max frame limit 900 frames (30s at 30 FPS) Split longer takes at scene cuts for cleanest camera estimation
sam_chunk 32 (hands 64) Controls SAM 3D Body tracking memory footprint
vggt_interval 10 Frame stride for VGGT-Omega moving camera estimation
FLOWHMR_SPACE_CAMERA 0 (static) or 1 (moving) Enable VGGT-Omega for handheld or dolly footage; disable for tripod shots

Local Python and CLI Setup

Clone the Apache 2.0 repository, prepare the PyTorch 2.8 / CUDA 12.8 environment, and download the weights along with your licensed SMPL-H body model.

git clone https://github.com/flowhmr/flowhmr.git
cd flowhmr
hf download fafsaf1/flowhmr-0.46B --local-dir ./checkpoints/flowhmr_latest

# Launch the local Gradio interface (after placing SAM 3D Body, YOLOX, and SMPL-H assets)
python tools/hf_space/app.py

The repository also includes tools/hf_space/build_bundle.py for packaging self hosted Gradio instances and supports pinned FBX SDK wheels on Linux x86_64 (Python 3.10) for automated FBX animation export into Blender, Maya, or Unreal Engine.

Filmmaker Production Playbook

  1. Capture stunt and blocking references on a phone. Directors can shoot an actor rehearsing a fight beat or stairwell run on a standard 30 FPS smartphone, drop the clip into FlowHMR, and export an FBX skeleton with physically grounded foot contacts straight into Blender or Unreal Engine for camera previsualization.
  2. Switch off VGGT for locked off tripod plates. If your rehearsal camera sits on a tripod, keep static camera mode active (FLOWHMR_SPACE_CAMERA=0). Skipping moving camera estimation eliminates subtle ground plane drift and cuts processing time.
  3. Isolate one performer in multi actor shots. On clips with multiple people in frame, FlowHMR pauses after person detection so you can click the exact actor to track before GPU motion generation runs.
  4. Drive cloud video generation with motion control. When you want to transfer real actor movement directly onto stylized AI video without 3D rigging, use Kling 3.0 Pro Motion Control on our motion control page inside the video workspace.

Related Reading


Sources