EditorNodesPricingBlog

RealisMotion: Open Source 3D World-Space Motion Control

November 15, 2026
RealisMotion: Open Source 3D World-Space Motion Control

DAMO Academy / Alibaba Group, RealisMotion (ICML 2026)

Share this post:

RealisMotion: Open Source 3D World-Space Motion Control

RealisMotion (JingyunLiang/RealisMotion) is an open source decomposed human motion control and video generation framework developed by DAMO Academy at Alibaba Group and accepted at ICML 2026 (arXiv:2508.08588). While traditional pose driven character animators lock the foreground actor, camera path, and background plate to a single driving video's 2D skeleton, RealisMotion builds a ground aware 3D world coordinate system that explicitly decouples foreground subject appearance, background video, 3D walking or running trajectory, and local body action.

Studio availability. RealisMotion is not available in AI FILMS Studio. This article covers a public research release. To generate motion controlled video in the Studio, see the models listed on the Motion Control page or in Studio Nodes.

RealisMotion decomposed human motion control placing a reference subject onto a custom 3D trajectory and action sequence

How RealisMotion Decouples Subject, Background, Trajectory, and Action

In standard human image animation, transferring a dance or stunt clip onto a character in a new scene fails whenever the camera angle, ground slope, or walking path differs from the driving footage. RealisMotion solves this by lifting human motion into a ground aware 3D world coordinate system before conditioning a Diffusion Transformer.

  1. Ground aware 3D world coordinate unprojection. Using GVHMR (SMPL-X estimation in world coordinates) and DepthPro focal length calibration, RealisMotion estimates the 3D ground plane of the target background video. User drawn 2D pixel trajectories are unprojected into 3D world coordinates with automatic speed alignment, stride matching, and global body orientation adjustment along the path tangent.
  2. Mix and match 3D motion editing. Once inside the 3D world coordinate space, directors can combine any body action from a motion bank (or a text to motion generator) with custom 3D trajectories (edit_trajectory, edit_trajectory_as_heart, affine_transform, or edit_trajectory_kickoff), while preserving the target actor's SMPL-X body shape parameter beta (or interpolating between adult SMPL-X and child SMIL-X templates via --kid).
  3. Spatially aligned multi buffer rendering. The edited 3D mesh sequence is projected back into camera space as spatially aligned depth maps, surface normal maps, vertex color semantic maps (cs_map.mp4), and HaMeR 3D hand meshes (hamer.mp4) with depth tested hand occlusion masking.
  4. Element wise DiT token injection. Inside the Diffusion Transformer, the foreground reference subject is injected as full attention tokens, the masked background video is concatenated along the channel dimension (using random background masking during training to prevent ghost duplicate actors), and the 3D trajectory and action buffers are injected via spatial addition.
RealisMotion 3D trajectory and global orientation control comparison

RealisMotion unprojects 2D paths into 3D ground coordinates, aligning body orientation and stride speed to the scene perspective. Credit: DAMO Academy.

RealisMotion action control and body shape preservation comparison

Combining SMPL-X semantic vertex maps with HaMeR hand meshes preserves limb geometry and finger articulation during complex actions. Credit: DAMO Academy.

RealisMotion T2V decomposed scene editing replacing character actions and trajectories inside a dynamic rooftop video plate

Checkpoints, Benchmarks & Hardware Matrix

RealisMotion provides two inference backbones on Hugging Face alongside the Trajectory100 evaluation dataset (jingyunliang/Trajectory100).

Checkpoint / Resource Hugging Face ID Supported Control Modes Target Hardware & Acceleration License / Status
RealisMotion T2V jingyunliang/RealisMotion Fully decomposed foreground subject, dynamic background video (V2V), 3D trajectory, and action control Single 40 to 80 GB GPU (H20 / A100 / H100) or 8 GPU torchrun; supports --enable-teacache and --save-gpu-memory Academic Research Release
RealisMotion I2V theFoxofSky/RealisDance-DiT Single reference image animation driven by 3D world space cs_map.mp4 and hamer.mp4 buffers Single 24 to 80 GB CUDA GPU with FlashAttention-3 and TeaCache Academic Research Release
Trajectory100 Benchmark jingyunliang/Trajectory100 100 test sequences evaluating 3D trajectory accuracy, orientation alignment, and action fidelity Evaluation benchmark dataset Open Research Dataset

Running RealisMotion 3D Motion Editing and Inference Locally

Clone the repository, install FlashAttention-3, and download the jingyunliang/RealisMotion checkpoint from Hugging Face.

git clone https://github.com/JingyunLiang/RealisMotion.git
cd RealisMotion
conda create -n realismotion python=3.10 -y
conda activate realismotion
pip install -r requirements.txt

huggingface-cli download --resume-download jingyunliang/RealisMotion \
  --local-dir pretrained_models/RealisMotion

To edit a character's 3D trajectory in world coordinates using GVHMR and render the conditioning maps (cs_map.mp4 and mask.mp4), run hmc/realismotion_render_demo.py, then pass the generated motion folder to inference_t2v.py.

# Step 1: Unproject custom 2D waypoints into 3D ground-aware world coordinates
export PYTHONPATH="YOUR_PATH/GVHMR/hmr4d:$PYTHONPATH"
python hmc/realismotion_render_demo.py \
  --video inputs/example_video/justin.mp4 \
  --motion_path inputs/motion_bank/tstageboy \
  --reference_path inputs/demo/justin \
  --output_root inputs/demo \
  --window_size 25 \
  --repeat_smpl 18 51 8 \
  --edit_type edit_trajectory \
  --edit_trajectory_args 815 500 952 1050 1192 502 \
  -s

# Step 2: Run decomposed T2V video generation with TeaCache acceleration
CUDA_VISIBLE_DEVICES=0 python inference_t2v.py \
  --ref_path inputs/demo/internalaffairs/0_input_video.mp4 \
  --ref_cs_map_path inputs/demo/internalaffairs/cs_map.mp4 \
  --ref_mask_path inputs/demo/internalaffairs/mask.mp4 \
  --background_path inputs/demo/internalaffairs/0_input_video.mp4 \
  --background_mask_path inputs/demo/internalaffairs/mask.mp4 \
  --motion_folder inputs/demo/internalaffairs_falldown_internalaffairs \
  --background_num_frames 97 \
  --motion_num_frames 97 \
  --prompt "Two figures on a city rooftop under an overcast sky, dramatic urban harbor backdrop." \
  --enable-teacache \
  --save-dir ./outputs

For related character animation and trajectory pipelines, compare RealisMotion with Wan-Animate-2 character animation and viewpoint control, NVIDIA ARDY real time 3D motion generation, and Time-to-Move (TTM) motion control.

Motion Control and Character Blocking in AI FILMS Studio

RealisMotion requires local GVHMR 3D human mesh estimation, DepthPro calibration, and SMPL-X rendering passes. To animate characters, transfer actor performances, or block motion trajectories directly in the browser, open the video workspace or build multi stage character and camera graphs in the Nodes Graph Editor.


Sources