EditorNodesPricingBlog

GeoState & World from Motion: Open Source 4D Camera AI

November 20, 2026
GeoState & World from Motion: Open Source 4D Camera AI

NVIDIA / GradientSpaces, World from Motion (arXiv:2607.01202) and Tongji University & Tencent, GeoState (arXiv:2610.22343)

Share this post:

GeoState & World from Motion: Open Source 4D Camera AI

GeoState (Tongji University, HKPolyU, and Tencent, arXiv:2610.22343, October 2026) and World from Motion (GradientSpaces/World-from-Motion, NVIDIA researchers, NeurIPS 2026, arXiv:2607.01202) attack the core frustration of AI cinematography: locking physical scene motion independently of the camera lens. GeoState introduces a view decoupled 3D geometry state so autoregressive world models forecast scene evolution without entangling it with camera shake, while World from Motion converts any ordinary monocular video into a freely reshootable dynamic 3D Gaussian Splatting (4DGS) scene that directors can rerender from brand new camera trajectories.

Studio availability. GeoState and World from Motion are not available in AI FILMS Studio. This article covers public open source research releases from late 2026. To direct camera motion and shot angles inside the Studio today, use Kling 3 Motion Control, WAN 3.0, or Seedance 2.5 on the Image to Video page or in the video workspace.

World from Motion turning standard monocular video into dynamic 4D Gaussian Splatting scenes for novel camera views

Why 2D Video Models Struggle with True Reshooting

In a standard 2D video generator, camera movement and subject motion are baked into the same flat pixel grid. If you love an actor's performance in a handheld take but wish the camera had orbited 40 degrees left on a crane, you have to regenerate the entire shot from scratch, losing the exact timing, expression, and lighting of the original take.

GeoState and World from Motion separate what happens in 3D space over time from where the camera happens to sit:

  1. View decoupled geometry states (GeoState, arXiv:2610.22343). Instead of predicting future 2D RGB frames tied to one camera frustum, GeoState encodes a scene tokenizer built on 3D geometry foundation features. An autoregressive world predictor rolls the 3D scene state forward independently of camera motion, and a ray queried geometry decoder renders any user specified or predicted camera path cleanly without compounding camera drift.
  2. Generative 4D Gaussian reconstruction from one camera (World from Motion, liyzzz/World-from-Motion). Starting from a single monocular video clip (shot on an iPhone, cinema camera, or generated by an AI video model), World from Motion builds an initial 4D geometric scaffold, conditions a video diffusion model on dense 4D renderings to hallucinate occluded back sides and fix disocclusion holes, and distills those consistent multi view renders back into one unified, real time renderable dynamic 3D Gaussian Splatting (4DGS) asset.
GeoState view decoupled 3D geometry tokenizer, autoregressive world predictor, and camera ray queried decoder

GeoState architecture decoupling scene geometry evolution from camera ray querying. Credit: Tongji University & Tencent (arXiv:2610.22343).

Entangling camera motion with scene dynamics forces 2D world models to waste capacity re-predicting static background parallax every time the lens pans.

GeoState's ray queried decoder lets directors forecast physical world motion once and inspect it from arbitrary virtual camera tracks.

Hardware, Capabilities, and Workflow Comparison

Framework Input Modality Core 3D/4D Representation Key Filmmaking Output Hardware Tier
GeoState (arXiv:2610.22343) Multi view / RGB frames + camera poses View decoupled 3D geometry state tokens Long horizon 3D world forecasting under arbitrary camera rays 24 GB – 48 GB VRAM research GPU
World from Motion (liyzzz/World-from-Motion) Single monocular RGB video Unified Dynamic 3D Gaussian Splatting (4DGS) Reshootable 4D scene playable at interactive frame rates 24 GB – 48 GB VRAM (RTX 4090 / A6000)

World from Motion bridges real world cinematography and generative AI VFX.

By conditioning a video diffusion model on dense 4D projections before distilling back into dynamic Gaussians, it fills unseen angles behind moving actors cleanly instead of leaving black stretched tears.

World from Motion two stage monocular video to 4D Gaussian Splatting reconstruction pipeline

World from Motion pipeline lifting monocular video into generative multi view video and unified 4D Gaussians. Credit: GradientSpaces / NVIDIA (arXiv:2607.01202).

Parameter Reference Table (World from Motion)

Parameter Recommended Value Effect on 4D Reconstruction
--video_path 24 fps or 30 fps monocular clip (3–8s) Source performance to lift into dynamic 3D Gaussians.
--novel_orbit_deg -45 to +45 degrees Angular range for generative disocclusion synthesis around the hero subject.
--num_gaussians 250k – 600k Higher Gaussian counts sharpen hair, cloth folds, and specular highlights at higher VRAM cost.
--export_format .ply sequence / .splat Standard 4D Gaussian format loadable in Blender, Unreal Engine, or web viewers.

Local Setup with Python (World from Motion)

Clone GradientSpaces/World-from-Motion and pull the official Hugging Face weights from liyzzz/World-from-Motion:

git clone https://github.com/GradientSpaces/World-from-Motion.git
cd World-from-Motion
pip install -r requirements.txt

huggingface-cli download liyzzz/World-from-Motion --local-dir ./checkpoints/world-from-motion
python reconstruct_4d.py \
  --video_path ./footage/dancer_handheld_take03.mp4 \
  --checkpoint_dir ./checkpoints/world-from-motion \
  --camera_trajectory ./trajectories/crane_orbit_35deg.json \
  --output_dir ./outputs/dancer_4dgs_reshoot

Filmmaker Production Playbook

  1. Fix framing mistakes in post without booking a reshoot. Lift a locked handheld or AI generated take through World from Motion into a 4D Gaussian asset, then animate a smooth dolly push or Dutch angle in Blender or Unreal Engine.
  2. Pair with SplatAnything and Pi4 for metric scale. Extract metric camera poses and dense 4D point tracks first so your novel camera trajectory stays locked to real world floor planes without floating feet.
  3. Avoid extreme 180 degree reveals on occluded faces. Stay within ±45 to ±60 degrees of the original camera frustum when reshooting close up dialogue so generative disocclusion stays faithful to the actor's true likeness.

Related Reading


Sources