EditorNodesPricingBlog

Go with the Flow & LeviTor: Open Source Motion Video AI

November 17, 2026
Updated: November 18, 2026
Go with the Flow & LeviTor: Open Source Motion Video AI

Ryan Burgert et al., Eyeline Studios & Netflix (CVPR 2025)

Share this post:

Go with the Flow & LeviTor: Open Source Motion Video AI

Go with the Flow (Motion-Controllable Video Diffusion Models Using Real-Time Warped Noise, arXiv:2501.08331) from Netflix Eyeline Studios and LeviTor (3D Trajectory Oriented Image-to-Video Synthesis, arXiv:2412.15214, CVPR 2025 Highlight) give filmmakers direct spatial and depth control over how objects move inside generated video shots. Go with the Flow (Eyeline-Labs/Go-with-the-Flow) steers CogVideoX-5B and Wan 2.1 by warping the initial Gaussian latent noise along optical flow fields without modifying the base model architecture. LeviTor (hlwang06/LeviTor) combines 2D K-means clustered object masks with metric depth values so characters and props can orbit around one another in 3D space with accurate mutual occlusion.

Studio availability. Go with the Flow and LeviTor are not available in AI FILMS Studio. This article covers public research releases. To control motion and animate stills in the Studio, see the models listed on the Motion Control and Image to Video pages.

LeviTor 3D trajectory oriented image-to-video synthesis handling foreground and background occlusions

How Go with the Flow Controls Motion via Real Time Warped Noise

Most motion controllable video diffusion models add heavy ControlNet branches or spatial adapter layers that must be retrained for every new base model. Go with the Flow eliminates adapter bloat by encoding motion directly into the structure of the starting latent noise.

  1. Distribution preserving noise warping. Standard video diffusion models expect independent and identically distributed Gaussian noise at each frame. Naively warping noise pixels along an optical flow field introduces bilinear interpolation blur and breaks Gaussian variance. Go with the Flow introduces a real time tree based pixel subdivision and consolidation algorithm on the GPU that warps noise along arbitrary 2D flow fields while keeping every frame strictly Gaussian.
  2. Cut and drag interactive animation GUI. Using the bundled Gradio cut and drag editor (cut_and_drag_gui.py), a storyboard artist cuts out one or more polygons from a starting frame, drags them along paths across the canvas, scales or rotates them, and exports a coarse animation template alongside an automatic optical flow field.
  3. Noise degradation (gamma) slider for rigidity control. At inference time, mixing a controlled fraction of fresh random noise (--degradation 0.5) into the warped noise lets the director dial between rigid camera/object adherence (0.0) and organic fluid secondary motion (0.7).
Go with the Flow optical flow noise warping and CogVideoX fine tuning pipeline

Go with the Flow warps initial Gaussian noise along optical flow fields with zero architectural changes to the base DiT. Credit: Eyeline Labs (arXiv:2501.08331).

Because the base diffusion architecture never changes, fine tuning `CogVideoX-5B-I2V` or `CogVideoX-2B-T2V` on warped noise requires zero extra parameters and zero inference latency overhead.

The same noise warping script also works for video-to-video motion transfer by extracting `RAFT` optical flow from a driving live action plate and applying it to a new prompt or first frame.

How LeviTor Solves 3D Depth Ordering and Mutual Occlusion

2D drag trajectories work well when objects move side to side in a single plane, but they fail when one character walks behind a pillar or two vehicles cross paths in depth. LeviTor extends trajectory control to three dimensions without requiring users to sculpt 3D meshes.

During training, LeviTor segments object masks with `SAM`, samples K-means clustered keypoint centers on each object, and pairs those clusters with monocular depth values to form sparse 3D control points.

At inference, the user draws a 2D path and assigns start and end depth values (`z`). LeviTor projects the clustered 3D points into depth ordered 2D masks so the closer object naturally occludes the farther object when their paths intersect.

LeviTor K-means clustered 3D trajectory and depth mask conditioning pipeline

LeviTor pairs K-means clustered object regions with depth values to resolve 3D occlusion ordering. Credit: Hao Wang et al., Ant Group (CVPR 2025).

Hardware, VRAM & Model Comparison

Model / Checkpoint Base Architecture Control Signal Hardware & VRAM License
Go with the Flow (CogVideoX-5B-I2V) 5B DiT fine tuned on warped Gaussian noise Cut and drag polygons or RAFT optical flow + --degradation 11 GB VRAM (with CPU offload) to 35 GB VRAM Open Weights (Eyeline-Labs/Go-with-the-Flow)
Go with the Flow (CogVideoX-2B-T2V) 2B DiT text-to-video with warped noise Optical flow warped noise from driving video 8 GB to 16 GB VRAM Open Weights
LeviTor (hlwang06/LeviTor) Stable Video Diffusion XT (512x512 / 576x1024, 16 frames) 2D drag trajectories + depth values (z) + SAM object clusters 16 GB to 24 GB VRAM Open Research Weights

Running Cut and Drag Motion Control Locally

To animate a still storyboard frame with Go with the Flow on a single GPU, clone Eyeline-Labs/Go-with-the-Flow, launch the cut and drag GUI to sketch your object trajectories, warp the noise, and run cut_and_drag_inference.py.

git clone https://github.com/Eyeline-Labs/Go-with-the-Flow.git
cd Go with the Flow
pip install -r requirements_local.txt

# Step 1: Launch interactive Cut and Drag polygon trajectory GUI
python cut_and_drag_gui.py

# Step 2: Warp initial Gaussian noise along the exported animation clip
python make_warped_noise.py outputs/my_cut_and_drag_cartridge.mp4 \
  --output_folder noise_warp_output_folder

# Step 3: Render 49 frame 480p video with CogVideoX-5B-I2V (runs in 11 GB VRAM with offload)
python cut_and_drag_inference.py noise_warp_output_folder \
  --prompt "A golden retriever running across a sunlit meadow while the camera pans left." \
  --degradation 0.5 \
  --output_mp4_path outputs/final_shot.mp4

To run LeviTor's interactive 3D depth trajectory Gradio app locally, download stabilityai/stable-video-diffusion-img2vid-xt and hlwang06/LeviTor and launch gradio_demo.py.

git clone https://github.com/robbyant-research/LeviTor.git
cd LeviTor
pip install -r requirements.txt

python scripts/gradio_demo.py

Storyboard Blocking & AI FILMS Studio Workflows

Cut and drag warped noise and 3D depth trajectories mirror how a film director marks blocking arrows on a printed storyboard. Instead of writing complex text prompts to describe which character crosses in front of another, the artist drags the foreground subject along a shallow depth curve and sends the background vehicle along a deep counter path.

For related open source motion control guides, read our coverage of Time-to-Move (TTM) cut and drag motion control, Wan-Move point trajectory video generation, RealisMotion 3D world space motion control, and MiniMax Music 3 and GGUF open source film score AI.

To animate storyboard stills and control character movement in the browser without local CUDA setup, visit the Motion Control and Image to Video pages or connect custom motion pipelines inside the Nodes Graph Editor.


Sources