NVIDIA GEN3C: Open Source 3D Camera Control Video AI

NVIDIA Toronto AI Lab, GEN3C (arXiv:2503.03751, Apache 2.0)
Share this post:
NVIDIA GEN3C: Open Source 3D Camera Control Video AI
GEN3C (3D-Informed World-Consistent Video Generation with Precise Camera Control, arXiv:2503.03751) from NVIDIA Toronto AI Lab is an open source, 3D cache guided video diffusion model recognized as a CVPR 2025 Highlight and released under the Apache 2.0 license. Available in both a 7B Cosmos Diffusion Transformer checkpoint (nvidia/GEN3C-Cosmos-7B) and a lighter Stable Video Diffusion (SVD) variant, GEN3C maintains an explicit spatiotemporal 3D point cloud cache so virtual camera moves follow exact 6-DoF trajectories without geometry drift or object popping.
Studio availability. GEN3C is not available in AI FILMS Studio. This article covers a public research release. To control camera movement in the Studio, see the models listed on the Motion Control page or open the video workspace.
GEN3C-Cosmos-7B rendering 3D cache point cloud projections into world consistent camera arcs
Single view image unprojected into a 3D cache and rendered along custom 6-DoF dolly and orbit paths
Solving Object Popping in Camera Conditioned Video Diffusion
Standard camera conditioned video generators pass 6-DoF camera extrinsics or Plücker ray embeddings directly into attention layers and rely on the neural network to imagine how 3D geometry rotates across frames. When a camera pans away from an actor or building and returns seconds later, models without explicit 3D memory hallucinate new architecture or alter lighting.
Explicit 3D reconstruction methods try to fit a neural radiance field or Gaussian splat before rendering, but monocular depth estimation errors leave torn edges and missing regions wherever the camera reveals previously occluded areas. GEN3C bridges both paradigms by pairing an explicit spatiotemporal 3D point cloud cache with a generative video diffusion backbone that inpaints disoccluded regions and repairs depth inaccuracies automatically.
GEN3C unprojects monocular depth into a 3D cache and renders novel viewpoints for the video generator. Credit: NVIDIA Toronto AI Lab (Apache 2.0).
Given a single input image or a dynamic monocular video, GEN3C estimates metric pixel depth and unprojects the colored pixels into 3D world space. When the user specifies a new camera path, the 3D cache is rasterized along those exact camera intrinsics and extrinsics.
Because the 3D cache handles perspective projection mathematically, the video diffusion model only needs to synthesize missing disoccluded pixels and refine surface shading.
Spatiotemporal 3D Cache & Multi View Feature Aggregation
To handle both static environments (from one or more photographs) and dynamic moving scenes (from monocular or multi camera video rigs), GEN3C organizes its cache across both space and time.
For a dynamic video of length `T` captured from `N` viewpoints, the spatiotemporal cache maintains an array of `N x T` point clouds. Rendering a target camera trajectory at timestamp `t` projects the `N` point clouds from frame `t` into 2D RGB images and binary disocclusion masks.
Those rendered RGB plates and visibility masks are encoded into latent space and concatenated with the noisy diffusion latents inside `GEN3C-Cosmos-7B`, allowing the transformer to resolve multi view conflicts and keep lighting consistent across 121 frame sequences.
GEN3C spatiotemporal 3D cache pipeline and multi view latent conditioning. Credit: Xuanchi Ren et al., NVIDIA (Apache 2.0).
Hardware, VRAM & Model Variant Matrix
| Checkpoint / Component | Base Architecture | Resolution & Frame Length | Hardware & VRAM | License |
|---|---|---|---|---|
GEN3C-Cosmos-7B |
NVIDIA Cosmos Diffusion-7B DiT | 704x1280 (720p), 121 frames |
28 GB to 40 GB VRAM (or FP8 / offloading on 24 GB) | Apache 2.0 (Code) / NVIDIA Open Model License |
GEN3C-SVD |
Stable Video Diffusion UNet (img2vid-xt) |
576x1024, 25 frames |
16 GB to 24 GB VRAM (consumer RTX 4090) | Apache 2.0 |
Depth Estimator (MoGe / Video-Depth-Anything) |
Monocular metric depth unprojection | Matches input image or video plate | Runs alongside main GPU pipeline | Apache 2.0 / MIT |
Running GEN3C and the Interactive Trajectory GUI Locally
Clone nv-tlabs/GEN3C, create the Conda environment, and download nvidia/GEN3C-Cosmos-7B from Hugging Face. GEN3C includes an interactive 3D browser GUI (proteus) where directors can load an image, orbit or dolly the virtual camera in real time against the point cloud preview, and export the exact trajectory matrix for diffusion rendering.
git clone https://github.com/nv-tlabs/GEN3C.git
cd GEN3C
conda env create --file environment.yml
conda activate gen3c
pip install -r requirements.txt
huggingface-cli download nvidia/GEN3C-Cosmos-7B --local-dir ./checkpoints/GEN3C-Cosmos-7B
Generate a 121 frame camera controlled shot from a single reference image using preset trajectories (zoom_in, orbit_left, spiral), or launch the interactive trajectory editor.
# Single image to 3D camera controlled video with GEN3C-Cosmos-7B
python cosmos_predict1/diffusion/inference/gen3c_single_image.py \
--checkpoint_dir ./checkpoints \
--input_image_path assets/demo/street_plate.png \
--trajectory orbit_left \
--video_save_folder ./outputs
# Dynamic monocular video rerendering along a novel camera path
python cosmos_predict1/diffusion/inference/gen3c_dynamic.py \
--checkpoint_dir ./checkpoints \
--input_video_path assets/demo/actor_walk.mp4 \
--trajectory zoom_in \
--video_save_folder ./outputs
Virtual Production Use Cases & AI FILMS Studio Workflows
In previsualization and virtual production, GEN3C turns a single concept illustration or location scout photo into a multi angle shot list. Because every camera move reads from the same 3D point cloud cache, a wide establishing shot, a medium dolly push, and a reverse angle over the shoulder maintain identical spatial geometry.
For related NVIDIA and open source world and camera guides, read our coverage of NVIDIA Cosmos 3 open world video generation, NVIDIA Lyra 2.0 3D world generation, SANA-WM minute scale world modeling, FaceCam portrait camera control, and SplatAnything and Pi4 3D and 4D scene reconstruction.
To direct camera movement and character motion in the cloud without local CUDA setup, open the Motion Control page or connect image plates and video generators inside the Nodes Graph Editor.
Sources
Continue Reading
Video & LipSync
- Video Generator
- Text to Video
- Image to Video
- Start-End Frame to Video
- Draw to Video
- Motion Control
- Video Enhancer
- Video Upscaler
- Video Extend
- Video to Video LipSync
- Audio to Video LipSync
- Image to Video LipSync
- Video FaceSwap
- Seedance 2.5
- WAN 3.0
- Seedance 2
- FLUX 3
- Minimax H3
- Vidu Q3 Pro
- Grok Imagine Video 1.5
- Gemini Omni
- Google Veo 3.1
- Kling 3.0 Pro
- Luma Ray 3.2
- LTX 2.3
- Happy Horse 1.1
- Kling 3.0 Motion
- ByteDance Upscaler
- InfiniteTalk
- InsightFace
Image & Edit
- AI Character
- AI Actor
- Art Generator
- Text to Image
- Image to Image
- LoRA Image to Image
- Draw to Edit
- Image LoRA Training
- Remove Background
- Image Enhancer
- MidJourney 8.0
- Krea 2
- OpenAI GPT Image 2.0
- Nano Banana Pro Ultra
- Nano Banana 2 Lite
- Nano Banana Pro
- Minimax Image
- Clarity Pro
- Z-Image Turbo LoRA
- Krea 2 Turbo LoRA
- Z-Image LoRA Trainer
- Qwen Image LoRA Trainer
- Background Erase Network 2 (BEN2)


