Holo-World: Unified Camera, Object, and Weather Control for Video Generation

Researchers at Alibaba Group published Holo-World on June 16, 2026, introducing an open source video generation model that handles camera trajectory, object motion, and weather conditions inside a single framework. Prior video control tools treated these as separate problems requiring separate modules. Holo-World addresses all three through a unified architecture, and it does so under an Apache 2.0 license that permits commercial use.

What Holo-World Controls

The model gives a filmmaker or technical artist control over three dimensions at once. Camera trajectory defines where the virtual camera moves through the scene. Object motion tracks how subjects within the frame behave independently of the camera. Weather control applies atmospheric conditions as a parametric layer over the entire output.

Each control operates independently of the others, which means changing the weather does not alter the camera path or subject motion. A scene shot with a forward dolly through a sunlit street can be re-rendered with snow, rain, fog, or cloud cover without re-running any other part of the generation process. For previsualization and iterative shot design, that decoupling removes a significant technical barrier.

The paper reports that Holo-World maintains temporal consistency across all three control channels simultaneously. That is the practical claim the architecture is built to support, and it is what distinguishes this approach from layering separate post production effects onto an existing video.

The Three Stream Architecture

Holo-World separates its control signals into three parallel streams before merging them at the generation stage. The camera stream encodes trajectory information from 3D point clouds. The object stream uses optical flow to represent how subjects move relative to the camera. The third stream carries the atmospheric or world state, which includes the five weather modes the model supports.

The point cloud encoding for camera control is notable because it operates in 3D space rather than 2D image space. Most camera control approaches in video diffusion models work with 2D projections of camera poses, which introduces ambiguity when the camera path curves or the depth of field changes significantly. Working from a 3D point cloud gives the model a more precise spatial reference. The optical flow object stream functions similarly: it carries motion in full 2D vector form rather than reduced keypoint representations, preserving more detail about how subjects deform or rotate across frames.

Weather Variants from a Single Source

The weather control system supports five atmospheric conditions: Snow, Rain, Fog, Cloud, and Sun. Each is applied as a learned atmospheric layer conditioned on the scene geometry and lighting from the source video. The result is that weather changes adapt to the scene rather than adding a generic overlay.

The samples below show what this produces in practice. The first two outputs show the camera path and rendered color output for the same scene.

The origin clip below is the source footage from which all weather variants are generated. The first frame image follows it as a static reference.

The four weather variants below are generated from that same origin clip. Each one applies a distinct atmospheric condition while preserving the camera path and subject motion.

Production Applications for Filmmakers

The most immediate application is previsualization. A director who wants to shoot a scene across multiple weather conditions has historically needed either on-location reshoots or extensive post production compositing. Holo-World opens a third path: generate the weather variants from a single reference clip during pre-production, evaluate them against the camera path, and commit to a shooting plan with that visual evidence already established.

The same workflow extends to animation and virtual production. Because the camera and object controls are defined in 3D space, the model outputs are compatible with depth maps and point cloud data that game engines and virtual production pipelines already consume. That makes Holo-World more immediately useful in mixed pipeline environments than tools that output only composited 2D video.

For teams interested in world state control more broadly, the DreamX interactive world model takes a different approach to the same problem, focusing on interactive scene manipulation rather than atmospheric control. Together, these two open source releases define a direction in video generation research where scene context (camera, objects, atmosphere) becomes as controllable as subject appearance.

The weights for Holo-World are hosted on the project's GitHub repository. The model runs on standard video diffusion infrastructure. Filmmakers who want to experiment with text-to-video and image-to-video generation using currently available commercial tools can start directly in AI FILMS Studio.

---

Sources

arXiv: Holo-World: Unified Camera, Object, and Weather Control for Video Generation GitHub: suxi123/Holo-World Project Page: xiangchenyin.github.io/Holo-World