OmniDirector: Camera Motion Cloning from Multi Shot Videos

OmniDirector is an open source method from Kuaishou's Kling team that clones camera motions from reference videos and applies them to animate static images. Released under the MIT license on June 11, 2026, it handles dolly zooms, bullet time effects, aerial fly throughs, and dramatic descents without requiring paired training data matching camera movements to specific scenes.

The model comes from the Kling research team at Kuaishou Technology, the same group behind Kling AI motion control and the broader Kuaishou AI video platform. Jiwen Liu and Shujuan Li are equal first authors, with additional collaborators from Tsinghua and Peking universities. The paper is arXiv:2606.13432.

The key advance OmniDirector makes over previous camera control systems is the elimination of the paired training data requirement. Most camera control research assumes matched video pairs showing the same scene with different camera movements. OmniDirector sidesteps that requirement entirely through the Camera Grid Representation, which encodes motion without reference to content.

Camera Grid Representation

OmniDirector introduces a Camera Grid Representation. Rather than learning from matched video pairs showing the same scene from different camera positions, it renders camera poses from a reference video into empty 3D space as a visual encoding. The model reads that grid and transfers the encoded trajectory to animate a source image.

This decoupling of motion from content eliminates the need for cross paired training data. Other camera control methods require matched examples tied to specific scene types. OmniDirector extracts camera behavior from any reference video regardless of what it depicts, making it applicable across portraits, architecture, wildlife, and AI generated imagery. A Hierarchical Prompt Expansion Agent coordinates character action and camera motion for multimodal control.

The visual encoding step is what makes cross domain transfer possible. When the model learns from a grid of camera positions rather than from paired content, the motion pattern is separated from the scene type at the representation level, not through post hoc augmentation.

Rendering camera poses into empty 3D space has a direct implication for production. The reference video does not need to contain the same type of scene as the target image. A wildlife documentary can supply the camera trajectory for a portrait shoot. A sports broadcast can provide the arc for an architectural sequence.

The Hierarchical Prompt Expansion Agent adds a text interface on top of the visual trajectory. Rather than specifying a camera path through the grid representation alone, the user can describe a combined character action and camera movement in natural language. The agent expands that description into the full multimodal control signal.

Dynamic Motion

The Dynamic Motion capability transfers camera trajectories from a single reference shot to a new scene. The model preserves scene geometry through aggressive moves, including rapid pans and fast dolly sequences, that would lose spatial coherence without the grid representation anchoring the trajectory.

Aggressive camera moves are where most AI video generation systems lose spatial coherence. The spatial references that hold a generated scene together during slow, stable sequences are not sufficient when the camera is moving fast. OmniDirector's grid representation provides a stronger anchor because the trajectory is encoded spatially rather than inferred from frame transitions.

The dolly sequences in the Dynamic Motion demo maintain background depth consistency across the full move. As the camera travels forward, the apparent relationship between near and far objects in the generated scene must scale correctly. The grid representation enforces this through the spatial encoding rather than leaving it to the model to infer.

Multi Shot Camera Cloning

Multi shot cloning applies a camera sequence across several consecutive shots, preserving compositional logic and transitions between angles. The model keeps scene coherence as the virtual camera cuts between positions, producing visual continuity that normally requires coordinated planning across multiple physical takes.

This capability connects directly to the challenge of multi shot video generation, where maintaining visual grammar across cuts is one of the hardest problems in AI filmmaking. OmniDirector approaches it from the camera side rather than the character or scene side.

The multi shot mode addresses a practical gap in AI video for narrative filmmaking. Single shot generation is well supported across multiple models. Generating a coherent visual language across a sequence of shots, each cut from a different angle, has been much harder. OmniDirector's camera cloning approach solves it through trajectory transfer rather than scene understanding.

For a director working through a sequence in pre-production, the multi shot capability means camera plans from reference films can be applied directly to previsualization footage. The compositional grammar of an existing sequence becomes transferable, not just a source of inspiration.

Scene Generalization

OmniDirector generalizes across different scene types without retraining for each domain. A camera trajectory extracted from a cityscape applies equally to a portrait, an animal shot, or AI generated content, because the camera grid encodes motion independently of scene content.

This cross domain capability is significant for practical filmmaking workflows. A director can reference any existing footage for its camera language, regardless of whether the content or setting matches the target scene. Shot vocabulary from one production can inform the visual grammar of an entirely different one.

The scene generalization results also demonstrate that OmniDirector's training data distribution is not the limiting factor for application range. Most AI video tools perform well on content types that match their training distribution and degrade on others. OmniDirector's camera grid architecture is content agnostic, which means its camera transfer quality stays consistent across scene types.

Productions working across multiple visual styles benefit most from this. A single reference clip can supply the camera trajectory for photorealistic footage, stylized animation reference, and AI generated content in the same project. The camera language remains consistent even when the content style changes.

Special Camera Effects

The special camera effects capability covers named cinematic techniques that are among the most technically demanding shots in physical production. The Hitchcock zoom, also called a dolly zoom, simultaneously moves the camera forward while zooming out, keeping the subject at the same apparent size while distorting the background. Bullet time effects freeze a moment and orbit the camera around it.

Other camera control research has approached these effects through separate models or constrained motion primitives. OmniDirector treats them as instances of the same camera cloning problem. A reference video that contains a dolly zoom provides the trajectory; the model executes it on any target image. Related approaches to camera control in AI video include CamCloneMaster, which focuses on extracting and reapplying movements from AI generated video sequences. AMAP-ML's DreamX-World 1.0 extends camera controlled motion to full world generation, applying six degree of freedom navigation to environments generated directly from text prompts.

Named cinematic techniques matter in production because they have shared cultural references. A director asking for a "Hitchcock zoom" on set is communicating a specific visual intention that the crew immediately understands. OmniDirector's ability to generate these effects from reference videos means that shared vocabulary now applies to AI generated imagery as well.

The bullet time effect is particularly significant. It requires a camera path that circles a frozen subject, an effect that in physical production needs either a circular rig of synchronized cameras or significant post production compositing. As a trajectory in OmniDirector, it is as accessible as any other camera move.

MIT License and Paper

OmniDirector is released under the MIT license, permitting commercial use. The paper is published on arXiv (2606.13432) alongside a code repository on GitHub. As of the June 11 release date, the repository contains the project documentation and reference implementation structure. The research comes from the same Kling team that has published consistently in AI video generation over the past two years.

The arXiv paper (2606.13432) provides the full technical specification of the Camera Grid Representation and the Hierarchical Prompt Expansion Agent. Production technical directors evaluating the system for deployment can review the architecture details and training methodology before committing to an integration.

Kuaishou's Kling research team has published multiple papers on camera control and AI video over the past two years. OmniDirector is part of a consistent research line rather than an isolated release. The same team's work on motion control underpins the Kling AI platform's commercial camera features, giving the research a direct production validation path.

The MIT license removes legal uncertainty for commercial production use. Filmmakers and studios can incorporate OmniDirector's camera cloning into pipelines without restrictions on usage or derivative work. The open source release also means the community can verify the system's behavior and adapt it for specialized production contexts.

Filmmakers experimenting with camera motion in AI video generation can use tools available today through AI FILMS Studio.

---

Sources

arXiv: OmniDirector: General Multi-Shot Camera Cloning without Cross-Paired Data GitHub: lisj575/OmniDirector Project Page: OmniDirector