Lance: ByteDance's Unified Video and Image Generation Model (Apache 2.0)

ByteDance Research released Lance on May 18, 2026, a 3 billion parameter open source model that handles text-to-video, text-to-image, video editing, image editing, and multimodal understanding within a single unified architecture. The license is Apache 2.0, confirming commercial use.

One Model, Eight Tasks

Most AI production pipelines stack separate specialist models for each task. The eight tasks Lance handles in a single architecture are: text-to-video, video editing, sequential video editing, structured video planning, visual question answering, video captioning, text-to-image, instruction-based image editing, and image understanding. Covering all eight within a single 3B parameter model is the architectural claim that the paper is designed to support. Each task in the list has specialist models that outperform Lance on that task alone; the case for Lance is the combination under one license at one model size.: one for text-to-video, another for image generation, a third for editing. Lance handles eight distinct tasks in a single architecture: text-to-video generation, video editing, sequential video editing across multiple turns, structured video planning, video understanding including visual question answering and captioning, text-to-image generation, instruction based image editing, and image understanding.

For a filmmaker or solo creator assembling a production workflow, that means one model covers the full pipeline from concept images to edited video clips, under a single license.

The Problem With Specialist Pipelines

A typical AI production workflow in early 2026 routes work across four or five separate models, each maintained by a different organization and licensed under different terms. A short film requiring text-to-video generation, background image creation, and editorial cuts might combine models from three different providers, each requiring separate integration, separate API calls, and potentially separate commercial licensing agreements.

The friction compounds in post production. When a director wants to adjust a clip's background while preserving subject motion, that instruction currently has to be decomposed into separate operations across separate tools. Lance handles the decomposition internally. The filmmaker describes what they want; the model decides which of the eight task heads to apply.

The practical outcome for small teams and solo creators is not just convenience. Reducing the number of models in the pipeline reduces the number of technical integrations that can fail and the number of licenses to track in a commercial production.

Benchmark Results

On VBench, the standard benchmark for video generation quality, Lance scores 85.11 overall, the highest among unified models in its published comparison.

VBench's 16 evaluation dimensions include: subject consistency (how stable objects and characters remain across frames), background consistency (how stable the environment remains), motion smoothness (whether motion flows without jerky transitions), dynamic degree (whether the motion is appropriately active for the described action), imaging quality (overall visual fidelity), and aesthetic quality (how visually appealing the output is independent of prompt adherence). Lance's subject consistency score of 94.52 is among the highest reported for any open source model in its parameter range, unified or specialist. Subject consistency reaches 94.52, background consistency 94.28, and temporal flicker 99.66. Its semantic score of 84.96 indicates strong alignment between text prompts and generated output.

On MVBench, which tests video understanding rather than generation, Lance scores 62.0, again the highest among unified models in the comparison.

Structured Video Planning

Structured video planning is the eighth task in Lance's architecture and the least commonly available in other models. It takes a high-level description of a video, specifically a narrative or sequence description rather than a prompt for a single clip, and generates a structured plan for how to produce it, including segment breakdowns, scene descriptions, and visual continuity guidance.

For short film production, structured planning is the gap between having a script and having a video generation plan. A script describes what happens in human narrative terms. A generation plan describes what each individual clip needs to show, in what order, and how they connect visually. Specialist models for video generation do not produce this kind of structured breakdown. Lance generates it as part of the same model that generates the clips themselves.

Text-to-Video Examples

Video Editing

Lance supports video editing guided by text instructions, covering background transformation, object manipulation, subject replacement, and style transfer. The sequential editing capability allows multiple modifications across linked edits: changing subject, appearance, background, and motion in sequence without regenerating from scratch.

Sequential Editing in Practice

Sequential editing is the capability that matters most for production workflows. A single video clip often needs several distinct modifications before it reaches its final state: a background replacement, then a subject appearance change, then a motion smoothing pass. Without sequential editing, each modification requires generating from scratch, discarding the previous state.

Lance preserves state across sequential edits. Each instruction in a chain modifies the result of the previous one rather than starting over. For a director iterating through visual options on a clip, that means dozens of variations become practical where previously only a handful were. The model maintains consistency in the preserved elements while applying the changes the instruction specifies.

The video understanding capability complements this. Visual question answering and captioning on existing footage lets a filmmaker query what a clip contains before deciding how to edit it, useful for archival work and for maintaining continuity across a long project.

The Dual Stream Architecture Explained

The "dual stream" in Lance's design refers to two parallel processing tracks that operate on the same input simultaneously. One track handles semantic understanding: what the input means, what concepts and objects are present, what relationships exist between them. The other track handles visual synthesis: generating the pixel-level content that expresses those concepts.

Most earlier unified models processed understanding and generation in a single shared pathway, which required the model to balance the two tasks simultaneously. The dual stream approach separates them: the understanding track produces representations that are then used to guide the generation track, rather than the two tasks competing for the same computational capacity in every layer. This is why Lance can handle visual question answering (a purely understanding task) and text-to-video generation (a purely generative task) without degrading either, which unified models with single-pathway architectures struggled to achieve.

How It Works

Lance uses a dual stream Mixture of Experts design that separates semantic understanding from visual generation while processing shared multimodal sequences. Positional encoding is handled by Modality Aware Rotary Positional Encoding (MaPE), which reduces interference between the different types of visual tokens the model processes simultaneously.

The model was trained from scratch using no more than 128 A100 GPUs. A staged multitask training approach with capability oriented objectives and adaptive data scheduling drives the separation between semantic comprehension and visual generation across all eight tasks.

The 3B Parameter Choice

3 billion parameters is large enough to produce high quality output across all eight tasks but small enough to run on local hardware with a single consumer GPU. Models above 7B often require either a data center GPU or significant inference optimization to run at practical speeds on desktop hardware.

ByteDance's choice to publish a 3B model means individual filmmakers with RTX 4090 or 5090 class hardware can run Lance locally without cloud costs. For a commercial production with ongoing generation needs, the difference between a model that runs locally and one that requires per call API costs compounds significantly over a project spanning multiple weeks. The Apache 2.0 license removes any restriction on that kind of deployment.

VBench 85.11: What the Score Means in Practice

VBench measures video generation quality across 16 dimensions including subject consistency, background consistency, motion smoothness, and prompt-following. An 85.11 overall score places Lance above all unified models in its comparison set and competitive with specialist text-to-video models that do not handle the other seven tasks.

The subject consistency score of 94.52 is particularly relevant for production use. Subject consistency measures whether a character or object remains visually stable across frames. Low subject consistency produces flickering or morphing subjects, which is the failure mode that most quickly marks AI generated content as AI generated to a viewer. At 94.52, Lance produces subject consistency comparable to specialist models that focus on that dimension exclusively.

ByteDance's Open Source Video Strategy

Lance arrives after ByteDance suspended the global rollout of Seedance 2.0 in March 2026 following cease and desist letters from five major Hollywood studios over copyright concerns. The two releases sit on opposite sides of the copyright debate: Seedance 2.0 was a closed commercial product accused of training on unlicensed content; Lance is an openly published research model released with full training detail.

ByteDance Research has maintained a consistent pattern of publishing models openly even while its commercial products face legal pressure. The open publication of Lance serves research credibility and allows the community to verify the architecture claims in the paper independently. For commercial studios, the Apache 2.0 license provides the legal clarity that Seedance 2.0's situation complicated.

ByteDance followed Lance in June 2026 with Bernini, a separate 14B model focused on reference guided video editing. The two releases together establish ByteDance Research as one of the most active open source contributors in the AI video generation field, with two major model releases within two weeks covering complementary capabilities: unified multitask generation with Lance and reference-guided editing with Bernini.

The Apache 2.0 Licensing Advantage

Apache 2.0 is among the most permissive open source licenses. It permits use, modification, and distribution for commercial purposes without requiring the derived work to carry the same license. This is more permissive than CC BY-SA 4.0 (which requires derivatives to use the same license) and more permissive than research-only licenses that prohibit commercial deployment entirely.

For a production company integrating Lance into a commercial workflow, Apache 2.0 means no per-use fees, no per-output royalty obligations, and no restriction on the commercial use of generated content. The license covers the model weights and code; it does not cover what the model generates, which is the usual boundary in open source AI licensing.

What It Means for Filmmakers

The practical case for a unified model is workflow compression. A production that needs AI generated backgrounds, character motion clips, and edited footage currently routes work through multiple separate tools and interfaces. Lance consolidates those steps into a single model under a single Apache 2.0 license, removing per-task licensing complexity for commercial productions.

The video understanding capability covers visual question answering and captioning on footage, adding a function specialist generation models typically cannot provide: automated analysis of existing video, useful for continuity checking, scene description, and archival tagging.

ByteDance Research followed Lance with Bernini, a separate 14B model focused on reference guided video editing using a two stage semantic planning architecture, released June 1, 2026.

Lance joins a growing set of open source video generation tools available to filmmakers. MOVA addresses audio synchronized video generation with a different architecture approach, and LTX 2.3 targets high resolution latent diffusion video output. Lance's differentiation is the unified architecture across generation, editing, and understanding in one model. Filmmakers can run AI generated video workflows through AI FILMS Studio's video workspace.

---

Sources

arXiv: Lance: Unified Multimodal Modeling by Multi-Task Synergy
GitHub: bytedance/Lance
Hugging Face: bytedance-research/Lance
Project Page: lance-project.github.io