LongCat-Video-Avatar 1.5: Open Source Multilingual Lip Sync

LongCat-Video-Avatar 1.5 ships two changes that matter: a full audio encoder swap from Wav2Vec2 to Whisper-Large, and step distillation that cuts inference from 20 steps to 8. Both are available under an MIT License, which permits commercial use without royalties or restrictions.

What Changed From 1.0

The original LongCat-Video-Avatar used Wav2Vec2 as its speech encoder. Wav2Vec2 was trained primarily on English audio, which limited accuracy on other languages. Version 1.5 replaces it with Whisper-Large, trained by OpenAI on 680,000 hours of multilingual speech across 99 languages.

Step distillation reduces the diffusion sampling steps from 20 to 8. An INT8 quantized version is also available for lower memory deployments. The generation modes carry over from 1.0: Audio-Text-to-Video (AT2V), Audio-Text-Image-to-Video (ATI2V), and video continuation from an existing clip.

Talking Avatar Samples

Comparison Against Commercial Models

The project benchmarks 1.5 against HeyGen, Kling Avatar 2.0, and OmniHuman-1.5. All three are commercial or closed-weights models. LongCat-Video-Avatar 1.5 is the only one in that comparison group with an MIT License.

Why Whisper-Large Matters for Languages Other Than English

Wav2Vec2's primary training data was English. Phoneme alignment for other languages in version 1.0 was serviceable at best. The shift to Whisper-Large changes this.

Whisper-Large covers 99 languages with training data proportional to actual spoken usage, not skewed toward English. For filmmakers working on dubbing, localization, or multilingual productions, that is the practical difference between a tool that works on one language and a tool that works on a production pipeline. Studios currently pay significant sums to dubbing houses for lip synced localized versions of theatrical releases. A commercially licensed open source model that handles 99 languages with comparable output quality is a direct alternative to that workflow.

None of the model's documentation makes this angle explicit. It follows from the encoder choice.

Multiple Speaker Generation

Multiple audio stream input lets each speaker drive their own lip sync track simultaneously. The model resolves which audio stream maps to which face in the frame, enabling scenes with multiple speakers without separate post processing per character.

Animation and Non Human Subjects

LongCat-Video-Avatar 1.5 generalizes beyond photorealistic faces. The model applies lip sync to anime characters and animals, which expands its application range to animated productions and character work that does not involve human subjects.

What Step Distillation Changes in Practice

Cutting diffusion sampling from 20 steps to 8 is more than a benchmark number. It reduces the compute cost of every generated second by more than half, which is the difference between a model that is fun to experiment with and one that a studio can run at production volume. When a localization pass covers dozens of languages and hundreds of shots, the per clip cost is what decides whether the workflow is viable.

The INT8 quantized build lowers the barrier further. Quantization trades a small amount of numerical precision for a large drop in memory use, letting the model run on hardware that could not host the full precision weights. For independent creators, that is often the gap between needing rented cloud GPUs and running locally on a single consumer card.

The Dubbing Economics No One States Outright

The most disruptive implication of the Whisper-Large swap sits in the localization budget. Theatrical and streaming releases routinely commission lip synced dubs for major markets, and those passes are billed per language by specialist houses. A commercially licensed model that handles 99 languages at comparable quality turns a recurring external cost into an in house pipeline step.

That does not eliminate the dubbing craft. Voice casting, translation, and performance direction still require human judgment, and a model that moves lips convincingly does not choose the right voice or adapt the script. What changes is the mechanical layer. The lip sync itself, historically slow and expensive to align by hand, becomes a fast and repeatable step that runs on the studio's own terms under an MIT License.

Where It Fits in a Production Pipeline

For most productions, LongCat-Video-Avatar 1.5 is not a standalone product but a component. Body motion, framing, and scene composition come from other tools, and this model handles the final layer of making a face speak in the target language. Its support for animated characters and animals means the same step works for a dubbed feature, an explainer video, and a stylized short without a separate tool for each.

The multiple speaker support matters here too. Dialogue scenes with several characters on screen used to require compositing each performer's lip sync separately. Resolving multiple audio streams to multiple faces in one pass removes a tedious manual stage, which is exactly the kind of friction that keeps small teams from attempting ambitious multi character work.

Licensing is the quiet advantage that ties these use cases together. The commercial systems in the project's own comparison keep their weights closed, which means a studio's pipeline depends on a vendor's pricing and availability. An MIT licensed model can be embedded, modified, and shipped inside a product without that dependency, which is often the deciding factor for teams building repeatable production systems.

A Realistic Read on Quality

Benchmarks against commercial systems are encouraging, but they are set by the model's own authors and tend to favor the scenarios the model handles well. Independent evaluation across difficult cases, fast speech, heavy accents, extreme head angles, and long unbroken takes, will determine how far the tool travels beyond demos. Open weights help here too, because they let researchers and studios run those tests themselves rather than trusting a vendor's marketing.

Public access to the weights on Hugging Face and the code on GitHub means that verification will not depend on the authors alone. That openness is what allows a claimed benchmark to become a trusted one, since anyone can reproduce the result or expose its limits.

LongCat Ecosystem Context

LongCat-Video-Avatar 1.5 builds on the same architecture documented in the LongCat Video Avatar guide, which covers the 13.6B DiT framework, Reference Skip Attention, and Cross-Chunk Latent Stitching. The 1.5 release focuses on the audio encoder upgrade and inference speed, not changes to the underlying generation architecture.

For extended duration generation without avatar constraints, LongCat Video's 15-minute coherent generation model addresses temporal consistency across long sequences. For full body motion transfer before lip sync is applied, SCAIL-2 drives body animation from a driving video without skeleton extraction and handles the same animated and nonhuman figures that LongCat-Video-Avatar 1.5 covers for lip sync.

Test lip sync and voice generation workflows with the latest AI models in the AI FILMS Studio voice workspace.

---

Sources

Project Page: LongCat-Video-Avatar 1.5 GitHub: meigen-ai/LongCat-Video-Avatar Hugging Face: meigen-ai/LongCat-Video-Avatar-1.5