NAVA: Joint Audio-Video Generation from a Single Prompt

NAVA (Native Audio-Visual Alignment for Generation) is an open source model from Baidu's ERNIE Team that generates synchronized 720p video and stereo audio from a single text prompt. Released May 28, 2026 on arXiv with Apache 2.0 licensing, it is the first model in this class to deliver joint audio and video output from one prompt without processing audio as a separate post generation step.

The model runs at 6.3 billion parameters and achieves inference in roughly one minute on an 8 GPU setup.

What NAVA Does

You give NAVA a text prompt. It returns video and audio together, synchronized at the generation level. The audio is not added after the video is rendered. Both streams emerge from the same model pass.

The output supports dual channel stereo, multi speaker timbre control, and language guided camera direction. Resolution targets 720p. The model handles a wide range of prompt types: nature scenes, music performances, dialogue clips, and abstract motion.

The Align-then-Fuse Architecture

NAVA is built on the Wan 2.2 backbone and uses an architecture the authors call Align-then-Fuse MMDiT (Multi-Modal Diffusion Transformer). Audio and video are processed in separate streams first, then merged at the diffusion transformer level before generation completes.

This is the opposite approach to models that generate video and then synthesize matching audio as a post-processing pass. Because both modalities share the same latent representation during generation, events in the audio are causally linked to events in the video frame by frame, not aligned retrospectively.

That structural difference explains NAVA's Verse-Bench Sync-C and Sync-D scores, which measure audio and video synchronization and distance. The paper reports new SOTA results on both metrics, as well as on video quality and audio word error rate, using 2 to 5 times fewer parameters than open source baselines at comparable quality.

Output Examples

Verse-Bench Results

NAVA sets a new SOTA on the Verse-Bench evaluation suite across four metrics: Sync-C (audio and video synchronization), Sync-D (audio and video distance), video quality, and audio word error rate. The ERNIE Team reports these results using 2 to 5 times fewer parameters than the open source baselines they compare against.

The efficiency figure matters for production use. A 6.3B parameter model that runs on 8 GPUs in under a minute is meaningfully closer to practical deployment than models requiring far larger compute budgets to reach comparable synchronization scores.

License and Access

NAVA is released under Apache 2.0, which permits commercial use. Weights are available on HuggingFace under ernie-research/NAVA. The paper is on arXiv at 2605.30073.

The Wan 2.2 backbone that NAVA extends is already a well-documented architecture for character animation and replacement. NAVA adds native audio output on top of that foundation, opening production workflows where synchronized audio and video need to be generated together rather than assembled in post.

Why Generating Audio and Video Together Is Hard

Most existing video models treat audio as a separate problem. The standard workflow is to generate video first, then synthesize or retrieve audio to match it. The result is timing mismatches where a clap in the video arrives slightly after the associated sound, dialogue audio does not align with lip movement, and footsteps land on silent frames. NAVA avoids this by generating both modalities from the same denoising pass.

The approach of adding audio after the fact also fails at fine grained synchronization. A viewer watching a music performance can detect misalignment between audio and video within 50 milliseconds, below the threshold of conscious awareness. Separate generation pipelines introduce timing errors at that scale even when the audio and video appear reasonable in isolation.

The Wan 2.2 Backbone

NAVA builds on Wan 2.2, the open source video model released by Alibaba. Building on an existing competitive video backbone means NAVA's video quality baseline is already strong before the audio layer is added. The researchers added synchronized stereo audio generation without retraining the video backbone from scratch, which is why the 6.3 billion parameter count stays manageable relative to the output quality.

Wan 2.2's temporal coherence across frames is the property NAVA depends on for audio alignment to work correctly. A model whose video generation produces inconsistent subject positions between frames would introduce synchronization errors regardless of audio generation quality. Starting from Wan 2.2 addresses that dependency before the audio training begins.

The Wan 2.2 backbone established strong character consistency across long video sequences. That characteristic is most relevant to NAVA's use case in dialogue scenes and music performances, where the visual subject must remain stable for audio synchronization to hold.

Multi-Speaker Timbre Control

NAVA supports independent voice timbre control for multiple speakers in the same generated clip. A prompt specifying two characters speaking can assign each a distinct voice profile that stays consistent across generations. For documentary style AI filmmaking where narration and character dialogue appear together, that capability removes a post production assembly step that would otherwise require separate audio generation for each speaker.

The mechanism anchors each speaker's timbre profile at generation time using reference audio or text descriptions. Each voice profile stays consistent across cuts and scene transitions within the same generated output. This is structurally different from post production dubbing, where voice consistency requires multiple sessions and manual matching.

Producers working with narration and character dialogue simultaneously can generate a complete scene in one NAVA pass. The alternative requires separate audio generation for each speaker followed by timing alignment in post. The NAVA approach eliminates that assembly step.

What Verse-Bench Measures

Verse-Bench evaluates four dimensions specific to joint audio and video generation: Sync-C (audio and video synchronization), Sync-D (audio and video distance), video quality, and audio word error rate. Standard video benchmarks like VBench measure video only. Verse-Bench is the first benchmark designed specifically to test whether audio and video events are causally aligned in generated content rather than just temporally proximate.

Sync-C and Sync-D measure distinct aspects of alignment. Sync-C evaluates whether the audio content category matches the video content category at each moment. Sync-D measures the temporal distance between corresponding audio and video events. A model that scores well on Sync-C but poorly on Sync-D produces the right kind of audio at the wrong time. NAVA leads the benchmark on both metrics.

NAVA vs. Post Production Audio Tools

Stable Audio 3 and MOSS-SoundEffect generate audio from text and can score or add sound effects to existing video in post production. NAVA generates audio natively alongside video from a single prompt. The pipeline difference determines where each tool fits. NAVA produces synchronized output in one step, while post production audio tools require a complete video before audio work begins.

The workflow sequencing difference matters for revision cycles. A creator using post production audio tools must lock the video edit before audio work can begin. Audio revisions after editing create rework at both stages. NAVA produces both outputs simultaneously, which makes the generated file the starting point for editing decisions rather than the endpoint for audio work.

That sequencing advantage is most valuable in iterative workflows where story elements change after the first draft. If dialogue is revised, a NAVA-based workflow re-generates both audio and video together. A post production audio workflow must regenerate audio separately and then re-sync.

The Apache 2.0 Commercial License

NAVA's Apache 2.0 release permits commercial use without royalty obligations. For a filmmaker generating synchronized audio and video for a paid project, that matters. The model weights, training code, and inference pipeline are all usable in commercial production without licensing fees.

Apache 2.0 also permits derivative works. A studio can fine-tune NAVA on proprietary footage and keep the resulting model private. That combination of commercial freedom and modification rights is less standard in model releases than creative commons licensing, which often restricts commercial use.

NAVA at 6.3 Billion Parameters in Context

The 2 to 5 times parameter efficiency figure the paper reports compares NAVA to open source baselines that achieve comparable synchronization scores at larger model sizes. A 6.3 billion parameter model that leads larger open source systems on all four Verse-Bench metrics is the central claim of the paper.

The Wan 2.2 backbone NAVA extends is itself a competitive video model. The paper reports that video quality scores hold or improve versus Wan 2.2 alone, which means the audio training did not compromise the visual foundation it builds on. The audio layer adds capability without degrading what the backbone already delivers.

For text-to-video and image-to-video generation in AI FILMS Studio, explore the video workspace to try the latest models.

---

Sources

arXiv: NAVA: Native Audio-Visual Alignment for Generation HuggingFace: ernie-research/NAVA