EditorNodesPricingBlog

Kandinsky 6.0: Open Source Audio-Video & Lip Sync AI

November 15, 2026
Kandinsky 6.0: Open Source Audio-Video & Lip Sync AI

Kandinsky Lab, Kandinsky 6.0 Video (MIT License)

Share this post:

Kandinsky 6.0: Open Source Audio-Video & Lip Sync AI

Kandinsky 6.0 Video (kandinskylab/kandinsky-6) is an open source foundation model suite released by Kandinsky Lab under the MIT license (arXiv:2610.05608) for synchronized video and 44 kHz audio generation. The release includes Kandinsky 6.0 Video Lite (3B), Kandinsky 6.0 Video Pro (29B), a dedicated Video Super Resolution pipeline (kandinskylab/kandinsky-6-sr) that scales 480p clips to Full HD 1920x1080, and the companion Kandinsky S2V 1.0 speech driven avatar model (arXiv:2610.19290).

Studio availability. Kandinsky 6.0 is not available in AI FILMS Studio. This article covers a public research release. To generate video in the Studio, see the models listed on the Text to Video page or in Studio Nodes.

Kandinsky 6.0 Video Pro visual stream generation sample

Dual Stream CrossDiT Architecture and Multimodal Modes

Earlier open source video generators synthesize silent frames and rely on separate Foley or dubbing passes to attach sound. Kandinsky 6.0 builds directly on the visual backbone of Kandinsky 5.0 Video and pairs a pretrained video stream with a newly trained audio stream inside a dual stream CrossDiT architecture.

  1. Bidirectional cross modal attention. Inside each fused CrossDiT block, video tokens (using 3D RoPE) and audio tokens (using 1D RoPE) apply self attention, attend to their respective Qwen2.5-VL and CLIP text embeddings, and exchange timing cues through bidirectional video to audio and audio to video attention layers.
  2. Six unified conditioning modes. The native pipeline supports text to audio video, text to silent video (--no-audio), image to audio video (--image), first and last keyframe interpolation (--image plus --last-image), audio conditioned video (--audio-file), and video conditioned Foley synthesis (--video-file).
  3. 10 step PiFlow distillation and SPEED acceleration. Distilled checkpoints use the PiflowScheduler to render 5 second clips (480x864 at 24 fps) in 10 steps at guidance_scale=1.0, while the spectral progressive diffusion strategy (SPEED) accelerates the 29B Pro model by an additional 14 percent.
  4. Full figure speech driven avatar animation (Kandinsky S2V 1.0). For dialogue shots where a recorded voice track drives a character plate, Kandinsky S2V 1.0 combines a 2.3B visual DiT (32 blocks) with Qwen2.5-Omni-7B audio tower features (25 tokens per second) and three scale FramePack temporal memory to animate lip sync, facial expressions, hand gestures, and upper body posture across 5 second chunks.
Kandinsky 6.0 CrossDiT dual stream video and audio transformer block

Each Kandinsky 6.0 CrossDiT block links the 3D RoPE video stream and 1D RoPE audio stream through bidirectional cross modal attention. Credit: Kandinsky Lab (MIT).

Kandinsky S2V 1.0 speech to video avatar pipeline architecture

Kandinsky S2V 1.0 injects Qwen2.5-Omni-7B speech tokens and FramePack memory into 13 visual DiT blocks for full figure lip sync. Credit: Kandinsky Lab (MIT).

Kandinsky S2V 1.0 full figure avatar animation driven by a single reference image and speech track (with audio)

Checkpoints, VRAM & License Matrix

Model Variant Hugging Face ID Parameters & Steps Target Hardware & Latency (5s SD / Full HD) License
Kandinsky 6.0 Pro Distill kandinskylab/Kandinsky-6.0-Pro-distill-5s-Diffusers 29B (10 steps, CFG=1.0) RTX 4090 24 GB (235s / 381s with block offload), H100 80 GB (87.8s / 133.2s) MIT
Kandinsky 6.0 Pro Base kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers 29B (50 steps, CFG=5.0) RTX 5090 32 GB (768s SD), A100 / H100 80 GB (351.5s SD with MagCache) MIT
Kandinsky 6.0 Lite Distill kandinskylab/Kandinsky-6.0-Lite-distill-5s-Diffusers 3B (10 steps, CFG=1.0) RTX 5060 Ti 16 GB (173.7s SD), RTX 4090 24 GB (68s SD), RTX 5090 32 GB (44.2s SD) MIT
Kandinsky 6.0 Video SR kandinskylab/Kandinsky-6.0-VSR-distilled2steps-5s-Diffusers Causal 3D K-VAE + SR DiT (2 steps) Scales 480x864 to 1920x1080 (2.25x scale) with NABLA flex attention MIT
Kandinsky S2V 1.0 Lite Distill kandinskylab/Kandinsky-S2V-1.0-Lite-distill-Diffusers 2.3B DiT (16 steps, 4.2 GiB bf16) Consumer 16 to 24 GB VRAM GPUs; 25 fps chunked avatar lip sync MIT
KVAE-Audio Tokenizer kandinskylab/KVAE-Audio 44.1 / 48 kHz Audio VAE Paired audio encoding and vocoding for T2AV, I2AV, and V2A modes MIT

Running Kandinsky 6.0 Locally With Diffusers and CLI

Kandinsky 6.0 ships native Hugging Face diffusers pipelines (Kandinsky6TI2VAPipeline and Kandinsky6SRPipeline) alongside a just CLI runner and serving integrations for vLLM-Omni, SGLang, and FastVideo. Spoken dialogue inside prompts is marked with <S> and <E> delimiters so the cross modal attention blocks align phoneme timing to mouth movement.

import torch
from diffusers import Kandinsky6SRPipeline, Kandinsky6TI2VAPipeline
from diffusers.utils import encode_video

pipe = Kandinsky6TI2VAPipeline.from_pretrained(
    "kandinskylab/Kandinsky-6.0-Pro-distill-5s-Diffusers",
    torch_dtype=torch.bfloat16,
)
pipe.enable_model_cpu_offload()

output = pipe(
    prompt="A film director on a soundstage turns toward the lens and says <S>Roll camera and sound<E>",
    height=480,
    width=864,
    num_frames=121,
    num_inference_steps=10,
    guidance_scale=1.0,
)

torch._inductor.config.max_autotune = True
sr_pipe = Kandinsky6SRPipeline.from_pretrained(
    "kandinskylab/Kandinsky-6.0-VSR-distilled2steps-5s-Diffusers",
    torch_dtype=torch.bfloat16,
)
sr_pipe.enable_model_cpu_offload()
sr_pipe.transformer.set_attention_backend("flex")
sr_pipe.transformer.compile_repeated_blocks(fullgraph=True)

hd_frames = sr_pipe(
    video=output.frames[0],
    resolution_scale=2.25,
    num_inference_steps=2,
).frames[0]

encode_video(
    hd_frames,
    fps=24,
    output_path="kandinsky6_1080p_sync.mp4",
    audio=output.audio[0][None],
    audio_sample_rate=pipe.audio_sample_rate,
)

To run the native repository CLI with hardware presets (rtx-4090, rtx-5090, rtx-5060-ti, or h100), first and last keyframe conditioning, or speech driven avatar generation, execute the commands below.

git clone https://github.com/kandinskylab/kandinsky-6.git
cd kandinsky-6
just setup
just download pro-distill

# Text to audio-video on RTX 4090 with super-resolution to 1080p
just generate 'A jazz saxophonist plays a solo under neon rain' \
  --configs rtx-4090 \
  --sr \
  --out sax_1080p.mp4

# First and last frame transition with synchronized audio
just generate 'The actor walks from the hallway into the sunlit studio' \
  --configs rtx-4090 \
  --image first_frame.png \
  --last-image last_frame.png \
  --out transition_sync.mp4

For dedicated dubbing and avatar workflows, compare Kandinsky 6.0 and S2V 1.0 with LatentSync 1.6 and MuseTalk 1.5, LongCat-Video-Avatar 1.5, and LTX 2.5 open weights audio video generation.

Audio-Video Generation and Lip Sync in AI FILMS Studio

Kandinsky 6.0 runs locally on CUDA GPUs. To generate video with native audio, dialogue lip sync, and keyframe transitions in the browser without managing local offload presets, open the video workspace or build multi step audio and video pipelines in the Nodes Graph Editor.


Sources