
Kandinsky Lab, Kandinsky 6.0 Video (MIT License)
Share this post:
Kandinsky 6.0: Open Source Audio-Video & Lip Sync AI
Kandinsky 6.0 Video (kandinskylab/kandinsky-6) is an open source foundation model suite released by Kandinsky Lab under the MIT license (arXiv:2610.05608) for synchronized video and 44 kHz audio generation. The release includes Kandinsky 6.0 Video Lite (3B), Kandinsky 6.0 Video Pro (29B), a dedicated Video Super Resolution pipeline (kandinskylab/kandinsky-6-sr) that scales 480p clips to Full HD 1920x1080, and the companion Kandinsky S2V 1.0 speech driven avatar model (arXiv:2610.19290).
Studio availability. Kandinsky 6.0 is not available in AI FILMS Studio. This article covers a public research release. To generate video in the Studio, see the models listed on the Text to Video page or in Studio Nodes.
Kandinsky 6.0 Video Pro visual stream generation sample
Dual Stream CrossDiT Architecture and Multimodal Modes
Earlier open source video generators synthesize silent frames and rely on separate Foley or dubbing passes to attach sound. Kandinsky 6.0 builds directly on the visual backbone of Kandinsky 5.0 Video and pairs a pretrained video stream with a newly trained audio stream inside a dual stream CrossDiT architecture.
- Bidirectional cross modal attention. Inside each fused CrossDiT block, video tokens (using 3D RoPE) and audio tokens (using 1D RoPE) apply self attention, attend to their respective
Qwen2.5-VLandCLIPtext embeddings, and exchange timing cues through bidirectional video to audio and audio to video attention layers. - Six unified conditioning modes. The native pipeline supports text to audio video, text to silent video (
--no-audio), image to audio video (--image), first and last keyframe interpolation (--imageplus--last-image), audio conditioned video (--audio-file), and video conditioned Foley synthesis (--video-file). - 10 step PiFlow distillation and SPEED acceleration. Distilled checkpoints use the
PiflowSchedulerto render 5 second clips (480x864at 24 fps) in 10 steps atguidance_scale=1.0, while the spectral progressive diffusion strategy (SPEED) accelerates the 29B Pro model by an additional 14 percent. - Full figure speech driven avatar animation (
Kandinsky S2V 1.0). For dialogue shots where a recorded voice track drives a character plate, Kandinsky S2V 1.0 combines a 2.3B visual DiT (32 blocks) withQwen2.5-Omni-7Baudio tower features (25 tokens per second) and three scale FramePack temporal memory to animate lip sync, facial expressions, hand gestures, and upper body posture across 5 second chunks.
Each Kandinsky 6.0 CrossDiT block links the 3D RoPE video stream and 1D RoPE audio stream through bidirectional cross modal attention. Credit: Kandinsky Lab (MIT).
Kandinsky S2V 1.0 injects Qwen2.5-Omni-7B speech tokens and FramePack memory into 13 visual DiT blocks for full figure lip sync. Credit: Kandinsky Lab (MIT).
Kandinsky S2V 1.0 full figure avatar animation driven by a single reference image and speech track (with audio)
Checkpoints, VRAM & License Matrix
| Model Variant | Hugging Face ID | Parameters & Steps | Target Hardware & Latency (5s SD / Full HD) | License |
|---|---|---|---|---|
| Kandinsky 6.0 Pro Distill | kandinskylab/Kandinsky-6.0-Pro-distill-5s-Diffusers |
29B (10 steps, CFG=1.0) |
RTX 4090 24 GB (235s / 381s with block offload), H100 80 GB (87.8s / 133.2s) | MIT |
| Kandinsky 6.0 Pro Base | kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers |
29B (50 steps, CFG=5.0) |
RTX 5090 32 GB (768s SD), A100 / H100 80 GB (351.5s SD with MagCache) | MIT |
| Kandinsky 6.0 Lite Distill | kandinskylab/Kandinsky-6.0-Lite-distill-5s-Diffusers |
3B (10 steps, CFG=1.0) |
RTX 5060 Ti 16 GB (173.7s SD), RTX 4090 24 GB (68s SD), RTX 5090 32 GB (44.2s SD) | MIT |
| Kandinsky 6.0 Video SR | kandinskylab/Kandinsky-6.0-VSR-distilled2steps-5s-Diffusers |
Causal 3D K-VAE + SR DiT (2 steps) |
Scales 480x864 to 1920x1080 (2.25x scale) with NABLA flex attention |
MIT |
| Kandinsky S2V 1.0 Lite Distill | kandinskylab/Kandinsky-S2V-1.0-Lite-distill-Diffusers |
2.3B DiT (16 steps, 4.2 GiB bf16) |
Consumer 16 to 24 GB VRAM GPUs; 25 fps chunked avatar lip sync | MIT |
| KVAE-Audio Tokenizer | kandinskylab/KVAE-Audio |
44.1 / 48 kHz Audio VAE | Paired audio encoding and vocoding for T2AV, I2AV, and V2A modes | MIT |
Running Kandinsky 6.0 Locally With Diffusers and CLI
Kandinsky 6.0 ships native Hugging Face diffusers pipelines (Kandinsky6TI2VAPipeline and Kandinsky6SRPipeline) alongside a just CLI runner and serving integrations for vLLM-Omni, SGLang, and FastVideo. Spoken dialogue inside prompts is marked with <S> and <E> delimiters so the cross modal attention blocks align phoneme timing to mouth movement.
import torch
from diffusers import Kandinsky6SRPipeline, Kandinsky6TI2VAPipeline
from diffusers.utils import encode_video
pipe = Kandinsky6TI2VAPipeline.from_pretrained(
"kandinskylab/Kandinsky-6.0-Pro-distill-5s-Diffusers",
torch_dtype=torch.bfloat16,
)
pipe.enable_model_cpu_offload()
output = pipe(
prompt="A film director on a soundstage turns toward the lens and says <S>Roll camera and sound<E>",
height=480,
width=864,
num_frames=121,
num_inference_steps=10,
guidance_scale=1.0,
)
torch._inductor.config.max_autotune = True
sr_pipe = Kandinsky6SRPipeline.from_pretrained(
"kandinskylab/Kandinsky-6.0-VSR-distilled2steps-5s-Diffusers",
torch_dtype=torch.bfloat16,
)
sr_pipe.enable_model_cpu_offload()
sr_pipe.transformer.set_attention_backend("flex")
sr_pipe.transformer.compile_repeated_blocks(fullgraph=True)
hd_frames = sr_pipe(
video=output.frames[0],
resolution_scale=2.25,
num_inference_steps=2,
).frames[0]
encode_video(
hd_frames,
fps=24,
output_path="kandinsky6_1080p_sync.mp4",
audio=output.audio[0][None],
audio_sample_rate=pipe.audio_sample_rate,
)
To run the native repository CLI with hardware presets (rtx-4090, rtx-5090, rtx-5060-ti, or h100), first and last keyframe conditioning, or speech driven avatar generation, execute the commands below.
git clone https://github.com/kandinskylab/kandinsky-6.git
cd kandinsky-6
just setup
just download pro-distill
# Text to audio-video on RTX 4090 with super-resolution to 1080p
just generate 'A jazz saxophonist plays a solo under neon rain' \
--configs rtx-4090 \
--sr \
--out sax_1080p.mp4
# First and last frame transition with synchronized audio
just generate 'The actor walks from the hallway into the sunlit studio' \
--configs rtx-4090 \
--image first_frame.png \
--last-image last_frame.png \
--out transition_sync.mp4
For dedicated dubbing and avatar workflows, compare Kandinsky 6.0 and S2V 1.0 with LatentSync 1.6 and MuseTalk 1.5, LongCat-Video-Avatar 1.5, and LTX 2.5 open weights audio video generation.
Audio-Video Generation and Lip Sync in AI FILMS Studio
Kandinsky 6.0 runs locally on CUDA GPUs. To generate video with native audio, dialogue lip sync, and keyframe transitions in the browser without managing local offload presets, open the video workspace or build multi step audio and video pipelines in the Nodes Graph Editor.
Sources
- kandinskylab/kandinsky-6 GitHub Repository
- Kandinsky 6.0 Video: Foundation Models for Synchronized Video and Audio Generation (arXiv:2610.05608)
- kandinskylab/kandinsky-6-sr Video Super-Resolution Repository
- Kandinsky S2V 1.0 Speech-to-Video Project Page & Paper (arXiv:2610.19290)
- Hugging Face Diffusers Kandinsky 6.0 Pipeline Documentation
Continue Reading
Video & LipSync
- Video Generator
- Text to Video
- Image to Video
- Start-End Frame to Video
- Draw to Video
- Motion Control
- Video Enhancer
- Video Upscaler
- Video Extend
- Video to Video LipSync
- Audio to Video LipSync
- Image to Video LipSync
- Video FaceSwap
- Seedance 2.5
- WAN 3.0
- Seedance 2
- FLUX 3
- Minimax H3
- Vidu Q3 Pro
- Grok Imagine Video 1.5
- Gemini Omni
- Google Veo 3.1
- Kling 3.0 Pro
- Luma Ray 3.2
- LTX 2.3
- Happy Horse 1.1
- Kling 3.0 Motion
- ByteDance Upscaler
- InfiniteTalk
- InsightFace
Image & Edit
- AI Character
- AI Actor
- Art Generator
- Text to Image
- Image to Image
- LoRA Image to Image
- Draw to Edit
- Image LoRA Training
- Remove Background
- Image Enhancer
- MidJourney 8.0
- Krea 2
- OpenAI GPT Image 2.0
- Nano Banana Pro Ultra
- Nano Banana 2 Lite
- Nano Banana Pro
- Minimax Image
- Clarity Pro
- Z-Image Turbo LoRA
- Krea 2 Turbo LoRA
- Z-Image LoRA Trainer
- Qwen Image LoRA Trainer
- Background Erase Network 2 (BEN2)


