MiDashengLM-Gen: One Apache 2.0 Model for Speech, Music and SFX

August 19, 2026

Architecture diagram from the MiDashengLM-Gen repository (Apache 2.0)

Share this post:

MiDashengLM-Gen: One Apache 2.0 Model for Speech, Music and SFX

Xiaomi Research released MiDashengLM-Gen on August 12, 2026. One model generates speech, music, sound effects and room tone, and it writes them into a single mixed scene rather than four separate stems.

The weights are on Hugging Face and the code is on GitHub, both under Apache 2.0. A paper describing the method went to arXiv the same day.

Every open audio release this blog has covered solved one slice of the problem. This one covers the whole brief in a single checkpoint, which is the reason it matters for a sound department.

The Input Format Is a Sound Design Brief

The part a filmmaker should look at first is how you talk to it. Input is a structured caption with six named fields, and you fill in the ones you need.

Field	What it describes
`<\|caption\|>`	The whole scene in one sentence
`<\|asr\|>`	The exact words spoken
`<\|speech\|>`	Voice, emotion and delivery
`<\|sfx\|>`	Sound effects
`<\|music\|>`	Music
`<\|env\|>`	Environment and ambience

Anything you leave out gets <|unknown|>. That is the same shape as a spotting note handed to a sound designer, which is unusual for a generation model and makes the output far easier to direct.

The project's own example reads like a cue sheet. A comedian delivers a punchline, the crowd laughs, a jazz sting lands, and the room is an intimate comedy club. Four fields, one scene.

What It Sounds Like

The samples below come from the project demo page. Each one shows the caption fields that produced it, so you can read the brief and hear the result.

Sound Effects

Caption: Loud bus roaring and voices

Speech: Multiple voices speaking indistinctly

SFX: A bus engine roaring with loud diesel motor and exhaust rumble

Env: Urban street environment with heavy vehicle traffic

Caption: Several pops and bursts of fireworks exploding

SFX: Sharp percussive pops and crackling bursts from fireworks explosions with reverberant decay.

Env: Outdoor open-air environment with firework display acoustics.

The first sample carries indistinct crowd voices under the engine. That is one generation rather than a dialogue pass mixed over an effects pass.

Music

Caption: This is an instrumental trumpet music piece. The trumpet is playing a calming main melody while the keyboard and the bass guitar are playing in the background. There is a slow tempo acoustic sounding drum beat playing the rhythm. The atmosphere is soothing. This piece could be used in the soundtrack of an art movie.

Music: A soothing instrumental with trumpet playing a calming melody over keyboard and bass guitar, accompanied by a slow acoustic drum beat.

Caption: Here we get to hear a hardcore metal piece. This is characterised, in part, by the rapid action on the kick drum, with notes being played at a rate of sixteenth notes on the kick. The drummer also frequents the ride cymbal. The guitar is grungy, gritty, fuzzy, heavy, deep, hardcore.

Music: A hardcore metal piece with rapid sixteenth-note kick drum patterns, frequent ride cymbal hits, and grungy, gritty, fuzzy electric guitar.

Mixed Audio

Caption: Male voice describing a vehicle's condition with engine idling in background.

ASR: The top is in excellent shape. It does have a gray interior.

Speech: Technical automotive assessment with enthusiastic tone.

SFX: Persistent engine idling with speech foreground.

Env: Indoor/outdoor hybrid space with significant engine interference.

Caption: Live piano and guitar performance with audience applause and background interference.

SFX: Distinct applause segments with persistent low-frequency interference.

Music: Live ensemble performance with alternating piano and guitar elements, moderate tempo, sustained musical introduction.

Env: Live venue recording with echoic acoustics and competing background noise.

The vehicle sample is the one to study. Dialogue, an idling engine and a described room came out of a single pass, and the transcript field controlled the exact words.

Multilingual Speech

Caption: Natural human speech

ASR: Whoever saves one life, saves the world entire.

Caption: Natural human speech in Spanish

ASR: Nada de móviles. Mente abierta. Sin desconfianza. Sin miedo. O los espíritus pueden molestarse.

Nine languages are supported. English, Chinese, Spanish, Portuguese, Russian, French, Japanese, Korean and German.

Emotional Speech

Caption: An angry person expressing intense frustration and rage in their speech

ASR: Do you remember the price we paid? Do you remember the dreams and promises we once had?

Speech: Aggressive harsh voice with raised volume, sharp tone, and fast intense pace

Caption: A cheerful person speaking with joy, excitement, and warm energy

ASR: Finally, I get to meet up with an old friend! Just thinking about chatting together makes me so happy.

Speech: Bright energetic voice with warm smile, light laughter, and upbeat rhythm

Caption: A sorrowful person speaking with deep sadness and heavy melancholy

ASR: The things are still there, but the people are gone. I want to speak, but tears come first.

Speech: Soft low trembling voice with slow heavy pace and sighing melancholic tone

Angry, happy and sad are the three emotion sets on the demo page. The delivery is described in the speech field as plain text, so you direct it the way you would direct an actor.

What Is Under the Hood

The architecture is documented in the repository and the paper. These are the numbers that decide whether it fits your machine and your delivery spec.

Component	Detail
LLM backbone	Qwen3-1.7B, fully fine tuned
Audio tokenizer	DashengTokenizer, 768 dimension latents at 25 Hz
Flow matching model	16 layers, hidden dimension 2048, 8 heads
Inference	10 step Euler ODE solver, CFG scale 2.0
Length control	Learned stop head for variable length output
Output	16 kHz mono WAV
Licence	Apache 2.0 on code and weights

Two things follow from that table. 16 kHz mono is a sketching and temp track format rather than a delivery format, so treat the output as a reference for a sound editor. A 1.7B backbone is small enough that the model is a realistic local install rather than a datacentre job.

The method is autoregressive generation with per token flow matching. In plain terms, the language model decides what comes next in the scene and the flow matching stage renders each step as continuous audio, with no quantisation into discrete codes.

How It Compares to the Other Open Audio Releases

Five open audio models have landed in 2026 with a real filmmaking use. Each one solved a different slice.

Model	Input	Covers	Licence
MiDashengLM-Gen	Structured text	Speech, music, SFX, ambience	Apache 2.0
Foley-Omni	Video	Speech, SFX, music from picture	Open
MOSS-SoundEffect v2.0	Text	Sound design	Open
Stable Audio 3	Text	Music and SFX	Open weight
MOSS-TTS v1.5	Text and reference	Voice, 48 kHz	Open

The split in that table is the useful part. Foley-Omni starts from picture, which is what you want when the cut exists. MiDashengLM-Gen starts from a written brief, which is what you have before the cut exists.

Sample rate is the other axis. MOSS-TTS delivers 48 kHz voice and this model delivers 16 kHz mixed scenes, so they answer different questions and a production may want both.

What This Means for Filmmakers

The practical use is pre production and temp tracks. Write the scene as a brief, generate the bed, and cut picture against something that already has the right shape.

Location scouting has an audio version of the same problem. A director can describe the room they want and hear a version of it before the location is booked, which is a cheaper conversation than a recce.

Nothing here is a delivery master. 16 kHz mono is a working format. The value is that the brief and the result stay in the same language, so a note like more room, less engine is a text edit rather than a remix.

MiDashengLM-Gen does not appear in any AI FILMS Studio picker, so it carries no roster row here. For finished audio in a project, the sound workspace and the music workspace cover sound effects and music generation, and the voice workspace handles dialogue.