AudioLDM2 & MusicGen: Open Source Foley and Score AI

Temp scores, room tone, and custom foley used to mean either a licensed sample library or a day in a recording booth. AudioLDM2 (cvssp/audioldm2, arXiv:2308.05734), Stable Audio Open (stabilityai/stable-audio-open-1.0, arXiv:2407.14358), and MusicGen (facebook/musicgen-small, arXiv:2306.05284) generate sound effects, atmospheric stems, and instrumental cues from a text prompt alone, running locally with no licensing fee per clip. An editor can prototype a scene's audio inside the sound workspace today, then reproduce the same brief offline with these three research releases for a final pass.

Studio availability. AudioLDM2, Stable Audio Open, and MusicGen are not available in AI FILMS Studio. These are local, open weights research releases. To generate music and sound effects in the Studio, use the sound workspace or the text to music page.

AudioLDM2: CLAP Conditioned Latent Diffusion

AudioLDM2 treats speech, music, and sound effects as one modeling problem instead of three separate ones. The paper defines a shared intermediate format called the Language of Audio (LOA), a sequence of vectors produced by AudioMAE, a self supervised audio representation model. A GPT-2 style language model translates text, image, or audio input into this LOA sequence, and a latent diffusion model (LDM) then denoises a VAE latent conditioned on that sequence.

Conditioning strength at each diffusion step comes from Contrastive Language-Audio Pretraining (CLAP), which embeds text and audio into one shared vector space so the diffusion UNet can measure how well an evolving audio latent matches the input prompt. The two released checkpoints share this pipeline but differ in UNet depth. cvssp/audioldm2 holds roughly 350 million UNet parameters, while cvssp/audioldm2-large scales that UNet to about 718 million parameters for denser, more detailed sound effect textures.

Stable Audio Open: A Diffusion Transformer for Variable Length Stereo

Stable Audio Open 1.0 targets a different constraint than AudioLDM2. Instead of fixed short clips, it generates variable length stereo audio up to 47 seconds at 44.1kHz, matching the length and channel count editors actually drop into a timeline. The architecture combines three core components, starting with an autoencoder that compresses the raw waveform into a short latent sequence, a frozen T5-base text encoder for prompt conditioning, and a diffusion transformer (DiT) that denoises inside that compressed latent space rather than a fixed length spectrogram grid.

Because the DiT operates on a flexible sequence length instead of a fixed tensor shape, the same model produces a 4 second foley hit or a 40 second ambient bed from the same checkpoint, just by changing the seconds_total conditioning value. Training data came from Freesound and the Free Music Archive under CC0, CC BY, or CC Sampling+ licenses, which is also why the resulting weights carry their own distinct license rather than a permissive one.

MusicGen: An Autoregressive Transformer Over EnCodec Tokens

Where AudioLDM2 and Stable Audio Open denoise a continuous latent, MusicGen generates discrete tokens one step at a time. The model trains a single stage autoregressive transformer over a 32kHz EnCodec tokenizer with 4 codebooks sampled at 50Hz. A small delay pattern staggers the four codebook streams against each other, so the transformer predicts all four in parallel instead of needing a separate cascaded model per codebook.

That design gives MusicGen only 50 autoregressive steps per second of audio, which keeps generation fast relative to its parameter count. facebook/musicgen-small holds 300 million parameters and is the practical choice for an 8GB card. The medium (1.5B) and large (3.3B) checkpoints produce noticeably richer instrumentation, but Meta's own documentation recommends at least 16GB of VRAM for the medium model.

Traditional foley work still means a mixer, a prop cart, and a booth timed to picture. The diagram below, from the Diff-Foley research project, frames that gap directly and is the source of this post's featured image.

Running AudioLDM2 and MusicGen Locally in Python

Both pipelines are available through Hugging Face diffusers and transformers with no custom inference code required:

import torch
import scipy
from diffusers import AudioLDM2Pipeline
from transformers import AutoProcessor, MusicgenForConditionalGeneration

# Option A: AudioLDM2 for a foley sound effect (CLAP-conditioned latent diffusion)
audioldm_pipe = AudioLDM2Pipeline.from_pretrained(
    "cvssp/audioldm2",
    torch_dtype=torch.float16,
)
audioldm_pipe.enable_model_cpu_offload()  # brings peak VRAM down toward 8GB

sfx_audio = audioldm_pipe(
    prompt="Footsteps on wet gravel with a distant metal door creaking shut",
    negative_prompt="Low quality, music, speech",
    num_inference_steps=200,
    audio_length_in_s=6.0,
).audios[0]
scipy.io.wavfile.write("foley_footsteps_gravel.wav", rate=16000, data=sfx_audio)

# Option B: MusicGen Small for a temp score cue (autoregressive over EnCodec tokens)
processor = AutoProcessor.from_pretrained("facebook/musicgen-small")
musicgen_model = MusicgenForConditionalGeneration.from_pretrained("facebook/musicgen-small")

inputs = processor(
    text=["Tense low string drone with a slow rising brass swell for a reveal shot"],
    padding=True,
    return_tensors="pt",
)
score_tokens = musicgen_model.generate(**inputs, max_new_tokens=512)
sampling_rate = musicgen_model.config.audio_encoder.sampling_rate
scipy.io.wavfile.write("temp_score_reveal.wav", rate=sampling_rate, data=score_tokens[0, 0].numpy())

Hardware, Output, and License Comparison

Model	Repository	Architecture	Minimum VRAM	Output	License
AudioLDM2	cvssp/audioldm2	350M UNet latent diffusion, CLAP + GPT-2 conditioning	~8 GB (enable_model_cpu_offload)	Mono, 16kHz, up to 10s+	CC BY-NC-SA 4.0
AudioLDM2-large	cvssp/audioldm2-large	718M UNet latent diffusion, CLAP + GPT-2 conditioning	~10 GB (offload) / 16 GB resident	Mono, 16kHz, up to 10s+	CC BY-NC-SA 4.0
Stable Audio Open	stabilityai/stable-audio-open-1.0	Autoencoder + T5-base + diffusion transformer (DiT)	~4 GB to 8 GB	Stereo, 44.1kHz, up to 47s	Stability AI Community License
MusicGen Small	facebook/musicgen-small	300M autoregressive transformer over EnCodec tokens	~3 GB to 8 GB	Mono, 32kHz, streaming length	Code: MIT / Weights: CC-BY-NC 4.0

Only Stable Audio Open and MusicGen Small comfortably fit an 8GB card without extra steps. AudioLDM2 reaches that range only with diffusers CPU offloading enabled, and AudioLDM2-large still benefits from a 10GB or larger GPU. None of the four license terms permit commercial redistribution of the weights themselves without review, though every generated output can be used under each project's stated terms. For sound design budgeting, read each model card before shipping generated audio in a paid production.

Music and Sound Design in AI FILMS Studio

Inside AI FILMS Studio, several live tools cover the same creative ground these four open releases target for local work. Suno Text to Music, MiniMax Music 2.5 Lyric to Music, and ACE-Step 1.5 Lyrics to Music are all live on the text to music page for full song and score generation in the cloud. Mirelo SFX 1.6 Text to SFX is live in the sound workspace for foley and sound effect generation without any local GPU.

For related open source audio options, see our guides on ACE-Step 1.5 open source music generation and Stable Audio 3, Stability AI's newer open weight successor to the Stable Audio Open model covered above. Inside the Nodes Graph Editor, a Mirelo SFX node can feed directly into a video node's audio track, letting a filmmaker pair a generated sound bed with a generated shot inside one saved graph.

---

Sources

AudioLDM2 Hugging Face (cvssp/audioldm2) | AudioLDM2 Paper (arXiv:2308.05734) | Stable Audio Open 1.0 Hugging Face (stabilityai/stable-audio-open-1.0) | Stable Audio Open Paper (arXiv:2407.14358) | MusicGen GitHub (facebookresearch/audiocraft) | MusicGen Paper (arXiv:2306.05284)