
HeartMuLa Team — heartlib Apache 2.0 music foundation model benchmarks
Share this post:
HeartMuLa: Open Source Apache 2.0 Music AI
HeartMuLa (HeartMuLa/heartlib, Apache 2.0 License) is an open source music foundation model suite combining HeartMuLa-oss-3B and HeartMuLa-oss-7B language models with HeartCodec, an ultra low frame rate neural audio codec, and built in lyric transcription tools. Designed for local lyrics to song composition, multilingual vocal synthesis, and controllable instrumental scoring, HeartMuLa gives independent filmmakers permissive commercial weights they can run on a single workstation GPU with zero subscription lock in.
Studio availability. Try the public Hugging Face demo below; HeartMuLa is not part of a paid AI FILMS Studio plan (use Suno in the music workspace for example instead).
Why HeartCodec and Apache 2.0 Matter for Indie Films
Commercial film distribution requires clean chain of title on every audio stem. Many popular open music checkpoints carry non commercial or territory restricted licenses that block festival screening and streaming platform delivery.
HeartMuLa solves both legal clearance and local compute cost:
- Permissive Apache 2.0 license across weights and code. Studios can generate temp scores, diegetic radio songs, and final instrumental beds locally and ship them in commercial projects with full Apache 2.0 rights.
- Ultra low bitrate
HeartCodectokenization. High frame rate audio codecs choke transformer context windows past 60 seconds.HeartCodeccompresses 48 kHz music into compact acoustic tokens soHeartMuLa-oss-3Bgenerates full multi minute songs rapidly on12 GB – 16 GBGPUs. - Structured lyric section tags and transcription. Filmmakers can format
[Intro],[Verse],[Chorus],[Bridge], and[Outro]blocks alongside style descriptors, and run HeartMuLa's companion lyrics transcription module (HeartTranscriptor) to analyze reference songs before generating variations.
HeartMuLa system overview integrating HeartCodec acoustic tokens, `HeartMuLa-oss` language models, and lyric alignment. Credit: HeartMuLa Team (Apache 2.0).
By compressing multi instrument stereo mixes into low frame rate HeartCodec tokens, the 3B checkpoint fits comfortably on consumer GPUs while retaining clean vocal phrasing and stereo drum transients.
Indie post teams can batch render ten scoring variations overnight on local hardware without paying per generation API fees.
Try HeartMuLa Live in the Browser
You can test HeartMuLa directly in the embedded Hugging Face Space below (mrfakename-heartmula.hf.space).
Hardware, Variants, and License Matrix
| Model Variant | Parameters | VRAM Needed | Core Strength | License |
|---|---|---|---|---|
HeartMuLa-oss-3B + HeartCodec |
3B LM + codec |
10 GB – 16 GB VRAM (BF16 / FP8) |
Fast consumer GPU scoring and multilingual vocals | Apache 2.0 |
HeartMuLa-oss-7B + HeartCodec |
7B LM + codec |
20 GB – 24 GB VRAM (BF16) |
Deeper harmonic complexity and rich acoustic arrangement | Apache 2.0 |
Parameter Reference Table
| Parameter | Recommended Value | Musical Effect |
|---|---|---|
--tags |
Comma separated genre, mood, and instruments | Example: "cinematic, dark ambient, analog synth, slow cello, 80 bpm". |
--lyrics |
Section tagged text or [Instrumental] |
Controls vocal timing or forces pure dialogue safe underscore. |
--cfg_scale |
1.5 – 2.5 |
Tightens style tag adherence without introducing metallic codec artifacts. |
--temperature |
0.85 – 1.0 |
Balances safe harmonic predictability against creative melodic turns. |
Local Setup with heartlib
git clone https://github.com/HeartMuLa/heartlib.git
cd heartlib
pip install -e .
huggingface-cli download HeartMuLa/HeartMuLa-oss-3B --local-dir ./ckpt/HeartMuLa-oss-3B
huggingface-cli download HeartMuLa/HeartCodec-oss --local-dir ./ckpt/HeartCodec-oss
from heartlib import HeartMuLaGenPipeline
import torch
pipe = HeartMuLaGenPipeline.from_pretrained(
"./ckpt",
device=torch.device("cuda"),
dtype=torch.bfloat16,
version="3B",
)
pipe(
{
"lyrics": "[Intro]\n[Instrumental]\n[Bridge]\n[Outro]",
"tags": "cinematic thriller underscore, pulsing low strings, distant brass swells, minor key, 90 bpm",
},
max_audio_length_ms=120_000,
save_path="scene_09_thriller_bed.wav",
cfg_scale=1.8,
temperature=0.9,
)
Filmmaker Production Playbook
- Use
HeartMuLa-oss-3Bfor fast Apache 2.0 temp tracks. Cut your rough assembly to clean 90 to 120 second instrumental beds generated locally, then refine with7Bor StepAudio-3-Music once picture locks. - Separate diegetic songs from score. Write short
[Verse]and[Chorus]lyric sheets for background diner radios or car stereos so you never face music clearance blocks at festivals. - Normalize stems to -24 LKFS before mixing. Drop generated WAV files into your NLE and trim sub 40 Hz rumble so footsteps and dialogue stay crisp.
Related Reading
- StepAudio-3-Music: Open Source 48 kHz Film Score AI
- ACE-Step 1.5: Open Source Music Generation Guide
- MiniMax Music 3 & GGUF: Open Source Film Score AI
- YuE2: Open Source Lyrics to Song Editable Score
Sources
Continue Reading
Video & LipSync
- Video Generator
- Text to Video
- Image to Video
- Start-End Frame to Video
- Draw to Video
- Motion Control
- Video Enhancer
- Video Upscaler
- Video Extend
- Video to Video LipSync
- Audio to Video LipSync
- Image to Video LipSync
- Video FaceSwap
- Seedance 2.5
- WAN 3.0
- Seedance 2
- FLUX 3
- Minimax H3
- Vidu Q3 Pro
- Grok Imagine Video 1.5
- Gemini Omni
- Google Veo 3.1
- Kling 3.0 Pro
- Luma Ray 3.2
- LTX 2.3
- Happy Horse 1.1
- Kling 3.0 Motion
- ByteDance Upscaler
- InfiniteTalk
- InsightFace
Image & Edit
- AI Character
- AI Actor
- Art Generator
- Text to Image
- Image to Image
- LoRA Image to Image
- Draw to Edit
- Image LoRA Training
- Remove Background
- Image Enhancer
- MidJourney 8.0
- Krea 2
- OpenAI GPT Image 2.0
- Nano Banana Pro Ultra
- Nano Banana 2 Lite
- Nano Banana Pro
- Minimax Image
- Clarity Pro
- Z-Image Turbo LoRA
- Krea 2 Turbo LoRA
- Z-Image LoRA Trainer
- Qwen Image LoRA Trainer
- Background Erase Network 2 (BEN2)


