
AI FILMS architecture diagram built from higurashi-ai Haru-4B (Apache 2.0) specs
Share this post:
Haru-4B: Open Source Full Duplex Speech Dialogue AI
Turn based voice pipelines kill the natural rhythm of dramatic dialogue. A standard ASR, LLM, and TTS chain forces one speaker to finish completely before the other starts reacting, stripping away overlapping banter, interruptions, and quick breaths. Haru-4B (higurashi-ai/Haru-4B), released in November 2026 under the Apache 2.0 license, is a native full duplex spoken dialogue foundation model running at 7.5 Hz. Every 133 milliseconds, Haru-4B listens to a frame of user speech and emits a frame of its own 24 kHz voice and text simultaneously across 9 languages. You can test this open source model free in your browser below with zero login.
Studio availability. Try the public Hugging Face demo below; Haru-4B is not part of a paid AI FILMS Studio plan (use ElevenLabs Eleven V3 Text to Speech in the audio workspace for example instead).
Try Haru-4B Live
Test simultaneous full duplex character dialogue and custom voice conditioning directly in the public Hugging Face Space below.
Architecture and Technical Design
Haru-4B eliminates separate audio transformers and discrete audio codebook tokens. Its backbone is a 32 layer Qwen3.5-4B text transformer (hidden size 2,560) paired with a causal VibeVoice acoustic tokenizer operating at 24 kHz (3,200 samples per 133 ms frame).
Interaction step latency comparison based on higurashi-ai Haru-4B 7.5 Hz frame specs
At every 133 ms tick, the model sums three embeddings into a single transformer position: the user's incoming 64 dimensional continuous speech latent, the model's previous speech latent, and the model's previous text token. Listening and speaking never block each other.
To decode continuous speech without multi step diffusion lag, a compact 4 layer MLP audio head uses MeanFlow sampling to generate the next 64 dimensional audio latent in 1 sampling step (default guidance scale 1.5). Character persona is controlled via a structured text card, while voice timbre comes from a clean 3 second to 3 minute reference recording.
The model supports English, Spanish, Portuguese, German, French, Chinese, Japanese, Russian, and Korean. Official code ships with a PyTorch web server, a native macOS MLX app for Apple Silicon laptops, and a vLLM-Omni serving plugin.
Hardware, VRAM, and License Matrix
| Deployment Option | Precision / Runtime | Typical Hardware | Frame Rate / Latency | License | Intended Use |
|---|---|---|---|---|---|
PyTorch Server (haru.server) |
BF16 (model.safetensors) |
NVIDIA GPU (~12 GB VRAM) | 7.5 Hz (133 ms/frame) | Apache 2.0 | Live interactive character reads |
| Native macOS MLX App | Apple Silicon MLX | M series Mac (16+ GB unified RAM) | Real time local duplex | Apache 2.0 | Portable on set table reads |
| vLLM-Omni Plugin | BF16 serving | Data center GPU | Multi session streaming | Apache 2.0 | Interactive narrative installations |
All weights (model.safetensors, config, and tokenizer) ship under Apache 2.0. Haru-4B includes no bundled voice clips, so teams supply licensed actor recordings.
Parameter Reference
| Parameter | Default / Range | Effect on Dialogue Delivery |
|---|---|---|
| Frame rate | 7.5 Hz (133 ms) |
Fixed full duplex tick covering 3,200 samples at 24 kHz |
guidance_scale |
1.5 (range 1.0 to 2.0) |
Balances natural vocal prosody against tight character adherence |
| Reference audio clip | 3 s to 3 min clean mono/stereo |
Sets target character voice timbre (single speaker only) |
| Sampling steps (MeanFlow) | 1 |
Single step continuous 64D latent generation |
| Audio hardware | Wired mic + wired headphones | Avoids 100+ ms Bluetooth round trip latency |
Local Python and CLI Setup
Clone the official repository, launch the local duplex server on port 8998, or drive a live DuplexSession directly in Python.
git clone https://github.com/higurashi-ai/Haru && cd Haru
pip install -e ./haru
python -m haru.server --hf-repo higurashi-ai/Haru-4B
from haru import DuplexSession, HaruModel
from haru.audio import load_audio, prepare_reference
model = HaruModel.from_pretrained("higurashi-ai/Haru-4B")
voice = prepare_reference(load_audio("actor_voice_ref.wav"))
character_card = "Name: Elena\nRole: Exhausted night shift detective questioning a witness."
session = DuplexSession(model, system_prompt=character_card, reference=voice)
frame = session.start()
while not frame.done:
# Pass 3200 samples (133 ms @ 24 kHz) of live or recorded user audio
frame = session.step(user_audio_chunk)
Filmmaker Production Playbook
- Run live improvised scene rehearsals with AI scene partners. Writers and directors can load a character card and a licensed temp voice clip, put on wired headphones, and improvise overlapping dialogue directly against Haru-4B to test pacing before casting.
- Record clean voice references with zero room echo. Because Haru-4B copies acoustic room tone from the reference clip, pass raw actor samples through Inworld Voice Isolator before calling
prepare_reference. - Keep scripted line delivery separate from improvisation. The authors note that Haru-4B is a pretrained base model built for natural conversational rhythm rather than strict verbatim reading. For exact script locked ADR lines, pair it with IndexTTS-2.5 or ElevenLabs Eleven V3 Text to Speech in AI FILMS Studio.
- Always use wired headphones during live captures. Speaker bleed back into the microphone confuses full duplex listening, and Bluetooth headsets add 100 ms or more of lag.
Related Reading
- Inworld Voice Isolator: Open Source Film Audio Cleanup
- IndexTTS-2.5 & 2.0: Open Source Duration & Emotion TTS
- Audio8-TTS Realistic Air: Open Source 23 Language Voice
- Explore cloud video and dialogue workflows on the AI Text to Video Generator page.
Sources
Continue Reading
Video & LipSync
- Video Generator
- Text to Video
- Image to Video
- Start-End Frame to Video
- Draw to Video
- Motion Control
- Video Enhancer
- Video Upscaler
- Video Extend
- Video to Video LipSync
- Audio to Video LipSync
- Image to Video LipSync
- Video FaceSwap
- Seedance 2.5
- WAN 3.0
- Seedance 2
- FLUX 3
- Minimax H3
- Vidu Q3 Pro
- Grok Imagine Video 1.5
- Gemini Omni
- Google Veo 3.1
- Kling 3.0 Pro
- Luma Ray 3.2
- LTX 2.3
- Happy Horse 1.1
- Kling 3.0 Motion
- ByteDance Upscaler
- InfiniteTalk
- InsightFace
Image & Edit
- AI Character
- AI Actor
- Art Generator
- Text to Image
- Image to Image
- LoRA Image to Image
- Draw to Edit
- Image LoRA Training
- Remove Background
- Image Enhancer
- MidJourney 8.0
- Krea 2
- OpenAI GPT Image 2.0
- Nano Banana Pro Ultra
- Nano Banana 2 Lite
- Nano Banana Pro
- Minimax Image
- Clarity Pro
- Z-Image Turbo LoRA
- Krea 2 Turbo LoRA
- Z-Image LoRA Trainer
- Qwen Image LoRA Trainer
- Background Erase Network 2 (BEN2)


