
HKUST, NJU, PKU & Kling Team, ReBind (arXiv:2607.14681, CC BY 4.0)
Share this post:
ReBind & MiVE: Open Source Multi-Reference Video Editing
ReBind (arXiv:2607.14681, developed by HKUST, NJU, UCAS, PKU, HKU, and the Kling Team) and MiVE (arXiv:2605.14664, accepted at ICML 2026 by MT Lab Meitu, BUPT, and Tsinghua University) are two complementary open research frameworks that solve the two hardest failure modes in reference guided video editing. ReBind pairs ReBind-Instruct (built on Qwen3-Omni-30B-A3B-Instruct) with ReBind-Edit (built on the 14B LTX-2.3 DiT backbone and Gemma-3-12B-IT) to eliminate attribute bleeding when combining multiple reference images into one shot. MiVE repurposes Qwen3-VL as a hierarchical multiscale feature extractor inside a unified self attention Diffusion Transformer so localized spatial textures from a reference frame propagate cleanly across rapid motion and complex lighting transitions.
Studio availability. ReBind and MiVE are not available in AI FILMS Studio. This article covers public research releases. To generate or edit video in the Studio, see the models listed on the Image to Video page or in Studio Nodes.
ReBind multi reference video editing combining character attributes and environmental props from separate reference plates
How ReBind Solves Multi Reference Attribute Bleeding
When a director supplies two or more reference images alongside a source video (for example, swapping an actor's wardrobe from Reference 1 while inserting a vehicle or lighting style from Reference 2), standard video editors suffer from reference ambiguity. General purpose vision language models write vague captions that fail to specify which visual attribute belongs to which reference image, causing the diffusion backbone to blend identities or apply textures to the wrong subject.
ReBind eliminates this bottleneck through a two model architecture trained on synthetic inverse editing triplets (100K editing pairs for the DiT editor and 40K SFT + 2K GRPO pairs for the instruction generator).
- Structured instructions with embedded reference tokens (
ReBind-Instruct). Fine tuned fromQwen3-Omni-30B-A3B-Instruct, ReBind-Instruct generates dense editing instructions that embed explicit reference tokens (<ref_1>,<ref_2>) directly at the semantic positions of the attributes they govern, while marking preserved source motion and unmodified audio context. - Two stage Reinforcement Learning (GRPO) with dual checklist rewards. After Stage 1 supervised fine tuning, Stage 2 applies Group Relative Policy Optimization (GRPO) guided by three reward signals. A dual checklist reward verifies content completeness, a referential consistency reward checks exact
object-descriptor-anchortriple matching, and a length penalty prevents token repetition collapse. - Lightweight 14B DiT reference binding (
ReBind-Edit). Built on the 14B parameterLTX-2.3Diffusion Transformer withGemma-3-12B-ITtext encoding, ReBind-Edit ingests the structured dense instructions and binds each visual attribute to its designated reference image at720presolution (81 frames), outperforming open source baselines (UniVideo,VINO,Kiwi-Edit, andOmniWeaving) on both IntelligentVBench and UniVBench (1,240 human annotated single and multi reference samples).
ReBind-Instruct uses GRPO reinforcement learning with dual checklist and referential consistency rewards to bind attributes to exact reference sources. Credit: ReBind Authors.
MiVE extracts early spatial features and deep semantic features from Qwen3-VL, fusing them inside a unified self attention DiT. Credit: MiVE Authors (ICML 2026).
How MiVE Preserves Fine Spatial Details via Multiscale VLM Features
While ReBind focuses on multi reference attribution, MiVE (Multiscale Vision Language Features for Reference Guided Video Editing, ICML 2026) addresses a structural flaw in how video editors encode an edited first frame (reference image) and a text prompt. Decoupled encoders (T5 + CLIP) suffer from modality gaps across separate cross attention branches, whereas unified VLMs (Qwen3-VL) discard localized high frequency spatial details by passing only final layer representations to the generator.
MiVE resolves this trade off through three architectural changes.
- Hierarchical multiscale extraction from
Qwen3-VL. Diagnostic cross modal attention maps show that Layer 1 ofQwen3-VLsharply localizes human silhouettes and surface textures, while the final layer captures global editing semantics. MiVE extracts both shallow and deep VLM hidden states and projects them through a multiscale adapter. - Reference aware latent encoding. The edited reference frame is encoded into the shared video VAE latent space alongside the source video latents so spatial edits align pixel for pixel with the starting frame.
- Unified self attention Diffusion Transformer. Rather than injecting text and reference features through separate cross attention layers, MiVE concatenates the multiscale conditional tokens
cdirectly with the visual video tokens and processes them jointly through unified self attention. Across a 60 video benchmark (Simple and Complex subsets) and30Ktraining pairs, unified self attention boosts Instruction Adherence from6.76(decoupled cross attention) to over8.5+and ranks highest in human preference studies.
MiVE reference guided video editing on the ICML 2026 Complex Benchmark preserving identity and lighting across motion
Architecture, Backbone & Benchmark Matrix
| Component / Framework | Base Backbone & Encoder | Core Conditioning Mechanism | Resolution & Training Scale | Venue / Status |
|---|---|---|---|---|
| ReBind-Instruct | Qwen3-Omni-30B-A3B-Instruct |
Two stage SFT (40K triplets) + GRPO (2K samples, 8 candidates) with referential triple matching |
Dense instructions with explicit <ref_i> tokens (trained on 32x H200 GPUs) |
arXiv:2607.14681 (CC BY 4.0) |
| ReBind-Edit | LTX-2.3 (14B DiT) + Gemma-3-12B-IT |
Cross attention attribute binding from structured multi reference instructions | 720p, 81 frames (100K editing triplets, 24 to 80 GB VRAM) |
arXiv:2607.14681 (rebind-project/rebind) |
| MiVE Multiscale Extractor | Qwen3-VL (Early + Final Layers) |
Hierarchical extraction combining Layer 1 spatial localization with deep semantic layers | 30K paired reference video editing examples | ICML 2026 (arXiv:2605.14664) |
| MiVE Unified DiT Editor | Video DiT + Shared 3D VAE | Unified self attention over concatenated multiscale VLM conditional tokens and video latents | Portrait 832x1104 & Widescreen 720p (24 to 80 GB VRAM) |
ICML 2026 (mivepaper.github.io) |
Structuring Multi Reference Video Edits in Python
In a local ReBind + LTX-2.3 workflow, the editing pipeline runs in two stages. First, ReBind-Instruct expands a brief user request, the source video, and multiple reference images into a dense instruction with explicit <ref_1> and <ref_2> bindings. Second, ReBind-Edit renders the 81 frame 720p edited clip.
import torch
from PIL import Image
from transformers import AutoModelForCausalLM, AutoProcessor
# Stage 1: Generate structured dense instructions with explicit reference tokens
instruct_id = "rebind-project/ReBind-Instruct-30B-A3B"
processor = AutoProcessor.from_pretrained(instruct_id)
instruct_model = AutoModelForCausalLM.from_pretrained(
instruct_id,
torch_dtype=torch.bfloat16,
device_map="auto",
)
ref_images = [Image.open("wardrobe_ref1.png"), Image.open("background_car_ref2.jpg")]
user_prompt = (
"Replace the actor's jacket with the leather trench coat in Reference 1 "
"and park the vintage coupe from Reference 2 behind him while keeping his walk."
)
# Stage 2: Pass dense reference-bound instruction into the 14B LTX-2.3 ReBind-Edit pipeline
dense_instruction = (
"Keep the male subject's walking motion and camera pan from the source video. "
"Bind the distressed black leather trench coat and silver lapel pins from <ref_1> "
"onto the subject's torso, and insert the red vintage coupe from <ref_2> along "
"the wet asphalt curb in the background while preserving the ambient street audio."
)
For single reference workflows where a concept artist paints directly over the first frame in an image editor, MiVE takes the source clip, the edited starting frame, and a short text instruction, extracting Layer 1 spatial features and final layer semantic features from Qwen3-VL before running unified self attention denoising.
For related open source video editing pipelines, read our guides to Kiwi-Edit open source video editing, LTX 2.3 open source video generation, and JoyAI-Video-Edit real time video editing.
Multi-Reference Video Editing in AI FILMS Studio
ReBind and MiVE are research frameworks designed for local CUDA workstations and multi GPU clusters. To edit existing footage, swap character outfits from reference plates, or propagate first frame visual edits across a shot in the browser, open the video workspace or chain image editing and video restyling models inside the Nodes Graph Editor.
Sources
- ReBind Official Project Page
- ReBind: Multi-Reference Video Editing via Structured Instructions with Explicit Reference Relationships (arXiv:2607.14681)
- MiVE Official Project Page & Interactive Comparison
- MiVE: Multiscale Vision Language Features for Reference Guided Video Editing (ICML 2026, arXiv:2605.14664)
Continue Reading
Video & LipSync
- Video Generator
- Text to Video
- Image to Video
- Start-End Frame to Video
- Draw to Video
- Motion Control
- Video Enhancer
- Video Upscaler
- Video Extend
- Video to Video LipSync
- Audio to Video LipSync
- Image to Video LipSync
- Video FaceSwap
- Seedance 2.5
- WAN 3.0
- Seedance 2
- FLUX 3
- Minimax H3
- Vidu Q3 Pro
- Grok Imagine Video 1.5
- Gemini Omni
- Google Veo 3.1
- Kling 3.0 Pro
- Luma Ray 3.2
- LTX 2.3
- Happy Horse 1.1
- Kling 3.0 Motion
- ByteDance Upscaler
- InfiniteTalk
- InsightFace
Image & Edit
- AI Character
- AI Actor
- Art Generator
- Text to Image
- Image to Image
- LoRA Image to Image
- Draw to Edit
- Image LoRA Training
- Remove Background
- Image Enhancer
- MidJourney 8.0
- Krea 2
- OpenAI GPT Image 2.0
- Nano Banana Pro Ultra
- Nano Banana 2 Lite
- Nano Banana Pro
- Minimax Image
- Clarity Pro
- Z-Image Turbo LoRA
- Krea 2 Turbo LoRA
- Z-Image LoRA Trainer
- Qwen Image LoRA Trainer
- Background Erase Network 2 (BEN2)


