
UT Austin & Roblox, MPEG-Diffusion (arXiv:2611.12069) and University of Rochester & Adobe, Reference Forcing
Share this post:
Reference Forcing & MPEG-Diffusion: Fast Long Video AI
Reference Forcing (University of Rochester and Adobe, November 2026) and MPEG-Diffusion (UT Austin and Roblox, arXiv:2611.12069, November 2026) are two complementary November 2026 open research breakthroughs that solve the two hardest bottlenecks in local Video Diffusion Transformers (DiTs): long form appearance drift and quadratic full resolution token cost. Reference Forcing anchors long autoregressive rollouts back to initial frame identities via region guided KV cache value correction, while MPEG-Diffusion dynamically processes only ~30% of full resolution tokens through dual resolution attention, accelerating Wan based video generation by up to 3x without degrading foreground sharpness.
Studio availability. Try the public MPEG-Diffusion demo below; these research methods are not part of a paid AI FILMS Studio plan (use WAN 3.0 or Seedance 2.5 in the video workspace).

Two Bottlenecks Holding Back Local Long Form Video
When indie filmmakers run open Wan or HunyuanVideo DiTs locally, two failure modes immediately surface:
- Identity and color drift past 10 seconds. Autoregressive long video generators feed their own previously denoised frames back as context. Small errors compound until faces deform, wardrobe colors shift, and backgrounds melt.
- Wasted compute on flat backgrounds. Standard Video DiTs spend the exact same quadratic attention FLOPs on out of focus skies and blurred studio walls as they do on an actor's eyes and hands.
Reference Forcing and MPEG-Diffusion fix both problems cleanly:
- Region guided teacher student anchoring (
Reference Forcing). Rather than locking the entire frame stiffly (which kills camera motion), Reference Forcing builds a segmentation guided teacher that forces foreground subjects and background zones in later chunks to attend selectively back to their matching regions in the initial anchor frame. It then distills that routing into a dedicated student attention path paired with KV cache value correction, keeping faces and sets locked while allowing free camera pans and actor movement. - Self selected dual resolution attention (
MPEG-Diffusion). Inspired by classical MPEG video codecs that allocate higher bitrates to salient macroblocks, MPEG-Diffusion lets the DiT's own internal attention heads self select visually important foreground regions at high resolution while encoding static or blurred backgrounds at low resolution. Queries attend to matching resolution keys and values, and a mixed resolution decoder blends the two streams without boundary seams, cutting total tokens to roughly 30% of a full resolution run.
Reference Forcing pipeline combining region guided initial frame anchoring with KV cache value correction. Credit: University of Rochester & Adobe.
Naive frame anchoring freezes subjects near their starting coordinates because every token attends globally to frame 0.
By separating regional identity lookup from motion dynamics and correcting drifted KV values directly inside the cache, Reference Forcing holds character consistency across long takes without freezing blocking.
Try MPEG-Diffusion Live in the Browser
You can inspect MPEG-Diffusion mixed resolution token allocation directly in the embedded Hugging Face Space below (hugging-apps-mpeg-diffusion.hf.space).
Architecture & Performance Matrix
| Technique | Core Mechanism | Primary Gain | Token / Compute Impact | Target Backbone |
|---|---|---|---|---|
| Reference Forcing | Region guided student attention + KV cache value correction | Eliminates subject/background drift across 30s–60s+ rollouts | Minimal overhead on causal KV cache | Autoregressive Video DiTs (Wan, causal forcing models) |
| MPEG-Diffusion | Self selected HR regions + dual resolution attention + mixed res decoder | ~3x faster inference with preserved foreground sharpness | Processes ~30% of full resolution tokens | Wan 2.1 / modern Video DiTs |
MPEG-Diffusion requires no manual foreground rotoscoping or user supplied saliency masks.
During early denoising steps, the DiT's internal attention maps automatically identify moving faces, hands, and hero objects to promote to high resolution while compressing low frequency backgrounds.
MPEG-Diffusion dual resolution attention and mixed resolution latent decoder architecture. Credit: UT Austin & Roblox (arXiv:2611.12069).
Parameter Reference Table
| Parameter | Recommended Value | Technical Effect |
|---|---|---|
hr_token_ratio (MPEG-Diffusion) |
0.25 – 0.35 |
Fraction of frame tokens promoted to full resolution; 0.30 hits the 3x speedup sweet spot. |
selection_step (MPEG-Diffusion) |
Step 3 to 5 |
Denoising step where DiT attention heads lock high resolution macroblock selection. |
anchor_frame_idx (Reference Forcing) |
0 (initial keyframe) |
Reference frame supplying clean identity and color statistics for KV value correction. |
value_correction_strength |
0.6 – 0.85 |
Controls how aggressively drifted KV cache values pull back toward anchor statistics. |
Local Integration Pattern (PyTorch / Diffusers)
Below is the standard local setup workflow for testing MPEG-Diffusion mixed resolution inference on a Wan backbone:
git clone https://github.com/scrya-com/mpeg-diffusion.git
cd mpeg-diffusion
pip install -r requirements.txt
import torch
from mpeg_diffusion import MixedResWanPipeline
pipe = MixedResWanPipeline.from_pretrained(
"Wan-AI/Wan2.1-T2V-1.3B-Diffusers",
torch_dtype=torch.bfloat16,
).to("cuda")
video = pipe(
prompt="35mm tracking shot of a cyclist weaving through misty pine trees at dawn, sharp foreground focus, soft bokeh forest background",
height=480,
width=832,
num_frames=81,
hr_token_ratio=0.30,
use_mixed_res_decoder=True,
num_inference_steps=30,
).frames[0]
Filmmaker Production Playbook
- Use
MPEG-Diffusionon shallow depth of field shots first. Close ups, two shots, and anamorphic portraits with soft bokeh backgrounds see the biggest quality to speed ratio because 70% of the frame naturally belongs in the low resolution stream. - Pair
Reference Forcingwith clean hero first frames. Because KV value correction anchors the entire rollout to frame0, start from an artifact free 4K still generated in MiniMax-H3-Image or FLUX 2 to prevent initial flaws from propagating. - Combine streaming scheduling with drift correction. Watch for upcoming community merges combining WaveForcing diagonal wavefront scheduling with Reference Forcing KV correction for real time, minute long single GPU takes.
Related Reading
- WaveForcing & MovieRoll: Open Source Long Video AI
- Apple FastFrames: Open Source Real Time Video Diffusion
- Self Gradient Forcing: Long Video Extrapolation
- BlockVid: Minute Long Video Generation
Sources
Continue Reading
Video & LipSync
- Video Generator
- Text to Video
- Image to Video
- Start-End Frame to Video
- Draw to Video
- Motion Control
- Video Enhancer
- Video Upscaler
- Video Extend
- Video to Video LipSync
- Audio to Video LipSync
- Image to Video LipSync
- Video FaceSwap
- Seedance 2.5
- WAN 3.0
- Seedance 2
- FLUX 3
- Minimax H3
- Vidu Q3 Pro
- Grok Imagine Video 1.5
- Gemini Omni
- Google Veo 3.1
- Kling 3.0 Pro
- Luma Ray 3.2
- LTX 2.3
- Happy Horse 1.1
- Kling 3.0 Motion
- ByteDance Upscaler
- InfiniteTalk
- InsightFace
Image & Edit
- AI Character
- AI Actor
- Art Generator
- Text to Image
- Image to Image
- LoRA Image to Image
- Draw to Edit
- Image LoRA Training
- Remove Background
- Image Enhancer
- MidJourney 8.0
- Krea 2
- OpenAI GPT Image 2.0
- Nano Banana Pro Ultra
- Nano Banana 2 Lite
- Nano Banana Pro
- Minimax Image
- Clarity Pro
- Z-Image Turbo LoRA
- Krea 2 Turbo LoRA
- Z-Image LoRA Trainer
- Qwen Image LoRA Trainer
- Background Erase Network 2 (BEN2)


