EditorNodesPricingBlog

Reference Forcing & MPEG-Diffusion: Fast Long Video AI

November 20, 2026
Reference Forcing & MPEG-Diffusion: Fast Long Video AI

UT Austin & Roblox, MPEG-Diffusion (arXiv:2611.12069) and University of Rochester & Adobe, Reference Forcing

Share this post:

Reference Forcing & MPEG-Diffusion: Fast Long Video AI

Reference Forcing (University of Rochester and Adobe, November 2026) and MPEG-Diffusion (UT Austin and Roblox, arXiv:2611.12069, November 2026) are two complementary November 2026 open research breakthroughs that solve the two hardest bottlenecks in local Video Diffusion Transformers (DiTs): long form appearance drift and quadratic full resolution token cost. Reference Forcing anchors long autoregressive rollouts back to initial frame identities via region guided KV cache value correction, while MPEG-Diffusion dynamically processes only ~30% of full resolution tokens through dual resolution attention, accelerating Wan based video generation by up to 3x without degrading foreground sharpness.

Studio availability. Try the public MPEG-Diffusion demo below; these research methods are not part of a paid AI FILMS Studio plan (use WAN 3.0 or Seedance 2.5 in the video workspace).

MPEG-Diffusion mixed resolution video generation focusing high resolution tokens on foreground subjects

Two Bottlenecks Holding Back Local Long Form Video

When indie filmmakers run open Wan or HunyuanVideo DiTs locally, two failure modes immediately surface:

  1. Identity and color drift past 10 seconds. Autoregressive long video generators feed their own previously denoised frames back as context. Small errors compound until faces deform, wardrobe colors shift, and backgrounds melt.
  2. Wasted compute on flat backgrounds. Standard Video DiTs spend the exact same quadratic attention FLOPs on out of focus skies and blurred studio walls as they do on an actor's eyes and hands.

Reference Forcing and MPEG-Diffusion fix both problems cleanly:

  • Region guided teacher student anchoring (Reference Forcing). Rather than locking the entire frame stiffly (which kills camera motion), Reference Forcing builds a segmentation guided teacher that forces foreground subjects and background zones in later chunks to attend selectively back to their matching regions in the initial anchor frame. It then distills that routing into a dedicated student attention path paired with KV cache value correction, keeping faces and sets locked while allowing free camera pans and actor movement.
  • Self selected dual resolution attention (MPEG-Diffusion). Inspired by classical MPEG video codecs that allocate higher bitrates to salient macroblocks, MPEG-Diffusion lets the DiT's own internal attention heads self select visually important foreground regions at high resolution while encoding static or blurred backgrounds at low resolution. Queries attend to matching resolution keys and values, and a mixed resolution decoder blends the two streams without boundary seams, cutting total tokens to roughly 30% of a full resolution run.
Reference Forcing teacher student regional attention anchoring and KV cache value correction diagram

Reference Forcing pipeline combining region guided initial frame anchoring with KV cache value correction. Credit: University of Rochester & Adobe.

Naive frame anchoring freezes subjects near their starting coordinates because every token attends globally to frame 0.

By separating regional identity lookup from motion dynamics and correcting drifted KV values directly inside the cache, Reference Forcing holds character consistency across long takes without freezing blocking.

Try MPEG-Diffusion Live in the Browser

You can inspect MPEG-Diffusion mixed resolution token allocation directly in the embedded Hugging Face Space below (hugging-apps-mpeg-diffusion.hf.space).

Architecture & Performance Matrix

Technique Core Mechanism Primary Gain Token / Compute Impact Target Backbone
Reference Forcing Region guided student attention + KV cache value correction Eliminates subject/background drift across 30s–60s+ rollouts Minimal overhead on causal KV cache Autoregressive Video DiTs (Wan, causal forcing models)
MPEG-Diffusion Self selected HR regions + dual resolution attention + mixed res decoder ~3x faster inference with preserved foreground sharpness Processes ~30% of full resolution tokens Wan 2.1 / modern Video DiTs

MPEG-Diffusion requires no manual foreground rotoscoping or user supplied saliency masks.

During early denoising steps, the DiT's internal attention maps automatically identify moving faces, hands, and hero objects to promote to high resolution while compressing low frequency backgrounds.

MPEG-Diffusion dual resolution attention and mixed resolution decoder architecture

MPEG-Diffusion dual resolution attention and mixed resolution latent decoder architecture. Credit: UT Austin & Roblox (arXiv:2611.12069).

Parameter Reference Table

Parameter Recommended Value Technical Effect
hr_token_ratio (MPEG-Diffusion) 0.25 – 0.35 Fraction of frame tokens promoted to full resolution; 0.30 hits the 3x speedup sweet spot.
selection_step (MPEG-Diffusion) Step 3 to 5 Denoising step where DiT attention heads lock high resolution macroblock selection.
anchor_frame_idx (Reference Forcing) 0 (initial keyframe) Reference frame supplying clean identity and color statistics for KV value correction.
value_correction_strength 0.6 – 0.85 Controls how aggressively drifted KV cache values pull back toward anchor statistics.

Local Integration Pattern (PyTorch / Diffusers)

Below is the standard local setup workflow for testing MPEG-Diffusion mixed resolution inference on a Wan backbone:

git clone https://github.com/scrya-com/mpeg-diffusion.git
cd mpeg-diffusion
pip install -r requirements.txt
import torch
from mpeg_diffusion import MixedResWanPipeline

pipe = MixedResWanPipeline.from_pretrained(
    "Wan-AI/Wan2.1-T2V-1.3B-Diffusers",
    torch_dtype=torch.bfloat16,
).to("cuda")

video = pipe(
    prompt="35mm tracking shot of a cyclist weaving through misty pine trees at dawn, sharp foreground focus, soft bokeh forest background",
    height=480,
    width=832,
    num_frames=81,
    hr_token_ratio=0.30,
    use_mixed_res_decoder=True,
    num_inference_steps=30,
).frames[0]

Filmmaker Production Playbook

  1. Use MPEG-Diffusion on shallow depth of field shots first. Close ups, two shots, and anamorphic portraits with soft bokeh backgrounds see the biggest quality to speed ratio because 70% of the frame naturally belongs in the low resolution stream.
  2. Pair Reference Forcing with clean hero first frames. Because KV value correction anchors the entire rollout to frame 0, start from an artifact free 4K still generated in MiniMax-H3-Image or FLUX 2 to prevent initial flaws from propagating.
  3. Combine streaming scheduling with drift correction. Watch for upcoming community merges combining WaveForcing diagonal wavefront scheduling with Reference Forcing KV correction for real time, minute long single GPU takes.

Related Reading


Sources