EditorNodesPricingBlog

PAN2: Open Source World Simulation Video AI

November 22, 2026
PAN2: Open Source World Simulation Video AI

Institute of Foundation Models (IFM) at MBZUAI (Apache 2.0)

Share this post:

PAN2: Open Source World Simulation Video AI

Standard video generators treat a prompt as a loose visual mood board, which breaks down as soon as a scene requires cause and effect over 15 or 20 seconds. PAN 2 (IFM/PAN2), developed by the Institute of Foundation Models (IFM) at MBZUAI and released under the Apache 2.0 license, approaches video generation as world simulation. Pairing a language based conceptual reasoner with a 29B parameter MMDiT perceptual backbone, PAN 2 simulates how objects, actors, and cameras evolve across up to 20 seconds at 704x1248 and 24 fps from a single text-to-video or image-to-video prompt. You can test this open source model free in your browser below with zero login.

Studio availability. Try the public Hugging Face demo below; PAN 2 is not part of a paid AI FILMS Studio plan (use MiniMax H3 Image to Video in the video workspace for example instead).

Try PAN 2 Live

Test text-to-video and image-to-video world simulation directly in the public Hugging Face Space below without local multi GPU hardware.

Architecture and Technical Design

PAN 2 implements the Generative Latent Prediction (GLP) architecture formalized in Critique of World Models (arXiv:2507.05169). Rather than forcing one diffusion network to invent both narrative logic and pixel texture simultaneously, PAN 2 splits the job into two distinct stages.

PAN 2 benchmark performance comparison across open source and closed world video models

PAN 2 benchmark scores by IFM at MBZUAI, Apache 2.0

First, the conceptual reasoning backbone reads the starting frame and user action instructions, then writes a structured, time grounded world state script specifying timed actor actions, physics outcomes, and camera moves.

Second, a frozen Qwen3.5-9B text encoder converts that structured world state into conditioning features for the 29B parameter MMDiT perceptual backbone, trained from scratch on hundreds of millions of annotated videos. The Wan 2.2 VAE decodes the predicted video latents back into crisp 24 fps footage.

Because one unified checkpoint serves both t2v and i2v tasks, production teams can stage an entire sequence from a single storyboard keyframe and steer how the shot unfolds across 241 frames (10 seconds) or 481 frames (20 seconds).

Human evaluation studies published by IFM show strong preference for PAN 2 on physical plausibility, action adherence, and long horizon coherence compared to standard clip generators. Filmmakers can also use PAN 2 for simulative pre visualization, testing two or three blocking variations from the same opening plate before committing to principal photography.

Official integration ships across Hugging Face Diffusers (PAN2Transformer3DModel) and vLLM serving recipes, simplifying batch rendering on cloud GPU clusters.

PAN 2 human evaluation preference chart on action adherence and visual realism

PAN 2 human evaluation results by IFM at MBZUAI, Apache 2.0

Hardware, VRAM, and License Matrix

Deployment Option Precision / Format Typical VRAM Output Ceiling License Intended Use
IFM/PAN2 (Diffusers BF16) BF16 Safetensors (29B DiT + Qwen3.5-9B + Wan 2.2 VAE) ~80 GB (H100 / A100 / B200) 704x1248, 24 fps, up to 20 s Apache 2.0 Commercial pre vis, world simulation
Multi GPU Node (generate.py) BF16 data parallel workers 1x 80 GB GPU per worker Batch T2V and I2V rollouts Apache 2.0 Studio shot generation queues
Quantized / CPU Offload FP8 / sequential offload ~32 to 48 GB Slower 10 s clips (241 frames) Apache 2.0 Local research workstations

The Apache 2.0 license covers both the code and the 29B weights with zero commercial revenue caps.

Parameter Reference

Parameter Default Value Production Guidance
NUM_STEPS 50 Full quality denoising schedule for complex physics
FLOW_SHIFT 9 Balances global motion layout with fine texture detail
GUIDANCE_SCALE 2.5 Applied across every step; prevents oversaturated highlights
VIDEO_LENGTH 241 frames (~10 s) Extend up to 481 frames (20 s) for long action beats
Resolution & FPS 704 x 1248 at 24 fps Matches theatrical 24 fps editorial timelines
I2V Temporal Latent 0 Unconditional anchor Locks opening frame identity without guidance distortion

Local Python and CLI Setup

Download the IFM/PAN2 perceptual backbone along with its frozen Qwen/Qwen3.5-9B text encoder and Wan-AI/Wan2.2-TI2V-5B-Diffusers VAE, then run inference with the official repository or Diffusers.

git clone https://github.com/ifm-ai/PAN2 && cd PAN2
pip install -r requirements.txt
hf download IFM/PAN2 --include "transformer/*" --local-dir ./weights/PAN2
hf download Qwen/Qwen3.5-9B
hf download Wan-AI/Wan2.2-TI2V-5B-Diffusers
# Convert diffusers weights once for native generate.py inference
python scripts/convert_checkpoint.py to-native \
  --input ./weights/PAN2/transformer \
  --output ./weights/PAN2-native/transformer

# Run text-to-video and image-to-video batch generation
bash inference/inference.sh ./weights/PAN2-native/transformer ./outputs inference/prompts/examples.json

Filmmaker Production Playbook

  • Run the conceptual reasoner before rendering. Feed your plain English director brief through inference/prompts_rewritter.py first. The 29B perceptual backbone was trained on structured, time stamped world state descriptions, so rewriting prompts with explicit timed actions yields far stronger physics and camera obedience.
  • Lock opening composition with image-to-video. Generate a crisp hero keyframe first, pass it as image_path (type: i2v), and let PAN 2 simulate 10 to 20 seconds of continuous camera and character movement from that exact frame.
  • Use 241 frames for blocking tests, 481 frames for master beats. Running 10 second previews cuts compute in half while you verify actor trajectories, then switch to 20 second rollouts once the motion path is approved.
  • Combine with video super resolution for 4K delivery. Native PAN 2 outputs sit at 704x1248. Pass approved takes through RelayVSR or SeedVR2 Video Upscaler in AI FILMS Studio to finish at crisp 1080p or 4K.

Related Reading


Sources

  1. IFM PAN 2 Model Hub on Hugging Face
  2. IFM PAN 2 Official GitHub Repository
  3. Critique of World Models Paper on arXiv (arXiv:2507.05169)
  4. PAN 2 Interactive Hugging Face Space