Iris-3B & PixelUMM: Open Source Pixel-Space Image & Video AI

Iris-3B (speridlabs/iris-3b) and NVIDIA's PixelUMM (nv-tlabs/PixelUMM) represent a major architectural shift in open source visual AI by removing the Variational Autoencoder (VAE) and generating directly in raw pixel space. Both released under the Apache 2.0 license, Iris-3B (arXiv:2610.09450) is a 3B pixel space Diffusion Transformer that generates 1024x1024 images in 0.7 seconds (via 4 step Iris-3B Turbo) and adapts with zero architectural changes to monocular depth estimation and image upscaling, while NVIDIA's PixelUMM (arXiv:2609.38597) is an 8B unified transformer without encoders that reads and writes raw image patches and 4 frame video tubes across text-to-image, text-to-video, and video understanding.

Studio availability. Iris-3B and PixelUMM are not available in AI FILMS Studio. This guide covers their open source Apache 2.0 releases and local deployment. In the Studio, use NanoBanana Pro, Flux 2 Max, Seedream 5.0 Lite, or Imagen 4 Ultra on the Text to Image page or in Studio Nodes.

Why Removing the VAE Matters for Filmmakers and VFX Artists

Virtually every latent diffusion model compresses RGB images by 8x or 16x through a lossy VAE before the transformer ever sees the image. That bottleneck causes familiar production artifacts including smeared small typography, waxy skin pores, color shifts in fine foliage, and temporal shimmer when latent upscaled frames are decoded back to RGB.

1. Sperid Labs' Iris-3B: One Pixel Space Prior for Generation, Depth & Restoration

Iris-3B (speridlabs/iris-3b) replaces the VAE entirely with a 24 block trunk (width 2560, 8 dual stream plus 16 single stream blocks with GQA and sandwich RMSNorm), a frozen Qwen3-VL-4B-Instruct text encoder, and a 4 block Pixel Transformer (PiT) head that decodes 16x16 patches directly into raw pixels with pixel level modulation.

• Iris-3B Turbo (4 steps, 0.7s). Distills the 100 step rectified flow base model into a 4 step generator that renders a 1 megapixel (1024x1024) image in 0.7 seconds on a single H100.
• Single pass monocular depth estimation. Adapted from the exact same pixel space backbone, the depth/ checkpoint predicts sharp relative depth maps in a single forward pass for depth compositing and 3D camera projection.
• Pixel accurate image restoration and upscaling. The upscaler/ checkpoint restores degraded or low resolution frames without inventing hallucinated VAE textures.

2. NVIDIA PixelUMM: Unified Image & Video Transformer Without Encoders

Developed by NVIDIA and the University of Waterloo, PixelUMM (nvidia/PixelUMM) eliminates both the VAE and the separate CLIP or SigLIP vision encoder. Built on a Qwen3-8B backbone with dedicated understanding and generation experts sharing one attention mechanism across text tokens, clean pixels, and noisy pixels, PixelUMM converts images into 16x16 pixel patches and videos into 4 frame pixel tubes. A single checkpoint (S8-F22-R05, plus the 480p and 720p S8-F18-R01 video checkpoint) handles four tasks.

1. Text to Image (--task t2i).
2. Text to Video (--task t2v at 24 FPS, 96 frames).
3. Image Understanding (--task image-vlm).
4. Video Understanding (--task video-vlm).

Architecture, VRAM & License Comparison

Model	Repository & Weights	Backbone & Parameters	Supported Tasks	Inference Speed / Hardware	License
Iris-3B Base & Turbo	speridlabs/iris-3b	3B DiT + 4 block PiT pixel head + Qwen3-VL-4B (~12 GB per task)	Text-to-image (1024x1024), 1 pass monocular depth, image restoration and upscaling	0.7 s at 1024x1024 (4 step Turbo on 1x H100); runs on 16 to 24 GB VRAM GPUs	Apache 2.0
NVIDIA PixelUMM	nvidia/PixelUMM (S8-F22-R05 & S8-F18-R01)	8B Qwen3-8B dual expert transformer	t2i, t2v (96 frames @ 24 FPS, up to 832x464), image-vlm, video-vlm	Single 24 to 48 GB VRAM GPU for inference; 7x 48 GB GPUs for 4 task training	Apache 2.0

Try Iris-3B Turbo (4 Step Pixel Space) Live on Hugging Face

Test 4 step image generation without a VAE directly in the embedded Iris-3B Turbo Hugging Face Space below, or open the full multi task demo (including depth and upscaling) at speridlabs-iris-3b-turbo.hf.space and speridlabs-iris-3b.hf.space.

Local CLI Commands for Iris-3B & NVIDIA PixelUMM

To install Iris-3B (Python 3.11+, PyTorch 2.7.1+) and generate 1024x1024 images or extract depth maps, run the following commands.

git clone https://github.com/speridlabs/iris-3b.git
cd iris-3b
pip install -e .

# Download the base checkpoint (~12 GB, excluding depth/upscaler/turbo subfolders if desired)
hf download speridlabs/iris-3b --local-dir checkpoints/iris-3b --exclude "depth/*" "upscaler/*" "turbo/*"

python scripts/sample.py \
  --checkpoint checkpoints/iris-3b \
  --prompt "Studio portrait of a Maasai elder wearing vibrant beaded jewelry, Rembrandt lighting, 35mm film grain"

To run NVIDIA PixelUMM across text-to-image, 96 frame text-to-video, and video understanding, run the following commands.

git clone https://github.com/nv-tlabs/PixelUMM.git
cd PixelUMM

export PIXELUMM_CKPT=/path/to/PixelUMM/S8-F22-R05
export PIXELUMM_QWEN_DIR=/path/to/qwen3-8b-config-tokenizer
export PIXELUMM_OUTPUT=./pixelumm-outputs

# Generate a 96 frame 24 FPS video directly in pixel space
python inference.py \
  --checkpoint "$PIXELUMM_CKPT" --llm-path "$PIXELUMM_QWEN_DIR" \
  --task t2v \
  --prompt "A golden retriever runs across a grassy field in bright daylight. The camera tracks smoothly beside the dog." \
  --height 176 --width 320 --frames 96 --fps 24 --steps 35 \
  --negative-prompt-file experiments/s8_f22_r07/t2v_negative_prompt.txt \
  --output "$PIXELUMM_OUTPUT/t2v.mp4"

You can also feed Iris-3B's 1024x1024 keyframes into Sperid Labs' Idol to generate metric 3D Gaussian Splat sets in 1 second, or compare other pixel space image generators in our guides to HiDream-O1 pixel native image generation and PRX-Pixel by Photoroom.

Generating Keyframes and Video in AI FILMS Studio

Iris-3B and PixelUMM run locally on your own GPU environment. To create hero keyframes, storyboards, and upscaled plates in the browser, open the image and video workspace to run NanoBanana Pro, Flux 2 Max, Seedream 5.0 Lite, and Imagen 4 Ultra, or chain image and video models in the Nodes Graph Editor.

---

Sources

• speridlabs/iris-3b GitHub Repository
• Iris-3B: Pixel-Space Generation & General Vision Learner (arXiv:2610.09450)
• speridlabs/iris-3b on Hugging Face
• nv-tlabs/PixelUMM GitHub Repository
• PixelUMM: Encoder-Free Unified Image and Video Understanding and Generation (arXiv:2609.38597)
• nvidia/PixelUMM on Hugging Face