PiD: NVIDIA Pixel Diffusion for Fast High Resolution Image Decoding

Researchers at NVIDIA's Spatial Intelligence Lab released PiD on May 22, 2026, a pixel diffusion decoder that replaces the VAE in latent diffusion pipelines. PiD reformulates decoding as a conditional diffusion process that denoises directly in pixel space, unifying decoding and upsampling into a single generative module.

The paper (arXiv 2605.23902) comes from Yifan Lu, Qi Wu, Jay Zhangjie Wu, Zian Wang, Huan Ling, Sanja Fidler, and Xuanchi Ren at NVIDIA.

What VAE Is and Why It Limits Resolution

Most text-to-image pipelines generate images in latent space, then decode them to pixels using a Variational Autoencoder. The VAE was designed for efficiency: operating in a compressed latent space is much faster than generating pixel values directly. But at high output resolutions, the VAE decode step becomes the primary performance bottleneck and introduces softness from the encode and decode round trip.

The VAE's compression also imposes a resolution ceiling. A latent space sized for 512-pixel images can be decoded to 1024 or 2048 pixels, but the upsampling process blurs fine detail rather than generating new pixel level information. The decoded image looks soft because the VAE is extrapolating, not generating. PiD is designed to solve this by replacing the decode step with a diffusion process that generates pixel level detail from scratch.

What PiD Does

PiD removes the VAE from the decode step. A sigma aware adapter injects noise corrupted versions of the source latent into the diffusion process, conditioning each denoising step on the latent content. Four inference steps via DMD2 distillation complete the decode.

The sigma aware adapter is the key architectural choice. It takes a noise corrupted version of the source latent as input at each denoising step, giving the model continuous access to the original encoded content throughout the process. This combines latent semantics with pixel level detail generation in a single forward pass, rather than in the two separate stages a VAE uses.

Speed and Resolution Numbers

On an RTX 5090, PiD converts 512×512 latents to 2048×2048 pixels in under 1 second using 13 GB of peak memory. On an NVIDIA GB200, latency drops to 210 milliseconds.

Against SeedVR2, a cascaded diffusion baseline, PiD runs 5.9× faster: 211.2 ms versus 1,237.5 ms per image at the same 2K output resolution on identical hardware.

4K Decoding in Action

PiD achieves 4× and 8× upscaling from the source latent resolution. The four step distilled model handles this at the latencies above, making 4K decode practical on consumer hardware for the first time at this speed.

The Compatible Latent Spaces

PiD is compatible with multiple latent spaces: FLUX.1, FLUX.2, SD3, Z-Image, and RAE. Compatibility across these architectures means PiD is not tied to a single base model lineage. A production team using FLUX.2 for generation can add PiD for decode without switching their upstream pipeline.

This also means PiD can be added to existing pipelines without retraining the upstream generation model. The decode step is swapped out; the generation model runs unchanged. That separation matters for studios that have already invested in fine-tuned or adapted generation models and do not want to retrain them to benefit from faster or sharper decoding.

DMD2 Distillation and the Four-Step Process

DMD2 is a distillation technique that reduces the number of diffusion steps required to produce a high quality output. Standard diffusion models require 20 to 50 denoising steps to generate an image. Distillation compresses that process into far fewer steps by training a student model to match the output of many steps in a smaller number.

PiD uses DMD2 distillation to reduce the decode to four steps while maintaining quality. The four step process is what delivers the sub-second latency at 2K resolution. Without distillation, a diffusion based decoder would be substantially slower than the VAE it replaces. DMD2 is what makes PiD practical rather than theoretical at these resolutions.

What 4K Decoding Means for Filmmakers

The practical consequence for filmmakers is access to generation quality at 4K that previously required either expensive proprietary pipelines or long inference times. A concept art workflow that generates 512-pixel latents and decodes to 4096×4096 output can now complete at under a second per image on an RTX 5090 class card.

That speed changes how iterative concept development works in production. Art directors who previously waited 10 to 30 seconds per decode can now generate dozens of high resolution variants in the time it previously took to produce one. The quality of the decoded output is also higher than VAE based decoding at the same latent resolution, because PiD generates pixel level detail rather than interpolating it.

Open Source Access

PiD is open source. Code is on GitHub at nv-tlabs/PiD and model weights are on Hugging Face under nvidia/PiD. Commercial use is permitted.

For filmmakers working with AI generated images, this approach addresses a long standing quality ceiling in latent decoding. HiDream-O1-Image removes the VAE from the generation architecture entirely. DyPE takes a third approach, enabling 4K output from existing models through training free position extrapolation. On the training side, Microsoft's Lens demonstrates how dense caption datasets cut foundation model compute requirements to 19% of the standard baseline.

Speed in Production Context

The numbers tell a production story. At sub-second latency per 2K image on an RTX 5090, a concept artist iterating through options in real time can generate 60 to 100 resolved variants per minute. The previous bottleneck with VAE decoding at 10 to 30 seconds per image limited iteration to 2 to 6 images per minute. That difference changes whether AI image generation can fit into an interactive creative workflow or must be treated as a batch process with offline review.

Filmmakers do not generate one image. They generate thousands during visual development, and the iteration speed at each stage determines whether AI tools are collaborative tools or rendering farms. PiD's latency profile places it in the collaborative category for the first time at 2K and 4K resolution.

Why VAE Compression Causes Softness

The standard VAE used in most latent diffusion pipelines compresses an image into a lower dimensional representation, then reconstructs it at decode time. The reconstruction step approximates the original pixel data but cannot recover fine high frequency detail that the compression discarded. At 512x512 output the perceptual loss is minimal. At 2048x2048 the approximation shows in soft edges, blurred textures, and reduced sharpness in areas with fine detail.

PiD bypasses the reconstruction problem by not compressing and decompressing at all. The sigma aware adapter uses the source latent as a conditioning signal throughout the diffusion process, and the model generates pixel data directly rather than approximating stored encoded information. The output sharpness at 2K and 4K reflects the generative process, not the limits of a reconstruction step.

Hardware and Accessibility

The RTX 5090 numbers represent peak performance. PiD also runs on lower tier hardware: the 13 GB peak memory requirement fits in the VRAM of most mid-range professional GPU configurations. The paper does not report RTX 4090 or RTX 3090 benchmarks, but the memory profile suggests the model is not restricted to the highest tier consumer GPUs.

The 210 millisecond latency on the NVIDIA GB200 is a data center figure. The RTX 5090 result at under 1 second per 2K image is the consumer hardware benchmark that matters for independent filmmakers and commercial studios without data center access.

The 2026 Open Source Image Model Context

PiD entered a landscape in mid-2026 where several approaches to high resolution generation were being released in parallel. HiDream-O1-Image removes the VAE from the generation architecture entirely. DyPE enables 4K output from existing models through training-free position extrapolation. Microsoft's Lens cuts foundation model compute requirements through dense caption datasets.

Each approach addresses a different point in the pipeline. PiD is specifically a decoder. It does not change how the upstream model generates latents. It changes only how those latents become pixels. That specificity means PiD can be dropped into existing workflows without touching the generation model, which is why its compatibility with FLUX.1, FLUX.2, SD3, Z-Image, and RAE matters: teams already using those models get faster and sharper decoding with a single substitution.

Where PiD Fits in the 2026 Open Source Landscape

The 2026 open source image generation landscape has seen several approaches to high resolution output released in parallel. HiDream-O1-Image removes the VAE from the generation architecture entirely and generates in pixel space from the start. DyPE enables 4K output from existing models through training-free position extrapolation at inference time. PiD addresses the decode step specifically, leaving the upstream generation model untouched.

Each approach targets a different moment in the pipeline. HiDream changes the architecture. DyPE changes inference behavior. PiD changes the decoder. The three can in principle be combined, though no combined evaluation has been published. For production teams, the choice between them depends on which bottleneck they are trying to solve: generation quality, output resolution, or decode speed.

The Commercial Use License

PiD is released under a commercial-use compatible license, with code on GitHub at nv-tlabs/PiD and weights on Hugging Face under nvidia/PiD. NVIDIA's decision to release a production-relevant decode technology under terms that permit commercial use is consistent with its strategy of building ecosystem adoption for its hardware through software tools.

A production pipeline that integrates PiD as its standard decoder is a pipeline running on NVIDIA GPUs and optimized for NVIDIA hardware. The license structure incentivizes adoption, and adoption creates hardware preference. Independent studios and post-production facilities can use PiD in client work without licensing negotiations.

What the Decoder Shift Means in Practice

PiD's replacement of the VAE decode step with a pixel diffusion process changes the nature of the generation pipeline's final stage. A VAE reconstruction is deterministic: the same latent produces the same output every time. A diffusion decode process is stochastic within the conditioning constraints: the sigma aware adapter anchors the output to the source latent's semantic content, but the pixel level details can vary across runs.

That variation has production implications. Art directors who want to explore fine detail options across multiple renders of the same scene can do so without changing the upstream latent, by re-running only the PiD decode step. The semantic content stays consistent because the latent stays the same. The pixel resolution detail changes because the diffusion process introduces controlled variation at each run.

To generate images in the cloud without local GPU setup, AI FILMS Studio provides access to the latest models in the image workspace.

---

Sources

arXiv: PiD: Fast and High-Resolution Latent Decoding with Pixel Diffusion Project Page: NVIDIA SIL: PiD GitHub: nv-tlabs/PiD Hugging Face: nvidia/PiD