EditorNodesPricingBlog

Apple Depth Pro 2: Open Source Metric Depth for VFX

November 22, 2026
Apple Depth Pro 2: Open Source Metric Depth for VFX

Apple Machine Learning Research (Depth Pro 2 teaser)

Share this post:

Apple Depth Pro 2: Open Source Metric Depth for VFX

Monocular depth models often force visual effects artists into an awkward trade. Single frame networks trace crisp silhouettes around hair and foliage, yet they flicker wildly when applied to moving footage. Video depth models stabilize motion across frames, yet they soften fine edges until foreground plates bleed into the background. Apple Depth Pro 2, published in November 2026 (arXiv:2611.26603), solves both failure modes at once by estimating sharp, temporally consistent metric depth in real meters at full input resolution across photos, videos, and live feeds. You can test this open source model free in your browser below with zero login.

Studio availability. Try the public Hugging Face demo below; Apple Depth Pro 2 is not part of a paid AI FILMS Studio plan (use Google VEO 3.1 Image to Video in the video workspace for example instead).

Try Apple Depth Pro 2 Live

Upload a production still or a short video clip into the public Hugging Face Space below to inspect full resolution metric depth maps and automatic focal length estimation in your browser.

Architecture and Technical Design

Apple Depth Pro 2 processes footage through a multi view transformer operating over 16 frame windows, followed by a dedicated Dense Prediction Transformer (DPT) decoder per frame. Instead of compressing shots down to a fixed low resolution square, the network preserves full spatial detail so thin cables, window mullions, and hair strands retain clean metric discontinuities.

Apple Depth Pro 2 multi view transformer and per frame DPT decoder architecture

Depth Pro 2 multi view transformer and DPT decoder pipeline by Apple Machine Learning Research

Every model tier pairs with a shared 23.5M parameter focal length estimator (apple/DepthPro2-focal, 94 MB). When camera EXIF metadata is stripped from a graded plate or an AI generated shot, the focal length head predicts horizontal focal length in pixels so the output remains calibrated in physical meters rather than arbitrary relative disparity.

Because sky regions and distant horizons sit at optical infinity, the network outputs infinite depth cleanly on sky pixels instead of clamping them onto a flat plane behind buildings.

Four model checkpoints scale cleanly from edge laptops to studio render nodes. DepthPro2-S carries 24M parameters (97 MB), DepthPro2-B carries 97M parameters (386 MB), DepthPro2-L carries 334M parameters (1.3 GB), and the flagship DepthPro2-H reaches 872M parameters (3.5 GB).

On the Sintel and Spring video depth benchmarks, Apple Depth Pro 2 pushes into the top right Pareto frontier for boundary sharpness (Boundary F1) and temporal stability (low Jitter). Prior single frame estimators achieve crisp boundaries at the cost of heavy frame to frame flicker, while prior video networks suppress jitter at the cost of blurry foreground edges.

For compositors pulling Z depth defocus, atmospheric fog, or volumetric light shafts in Nuke and Fusion, low temporal jitter eliminates the pulsing focus breathing that usually ruins synthetic lens blur on handheld plates.

Boundary F1 sharpness versus temporal jitter chart comparing Apple Depth Pro 2 with prior depth models

Sharpness vs temporal consistency on Sintel and Spring by Apple Machine Learning Research

Hardware, VRAM, and License Matrix

Model Variant Parameters Weight Size Typical GPU VRAM (BF16) License Best Production Role
apple/DepthPro2-S 24M 97 MB ~2 to 4 GB Apple AMLR Research Live camera feeds, on set preview
apple/DepthPro2-B 97M 386 MB ~4 to 6 GB Apple AMLR Research Fast editorial dailies and pre vis
apple/DepthPro2-L 334M 1.3 GB ~8 to 12 GB Apple AMLR Research Balanced workstation Z depth passes
apple/DepthPro2-H 872M 3.5 GB ~12 to 16 GB Apple AMLR Research Hero VFX plates, hair and foliage edges
apple/DepthPro2-focal 23.5M 94 MB Shared (~0.5 GB) Apple AMLR Research Automatic focal length estimation

Check the Apple Machine Learning Research model license (LICENSE_MODEL) in the official repository before deploying weights in commercial client pipelines.

Parameter Reference

Parameter / Method Recommended Setting Compositing Effect
precision torch.bfloat16 Native GPU precision for fast inference with minimal VRAM
f_px Camera EXIF focal px or None Pass exact lens focal length when known; set None to trigger DepthPro2-focal
infer_video(frames) 16 frame temporal batches Maintains smooth frame to frame metric consistency across moving shots
Output tensor ["depth"] Float32 meters (H, W) Export as 32 bit EXR to preserve full physical distance range and sky infinity
upright(frame) Applied before transform Corrects rotated smartphone or cinema camera orientation metadata

Local Python and CLI Setup

Install the depth_pro_2 package directly from Apple's GitHub repository and load any of the four sizes in Python with bfloat16 acceleration.

pip install "git+https://github.com/apple-aiml-research/ml-depth-pro-2"
pip install av safetensors huggingface_hub
from functools import partial
import av
import torch
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from depth_pro_2 import DepthPro2, load_rgb, upright
from depth_pro_2.model import to_normalized_tensor
from depth_pro_2.network.fov import FOVEstimator

model = DepthPro2.from_pretrained("apple/DepthPro2-H", precision=torch.bfloat16)
focal_weights = hf_hub_download("apple/DepthPro2-focal", "model.safetensors")
model.fov = FOVEstimator(torch.bfloat16)
model.fov.load_state_dict(load_file(focal_weights))
model = model.to("cuda").eval()

transform = partial(to_normalized_tensor, device=torch.device("cuda"))

# Single photo with automatic focal length estimation
image, f_px = load_rgb("plate_001.jpg")
depth_m = model.infer(transform(image), f_px=f_px)["depth"]

# Full video sequence in 16 frame temporal windows
video = av.open("shot_010.mp4")
frames = (transform(upright(f)) for f in video.decode(video=0))
for idx, pred in enumerate(model.infer_video(frames)):
    frame_depth = pred["depth"]  # meters, (H, W)

Filmmaker Production Playbook

  • Export 32 bit EXR passes instead of 8 bit PNGs. Depth Pro 2 predicts true metric distances in meters. Quantizing those values into 8 bit grayscale introduces stair stepping on shallow depth of field racks. Save raw float tensors to OpenEXR so Nuke, DaVinci Resolve Fusion, or Blender can pull smooth optical defocus and volumetric haze.
  • Supply known lens metadata on cinema plates. If your shot was captured on a 35 mm prime with a known sensor width, compute f_px and pass it into model.infer. Use DepthPro2-focal automatically on archival footage or AI generated clips where lens metadata is absent.
  • Pair metric depth with relighting and matting passes. Combine a clean Depth Pro 2 Z pass with alpha mattes from BRIA RMBG 3.0 or surface normals from Open Vision Banana to build a complete 2.5D relighting and depth compositing stack from a single RGB camera.
  • Match tiers to your workstation stage. Run DepthPro2-S or DepthPro2-B during editorial timing tests, then switch to DepthPro2-H for final hero plates where thin hair edges and motion blur matter most.

Related Reading


Sources

  1. Apple Depth Pro 2 Model Hub on Hugging Face
  2. Apple ML Depth Pro 2 Official GitHub Repository
  3. Depth Pro 2 Technical Report on arXiv (arXiv:2611.26603)
  4. Depth Pro 2 Interactive Hugging Face Space