
Apple Machine Learning Research (Depth Pro 2 teaser)
Share this post:
Apple Depth Pro 2: Open Source Metric Depth for VFX
Monocular depth models often force visual effects artists into an awkward trade. Single frame networks trace crisp silhouettes around hair and foliage, yet they flicker wildly when applied to moving footage. Video depth models stabilize motion across frames, yet they soften fine edges until foreground plates bleed into the background. Apple Depth Pro 2, published in November 2026 (arXiv:2611.26603), solves both failure modes at once by estimating sharp, temporally consistent metric depth in real meters at full input resolution across photos, videos, and live feeds. You can test this open source model free in your browser below with zero login.
Studio availability. Try the public Hugging Face demo below; Apple Depth Pro 2 is not part of a paid AI FILMS Studio plan (use Google VEO 3.1 Image to Video in the video workspace for example instead).
Try Apple Depth Pro 2 Live
Upload a production still or a short video clip into the public Hugging Face Space below to inspect full resolution metric depth maps and automatic focal length estimation in your browser.
Architecture and Technical Design
Apple Depth Pro 2 processes footage through a multi view transformer operating over 16 frame windows, followed by a dedicated Dense Prediction Transformer (DPT) decoder per frame. Instead of compressing shots down to a fixed low resolution square, the network preserves full spatial detail so thin cables, window mullions, and hair strands retain clean metric discontinuities.
Depth Pro 2 multi view transformer and DPT decoder pipeline by Apple Machine Learning Research
Every model tier pairs with a shared 23.5M parameter focal length estimator (apple/DepthPro2-focal, 94 MB). When camera EXIF metadata is stripped from a graded plate or an AI generated shot, the focal length head predicts horizontal focal length in pixels so the output remains calibrated in physical meters rather than arbitrary relative disparity.
Because sky regions and distant horizons sit at optical infinity, the network outputs infinite depth cleanly on sky pixels instead of clamping them onto a flat plane behind buildings.
Four model checkpoints scale cleanly from edge laptops to studio render nodes. DepthPro2-S carries 24M parameters (97 MB), DepthPro2-B carries 97M parameters (386 MB), DepthPro2-L carries 334M parameters (1.3 GB), and the flagship DepthPro2-H reaches 872M parameters (3.5 GB).
On the Sintel and Spring video depth benchmarks, Apple Depth Pro 2 pushes into the top right Pareto frontier for boundary sharpness (Boundary F1) and temporal stability (low Jitter). Prior single frame estimators achieve crisp boundaries at the cost of heavy frame to frame flicker, while prior video networks suppress jitter at the cost of blurry foreground edges.
For compositors pulling Z depth defocus, atmospheric fog, or volumetric light shafts in Nuke and Fusion, low temporal jitter eliminates the pulsing focus breathing that usually ruins synthetic lens blur on handheld plates.
Sharpness vs temporal consistency on Sintel and Spring by Apple Machine Learning Research
Hardware, VRAM, and License Matrix
| Model Variant | Parameters | Weight Size | Typical GPU VRAM (BF16) | License | Best Production Role |
|---|---|---|---|---|---|
apple/DepthPro2-S |
24M | 97 MB | ~2 to 4 GB | Apple AMLR Research | Live camera feeds, on set preview |
apple/DepthPro2-B |
97M | 386 MB | ~4 to 6 GB | Apple AMLR Research | Fast editorial dailies and pre vis |
apple/DepthPro2-L |
334M | 1.3 GB | ~8 to 12 GB | Apple AMLR Research | Balanced workstation Z depth passes |
apple/DepthPro2-H |
872M | 3.5 GB | ~12 to 16 GB | Apple AMLR Research | Hero VFX plates, hair and foliage edges |
apple/DepthPro2-focal |
23.5M | 94 MB | Shared (~0.5 GB) | Apple AMLR Research | Automatic focal length estimation |
Check the Apple Machine Learning Research model license (LICENSE_MODEL) in the official repository before deploying weights in commercial client pipelines.
Parameter Reference
| Parameter / Method | Recommended Setting | Compositing Effect |
|---|---|---|
precision |
torch.bfloat16 |
Native GPU precision for fast inference with minimal VRAM |
f_px |
Camera EXIF focal px or None |
Pass exact lens focal length when known; set None to trigger DepthPro2-focal |
infer_video(frames) |
16 frame temporal batches | Maintains smooth frame to frame metric consistency across moving shots |
Output tensor ["depth"] |
Float32 meters (H, W) |
Export as 32 bit EXR to preserve full physical distance range and sky infinity |
upright(frame) |
Applied before transform | Corrects rotated smartphone or cinema camera orientation metadata |
Local Python and CLI Setup
Install the depth_pro_2 package directly from Apple's GitHub repository and load any of the four sizes in Python with bfloat16 acceleration.
pip install "git+https://github.com/apple-aiml-research/ml-depth-pro-2"
pip install av safetensors huggingface_hub
from functools import partial
import av
import torch
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from depth_pro_2 import DepthPro2, load_rgb, upright
from depth_pro_2.model import to_normalized_tensor
from depth_pro_2.network.fov import FOVEstimator
model = DepthPro2.from_pretrained("apple/DepthPro2-H", precision=torch.bfloat16)
focal_weights = hf_hub_download("apple/DepthPro2-focal", "model.safetensors")
model.fov = FOVEstimator(torch.bfloat16)
model.fov.load_state_dict(load_file(focal_weights))
model = model.to("cuda").eval()
transform = partial(to_normalized_tensor, device=torch.device("cuda"))
# Single photo with automatic focal length estimation
image, f_px = load_rgb("plate_001.jpg")
depth_m = model.infer(transform(image), f_px=f_px)["depth"]
# Full video sequence in 16 frame temporal windows
video = av.open("shot_010.mp4")
frames = (transform(upright(f)) for f in video.decode(video=0))
for idx, pred in enumerate(model.infer_video(frames)):
frame_depth = pred["depth"] # meters, (H, W)
Filmmaker Production Playbook
- Export 32 bit EXR passes instead of 8 bit PNGs. Depth Pro 2 predicts true metric distances in meters. Quantizing those values into 8 bit grayscale introduces stair stepping on shallow depth of field racks. Save raw float tensors to OpenEXR so Nuke, DaVinci Resolve Fusion, or Blender can pull smooth optical defocus and volumetric haze.
- Supply known lens metadata on cinema plates. If your shot was captured on a 35 mm prime with a known sensor width, compute
f_pxand pass it intomodel.infer. UseDepthPro2-focalautomatically on archival footage or AI generated clips where lens metadata is absent. - Pair metric depth with relighting and matting passes. Combine a clean Depth Pro 2 Z pass with alpha mattes from BRIA RMBG 3.0 or surface normals from Open Vision Banana to build a complete 2.5D relighting and depth compositing stack from a single RGB camera.
- Match tiers to your workstation stage. Run
DepthPro2-SorDepthPro2-Bduring editorial timing tests, then switch toDepthPro2-Hfor final hero plates where thin hair edges and motion blur matter most.
Related Reading
- Depth Anything 3: Open Source Metric Depth for VFX
- BRIA RMBG 3.0: Open Source VFX Background Removal
- FlowHMR: Open Source 3D Human Motion Capture
- Explore cloud video production models on the AI Image to Video Generator page.
Sources
Continue Reading
Video & LipSync
- Video Generator
- Text to Video
- Image to Video
- Start-End Frame to Video
- Draw to Video
- Motion Control
- Video Enhancer
- Video Upscaler
- Video Extend
- Video to Video LipSync
- Audio to Video LipSync
- Image to Video LipSync
- Video FaceSwap
- Seedance 2.5
- WAN 3.0
- Seedance 2
- FLUX 3
- Minimax H3
- Vidu Q3 Pro
- Grok Imagine Video 1.5
- Gemini Omni
- Google Veo 3.1
- Kling 3.0 Pro
- Luma Ray 3.2
- LTX 2.3
- Happy Horse 1.1
- Kling 3.0 Motion
- ByteDance Upscaler
- InfiniteTalk
- InsightFace
Image & Edit
- AI Character
- AI Actor
- Art Generator
- Text to Image
- Image to Image
- LoRA Image to Image
- Draw to Edit
- Image LoRA Training
- Remove Background
- Image Enhancer
- MidJourney 8.0
- Krea 2
- OpenAI GPT Image 2.0
- Nano Banana Pro Ultra
- Nano Banana 2 Lite
- Nano Banana Pro
- Minimax Image
- Clarity Pro
- Z-Image Turbo LoRA
- Krea 2 Turbo LoRA
- Z-Image LoRA Trainer
- Qwen Image LoRA Trainer
- Background Erase Network 2 (BEN2)


