EditorNodesPricingBlog

Open Vision Banana: Open Source VFX Depth & Normals

November 22, 2026
Open Vision Banana: Open Source VFX Depth & Normals

FunAILab Open Vision Banana 9B zero shot multi task outputs

Share this post:

Open Vision Banana: Open Source VFX Depth & Normals

Building a multi pass compositing pipeline normally means maintaining separate checkpoints for semantic segmentation, instance masks, depth maps, surface normals, and image editing. Open Vision Banana (FunAILab/Open-Vision-Banana, arXiv:2611.15844) unifies all seven vision and editing tasks inside a single FLUX.2-klein Diffusion Transformer checkpoint available in both 4B and 9B parameter sizes. Filmmakers switch between semantic masks, referring object cutouts, depth, surface normals, and prompt driven edits simply by changing the [TASK: ...] prefix in the text prompt. You can test this open source model free in your browser below with zero login.

Studio availability. Try the public Hugging Face demo below; Open Vision Banana is not part of a paid AI FILMS Studio plan (use FLUX.2 Klein 9B Image to Image in the image workspace for example instead).

Try Open Vision Banana Live

Upload a film still or set photo into the public Hugging Face Space below and switch between segmentation, depth, and surface normal passes directly in your browser.

Architecture and Technical Design

Open Vision Banana swaps a fine tuned Flux2Transformer2DModel directly into the standard Flux2KleinPipeline from Hugging Face Diffusers. The team publishes both 4B and 9B checkpoints in two variants: zeroshot (trained without evaluation benchmark splits) and nonzeroshot (weight averaged across steps 82,500 to 97,500 with full task coverage).

Open Vision Banana unified generative visual perception recipe and training framework

Open Vision Banana recipe and unified task architecture by FunAILab

Instead of attaching specialized decoder heads for every pass, Open Vision Banana formats visual perception as conditional image-to-image synthesis. A structured prompt prefix such as [TASK: SEMANTIC_SEGMENTATION], [TASK: REFERRING_SEGMENTATION], or [TASK: INSTANCE_SEGMENTATION] instructs the transformer to paint exact RGB class colors or continuous surface normal vectors directly in latent space.

The pipeline preserves the input's native aspect ratio up to 1 megapixel, then resizes outputs back to plate dimensions using nearest neighbor interpolation for crisp masks or bilinear interpolation for smooth depth and normal maps.

For VFX compositors, extracting surface normals, depth, and referring object masks from the same 1 megapixel backbone guarantees pixel accurate alignment across every AOV pass without running three unrelated models.

Street scenes with reflective chrome, overlapping vehicles, and complex shadows highlight why generative perception beats rigid classifiers. Referring segmentation lets an artist type a plain description such as "motorcycle on the left" and receive a solid red mask on pure black for instant rotoscope holdouts.

Because the checkpoint retains full FLUX.2-klein generative priors, the same loaded pipeline can run cleanup edits or relighting concepts right after exporting its geometry passes.

Street motorcycle production test plate from the Open Vision Banana Space

Street test plate from Open Vision Banana Space

Hardware, VRAM, and License Matrix

Checkpoint Variant Base Model Typical VRAM (BF16) Recommended Steps License Intended Use
final_models/4B/nonzeroshot FLUX.2-klein-base-4B ~12 to 16 GB 28 steps (CFG 3.5) FLUX Non Commercial Fast workstation multi pass AOVs
final_models/9B/nonzeroshot FLUX.2-klein-base-9B ~22 to 28 GB (~14 GB CPU offload) 28 steps (CFG 3.5) FLUX Non Commercial Hero segmentation, depth, normals
final_models/9B/zeroshot FLUX.2-klein-base-9B ~22 to 28 GB 28 steps (CFG 3.5) FLUX Non Commercial Academic zero shot benchmarking

The weights inherit the FLUX Non Commercial License from black-forest-labs/FLUX.2-klein-base-9B, so verify licensing terms before using checkpoints in paid commercial client deliveries.

Parameter Reference

Parameter Recommended Value Effect on VFX Passes
num_inference_steps 28 Clean mask boundaries and smooth normal gradients
guidance_scale 3.5 Enforces exact RGB palette compliance in segmentation prompts
Resize filter (masks) Image.NEAREST Prevents anti aliased fringe colors along mask edges
Resize filter (depth / normals) Image.BILINEAR Keeps continuous depth and surface orientation gradients smooth
Max resolution Up to 1 megapixel Preserves original cinema aspect ratio cleanly

Local Python and CLI Setup

Install Diffusers 0.37.0 or newer, load the weight averaged 9B transformer into Flux2KleinPipeline, and run any perception task by swapping the prompt template.

pip install "diffusers>=0.37.0" transformers accelerate safetensors pillow
import torch
from diffusers import Flux2KleinPipeline, Flux2Transformer2DModel
from diffusers.utils import load_image
from PIL import Image

dtype = torch.bfloat16
transformer = Flux2Transformer2DModel.from_pretrained(
    "FunAILab/Open-Vision-Banana",
    subfolder="final_models/9B/nonzeroshot/checkpoint-avg82500-85000-87500-92500-97500_uniform/transformer",
    torch_dtype=dtype,
)
pipe = Flux2KleinPipeline.from_pretrained(
    "black-forest-labs/FLUX.2-klein-base-9B",
    transformer=transformer,
    torch_dtype=dtype,
)
pipe.enable_model_cpu_offload()

image = load_image("street_plate.jpg")
prompt = (
    "[TASK: REFERRING_SEGMENTATION] Segment the region that best matches this "
    'description: "motorcycle in the foreground". Paint the matching region red [220, 30, 30] '
    "and leave everything else black [0, 0, 0]. Output only the segmentation mask image."
)

result = pipe(
    prompt=prompt,
    image=image,
    guidance_scale=3.5,
    num_inference_steps=28,
    generator=torch.Generator("cuda").manual_seed(0),
).images[0]
result.resize(image.size, Image.NEAREST).save("moto_mask.png")

Filmmaker Production Playbook

  • Generate matched normal and depth AOVs for 2.5D relighting. Run the same hero still through the depth and surface normal prompts, then feed those aligned maps into LBM Relighting and FLUX.2-Klein Relight Brush or a Nuke Relight node to add practical key light bounce in post.
  • Keep color dictionaries explicit for cryptomatte style masks. In [TASK: SEMANTIC_SEGMENTATION], assign high contrast RGB triplets (for example [255, 0, 13] for actors and [2, 204, 254] for sky) so downstream color keyers separate layers cleanly with zero spill.
  • Use enable_model_cpu_offload() on 16 GB GPUs. The 9B transformer runs comfortably on consumer 16 GB cards with CPU offloading enabled, while the 4B checkpoint fits natively in VRAM for interactive batch passes.
  • Fix your random seed across passes. Passing a fixed manual_seed(0) keeps sampling behavior deterministic when extracting multiple referring masks from the same plate.

Related Reading


Sources

  1. FunAILab Open Vision Banana Model Hub on Hugging Face
  2. Open Vision Banana Paper on arXiv (arXiv:2611.15844)
  3. Open Vision Banana GitHub Repository
  4. Open Vision Banana Interactive Space