
FunAILab Open Vision Banana 9B zero shot multi task outputs
Share this post:
Open Vision Banana: Open Source VFX Depth & Normals
Building a multi pass compositing pipeline normally means maintaining separate checkpoints for semantic segmentation, instance masks, depth maps, surface normals, and image editing. Open Vision Banana (FunAILab/Open-Vision-Banana, arXiv:2611.15844) unifies all seven vision and editing tasks inside a single FLUX.2-klein Diffusion Transformer checkpoint available in both 4B and 9B parameter sizes. Filmmakers switch between semantic masks, referring object cutouts, depth, surface normals, and prompt driven edits simply by changing the [TASK: ...] prefix in the text prompt. You can test this open source model free in your browser below with zero login.
Studio availability. Try the public Hugging Face demo below; Open Vision Banana is not part of a paid AI FILMS Studio plan (use FLUX.2 Klein 9B Image to Image in the image workspace for example instead).
Try Open Vision Banana Live
Upload a film still or set photo into the public Hugging Face Space below and switch between segmentation, depth, and surface normal passes directly in your browser.
Architecture and Technical Design
Open Vision Banana swaps a fine tuned Flux2Transformer2DModel directly into the standard Flux2KleinPipeline from Hugging Face Diffusers. The team publishes both 4B and 9B checkpoints in two variants: zeroshot (trained without evaluation benchmark splits) and nonzeroshot (weight averaged across steps 82,500 to 97,500 with full task coverage).
Open Vision Banana recipe and unified task architecture by FunAILab
Instead of attaching specialized decoder heads for every pass, Open Vision Banana formats visual perception as conditional image-to-image synthesis. A structured prompt prefix such as [TASK: SEMANTIC_SEGMENTATION], [TASK: REFERRING_SEGMENTATION], or [TASK: INSTANCE_SEGMENTATION] instructs the transformer to paint exact RGB class colors or continuous surface normal vectors directly in latent space.
The pipeline preserves the input's native aspect ratio up to 1 megapixel, then resizes outputs back to plate dimensions using nearest neighbor interpolation for crisp masks or bilinear interpolation for smooth depth and normal maps.
For VFX compositors, extracting surface normals, depth, and referring object masks from the same 1 megapixel backbone guarantees pixel accurate alignment across every AOV pass without running three unrelated models.
Street scenes with reflective chrome, overlapping vehicles, and complex shadows highlight why generative perception beats rigid classifiers. Referring segmentation lets an artist type a plain description such as "motorcycle on the left" and receive a solid red mask on pure black for instant rotoscope holdouts.
Because the checkpoint retains full FLUX.2-klein generative priors, the same loaded pipeline can run cleanup edits or relighting concepts right after exporting its geometry passes.
Street test plate from Open Vision Banana Space
Hardware, VRAM, and License Matrix
| Checkpoint Variant | Base Model | Typical VRAM (BF16) | Recommended Steps | License | Intended Use |
|---|---|---|---|---|---|
final_models/4B/nonzeroshot |
FLUX.2-klein-base-4B |
~12 to 16 GB | 28 steps (CFG 3.5) |
FLUX Non Commercial | Fast workstation multi pass AOVs |
final_models/9B/nonzeroshot |
FLUX.2-klein-base-9B |
~22 to 28 GB (~14 GB CPU offload) | 28 steps (CFG 3.5) |
FLUX Non Commercial | Hero segmentation, depth, normals |
final_models/9B/zeroshot |
FLUX.2-klein-base-9B |
~22 to 28 GB | 28 steps (CFG 3.5) |
FLUX Non Commercial | Academic zero shot benchmarking |
The weights inherit the FLUX Non Commercial License from black-forest-labs/FLUX.2-klein-base-9B, so verify licensing terms before using checkpoints in paid commercial client deliveries.
Parameter Reference
| Parameter | Recommended Value | Effect on VFX Passes |
|---|---|---|
num_inference_steps |
28 |
Clean mask boundaries and smooth normal gradients |
guidance_scale |
3.5 |
Enforces exact RGB palette compliance in segmentation prompts |
| Resize filter (masks) | Image.NEAREST |
Prevents anti aliased fringe colors along mask edges |
| Resize filter (depth / normals) | Image.BILINEAR |
Keeps continuous depth and surface orientation gradients smooth |
| Max resolution | Up to 1 megapixel | Preserves original cinema aspect ratio cleanly |
Local Python and CLI Setup
Install Diffusers 0.37.0 or newer, load the weight averaged 9B transformer into Flux2KleinPipeline, and run any perception task by swapping the prompt template.
pip install "diffusers>=0.37.0" transformers accelerate safetensors pillow
import torch
from diffusers import Flux2KleinPipeline, Flux2Transformer2DModel
from diffusers.utils import load_image
from PIL import Image
dtype = torch.bfloat16
transformer = Flux2Transformer2DModel.from_pretrained(
"FunAILab/Open-Vision-Banana",
subfolder="final_models/9B/nonzeroshot/checkpoint-avg82500-85000-87500-92500-97500_uniform/transformer",
torch_dtype=dtype,
)
pipe = Flux2KleinPipeline.from_pretrained(
"black-forest-labs/FLUX.2-klein-base-9B",
transformer=transformer,
torch_dtype=dtype,
)
pipe.enable_model_cpu_offload()
image = load_image("street_plate.jpg")
prompt = (
"[TASK: REFERRING_SEGMENTATION] Segment the region that best matches this "
'description: "motorcycle in the foreground". Paint the matching region red [220, 30, 30] '
"and leave everything else black [0, 0, 0]. Output only the segmentation mask image."
)
result = pipe(
prompt=prompt,
image=image,
guidance_scale=3.5,
num_inference_steps=28,
generator=torch.Generator("cuda").manual_seed(0),
).images[0]
result.resize(image.size, Image.NEAREST).save("moto_mask.png")
Filmmaker Production Playbook
- Generate matched normal and depth AOVs for 2.5D relighting. Run the same hero still through the depth and surface normal prompts, then feed those aligned maps into LBM Relighting and FLUX.2-Klein Relight Brush or a Nuke Relight node to add practical key light bounce in post.
- Keep color dictionaries explicit for cryptomatte style masks. In
[TASK: SEMANTIC_SEGMENTATION], assign high contrast RGB triplets (for example[255, 0, 13]for actors and[2, 204, 254]for sky) so downstream color keyers separate layers cleanly with zero spill. - Use
enable_model_cpu_offload()on 16 GB GPUs. The 9B transformer runs comfortably on consumer 16 GB cards with CPU offloading enabled, while the 4B checkpoint fits natively in VRAM for interactive batch passes. - Fix your random seed across passes. Passing a fixed
manual_seed(0)keeps sampling behavior deterministic when extracting multiple referring masks from the same plate.
Related Reading
- LBM Relighting & FLUX.2-Klein Relight Brush: VFX AI
- BRIA FIBO 2.0 Edit: Open Source JSON Image Editing
- Depth Anything 3: Open Source Metric Depth for VFX
- Explore cloud image editing models on the AI Image to Image Generator page.
Sources
Continue Reading
Video & LipSync
- Video Generator
- Text to Video
- Image to Video
- Start-End Frame to Video
- Draw to Video
- Motion Control
- Video Enhancer
- Video Upscaler
- Video Extend
- Video to Video LipSync
- Audio to Video LipSync
- Image to Video LipSync
- Video FaceSwap
- Seedance 2.5
- WAN 3.0
- Seedance 2
- FLUX 3
- Minimax H3
- Vidu Q3 Pro
- Grok Imagine Video 1.5
- Gemini Omni
- Google Veo 3.1
- Kling 3.0 Pro
- Luma Ray 3.2
- LTX 2.3
- Happy Horse 1.1
- Kling 3.0 Motion
- ByteDance Upscaler
- InfiniteTalk
- InsightFace
Image & Edit
- AI Character
- AI Actor
- Art Generator
- Text to Image
- Image to Image
- LoRA Image to Image
- Draw to Edit
- Image LoRA Training
- Remove Background
- Image Enhancer
- MidJourney 8.0
- Krea 2
- OpenAI GPT Image 2.0
- Nano Banana Pro Ultra
- Nano Banana 2 Lite
- Nano Banana Pro
- Minimax Image
- Clarity Pro
- Z-Image Turbo LoRA
- Krea 2 Turbo LoRA
- Z-Image LoRA Trainer
- Qwen Image LoRA Trainer
- Background Erase Network 2 (BEN2)


