FLUX ControlNet & FLUX.2 GGUF: Open Weights Shot Design

Locking a character's pose across a shot list, then holding that same character steady through a reverse angle, is the hardest continuity problem in AI assisted previsualization. FLUX.1 ControlNet (InstantX/FLUX.1-dev-Controlnet-Union) solves the pose problem with Canny and Depth conditioning, while FLUX.2 GGUF quantization brings the newer, larger FLUX.2 family down to 6 GB to 13 GB of VRAM. Both run locally and pair naturally with the Studio's image-to-image and character generator tools for planning a shot before it reaches the cloud.

Studio availability. FLUX.1 ControlNet, FLUX.2 Dev GGUF, FLUX.2 Klein GGUF, and Stable Diffusion 3.5 are not available in AI FILMS Studio. These are local open weights deployments of research releases, and the cloud hosted FLUX.2 Pro and FLUX.2 Klein 9B models remain live in the Studio. To generate images in the Studio, see the models listed on the image to image page.

FLUX.2 Klein Distillation: Sub Second Inference on Consumer GPUs

Released in January 2026, the FLUX.2 [klein] family (Black Forest Labs, github.com/black-forest-labs/flux2) is built to run generation and multi reference editing in a single unified model, under one second per image on consumer hardware. The flagship klein 9B model pairs a 9 billion parameter flow transformer with an 8 billion parameter Qwen3 text embedder, distilled down from 50 steps to just 4 inference steps.

1. klein 4B. Fits in roughly 8 GB of VRAM on an RTX 3090 or RTX 4070. It is the only FLUX.2 variant released under Apache 2.0, so its weights can be fine tuned and redistributed commercially without restriction.
2. klein 9B and 9B KV. The larger model trades some speed for higher fidelity text to image output. A separate 9B KV checkpoint caches key and value states across multi reference edits, making it faster than the 4B model for continuity heavy editing passes. Both the 9B and 9B KV checkpoints ship under the FLUX Non-Commercial License.
3. klein 4B Base and 9B Base. Undistilled, 50 step versions intended for LoRA training and research, not for production speed.

Canny and Depth ControlNet Conditioning for Character Pose Locking

FLUX.1 ControlNet adapts the original ControlNet architecture (Zhang, Rao, and Agrawala, arXiv:2302.05543) to the 12 billion parameter black-forest-labs/FLUX.1-dev base model. The adapter locks a frozen copy of the base model's encoder layers and routes a second, trainable copy through zero convolutions, meaning the new conditioning layers start at zero weight and only grow as training proceeds. That design lets the adapter learn spatial control without corrupting the pretrained diffusion weights underneath it.

The 3.3 billion parameter InstantX/FLUX.1-dev-Controlnet-Union adapter packs seven control modes into one checkpoint, selected by an integer control_mode flag:

Mode	Conditioning Type	Use for Shot Design
0	Canny edge	Lock a character silhouette and costume edges from a storyboard sketch
2	Depth map	Hold camera blocking and set geometry constant across coverage angles
4	Pose	Keep a figure's limb position fixed while changing lighting or wardrobe
3	Blur	Guide composition from a rough layout pass without hard edges

A newer Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro-2.0 checkpoint improves control fidelity on the same seven modes. Both adapters require the full black-forest-labs/FLUX.1-dev base model in memory alongside the adapter, so Canny and Depth conditioned shot locking is a workstation grade workflow rather than a laptop one. For character continuity work that does not need pixel accurate pose matching, see our character generation tutorial and the Studio's character generator page.

GGUF Block Quantization and VRAM Tradeoffs

GGUF quantization splits transformer weights into super blocks of 256 values, then assigns each sub block a non uniform bit width based on how much that block affects output quality. The resulting Q4_K_M and Q3_K_S formats keep attention projections and normalization layers closer to full precision while compressing less sensitive feed forward blocks harder, which is why a 4 bit quantized model still holds visible detail that naive uniform quantization would lose.

Community converter city96 published city96/FLUX.2-dev-gguf, a direct quantization of the 32 billion parameter FLUX.2 Dev transformer. The Q3_K_S variant is a 15.8 GB file that, combined with CPU offloading of inactive blocks, holds the resident working set inside roughly 13 GB of VRAM. A parallel conversion, unsloth/FLUX.2-klein-9B-GGUF, applies a dynamic per layer precision method to the 9B klein model, producing a 5.91 GB Q4_K_M file that runs inside 6 GB of VRAM, the same class of hardware required for the Wan 2.2 GGUF video models covered in our Wan 2.2 GGUF breakdown. As with any quantized derivative, the GGUF files carry the same non commercial restrictions as the original FLUX.2 weights they were converted from.

Stable Diffusion 3.5's MMDiT Architecture for Comparison

Where FLUX.1 and FLUX.2 use a rectified flow transformer, Stable Diffusion 3.5 (stabilityai/stable-diffusion-3.5-medium and stabilityai/stable-diffusion-3.5-large) runs on a Multimodal Diffusion Transformer (MMDiT), described in arXiv:2403.03206. MMDiT keeps separate weight streams for text and image tokens through most of the network, then joins them through shared attention blocks, and adds QK normalization to stabilize training at the 8.1 billion parameter Large scale.

SD3.5 uses three fixed, pretrained text encoders, a CLIP-L, a CLIP-G, and a T5-XXL, run together for every prompt. The 2.5 billion parameter Medium checkpoint runs text to image generation inside roughly 10 GB of VRAM, while the 8.1 billion parameter Large checkpoint needs closer to 18 GB at full bfloat16 precision, or down to single digit VRAM with 4 bit quantization through bitsandbytes. Both variants ship under the Stability AI Community License, free for research and for organizations with under one million dollars in annual revenue. For a look at the newer FLUX.2 Pro cloud pipeline these open weights models are compared against, see our FLUX.2 production image generation guide, and for scene level multi angle continuity, our Qwen Image Edit multi angle scene generation article.

Running FLUX ControlNet and FLUX.2 GGUF Locally in Python

FluxControlNetPipeline in diffusers loads the Union adapter alongside the FLUX.1 Dev base model to lock a character's pose from a Canny edge map:

import torch
from diffusers import FluxControlNetModel, FluxControlNetPipeline
from diffusers.utils import load_image

base_model = "black-forest-labs/FLUX.1-dev"
controlnet_model = "InstantX/FLUX.1-dev-Controlnet-Union"

controlnet = FluxControlNetModel.from_pretrained(controlnet_model, torch_dtype=torch.bfloat16)
pipe = FluxControlNetPipeline.from_pretrained(
    base_model, controlnet=controlnet, torch_dtype=torch.bfloat16
)
pipe.enable_model_cpu_offload()

canny_reference = load_image("hero_pose_canny_edges.png")

locked_pose_shot = pipe(
    prompt="Low angle hero shot of a detective in a rain soaked trench coat, neon backlight, 35mm cinema lens",
    control_image=canny_reference,
    control_mode=0,  # 0 = Canny edge conditioning
    width=1024,
    height=1024,
    controlnet_conditioning_scale=0.65,
    num_inference_steps=24,
    guidance_scale=3.5,
).images[0]
locked_pose_shot.save("locked_pose_shot.png")

For VRAM constrained GPUs, diffusers loads a .gguf transformer file directly through GGUFQuantizationConfig, here running FLUX.2 Klein 9B at Q4_K_M inside 6 GB of VRAM:

import torch
from diffusers import FluxTransformer2DModel, Flux2KleinPipeline, GGUFQuantizationConfig
from diffusers.utils import load_image

gguf_file = "https://huggingface.co/unsloth/FLUX.2-klein-9B-GGUF/blob/main/flux-2-klein-9b-Q4_K_M.gguf"

transformer = FluxTransformer2DModel.from_single_file(
    gguf_file,
    quantization_config=GGUFQuantizationConfig(compute_dtype=torch.bfloat16),
    torch_dtype=torch.bfloat16,
)

pipe = Flux2KleinPipeline.from_pretrained(
    "black-forest-labs/FLUX.2-klein-9B",
    transformer=transformer,
    torch_dtype=torch.bfloat16,
)
pipe.enable_model_cpu_offload()

reverse_angle_shot = pipe(
    prompt="Same character, same lighting, reverse angle coverage of the dialogue scene",
    image=[load_image("establishing_shot.png")],
    num_inference_steps=4,
    guidance_scale=4.0,
).images[0]
reverse_angle_shot.save("reverse_angle_shot.png")

Hardware and License Comparison

Model Variant	Repository	Architecture	Minimum VRAM	Disk Size	License
FLUX.1 ControlNet Union	InstantX/FLUX.1-dev-Controlnet-Union	12B base + 3.3B adapter (bfloat16)	24 GB VRAM	~30 GB	FLUX.1 [dev] Non-Commercial License
FLUX.2 Dev GGUF (Q3_K_S)	city96/FLUX.2-dev-gguf	32B rectified flow transformer	13 GB VRAM	~15.8 GB	FLUX.2-dev Non-Commercial License
FLUX.2 Klein 9B GGUF (Q4_K_M)	unsloth/FLUX.2-klein-9B-GGUF	9B distilled flow transformer	6 GB VRAM	~5.9 GB	FLUX Non-Commercial License
Stable Diffusion 3.5 Medium	stabilityai/stable-diffusion-3.5-medium	2.5B MMDiT (bfloat16)	10 GB VRAM	~5 GB	Stability AI Community License
Stable Diffusion 3.5 Large	stabilityai/stable-diffusion-3.5-large	8.1B MMDiT (bfloat16)	18 GB VRAM	~16.5 GB	Stability AI Community License

Generating with FLUX and Stable Diffusion in AI FILMS Studio

These five open weights variants run on local hardware, but AI FILMS Studio hosts the newer commercial FLUX.2 generations directly in the cloud, with no VRAM planning required:

• FLUX.2 Pro Text to Image. Live in the Image Workspace and on the image to image page for high resolution keyframe and poster generation.
• FLUX.2 Klein 9B Image to Image & FLUX.2 Klein LoRA. Both live in the Studio, covering fast reference conditioned edits and custom LoRA styles without any local GPU setup.
• Nodes Graph Editor. Inside the Nodes Graph Editor, a FLUX.2 Pro or FLUX.2 Klein image node can feed directly into a character consistency node, then into an image to video node, chaining shot design and animation in a single visual workflow.

---

Sources

ControlNet Paper (arXiv:2302.05543) | FLUX.2 GitHub (black-forest-labs/flux2) | FLUX.1 ControlNet Union (InstantX/FLUX.1-dev-Controlnet-Union) | Stable Diffusion 3.5 MMDiT Paper (arXiv:2403.03206) | Stable Diffusion 3.5 Large (stabilityai/stable-diffusion-3.5-large)