SAM-MT Achieves 36 FPS Multi-Target Video Segmentation With 20 Subjects on One GPU

July 16, 2026

SAM-MT dense scene segmentation, FudanCVL, CC BY-NC-SA 4.0

Share this post:

SAM-MT Achieves 36 FPS Multi-Target Video Segmentation With 20 Subjects on One GPU

Researchers at Fudan University and Shanghai University of Finance and Economics have released SAM-MT, a real-time interactive video segmentation model that maintains over 36 frames per second while tracking 10 simultaneous targets, or 35 FPS with 20 targets, on a single NVIDIA RTX A6000 GPU. The paper appears at ECCV 2026, one of the top venues in computer vision research.

The core problem SAM-MT addresses is efficiency. Existing video segmentation approaches, including Meta's SAM2, process each target independently. Adding more targets multiplies latency proportionally. SAM-MT decouples that relationship. It processes multiple targets inside a shared global context, so tracking 20 subjects costs nearly the same compute time as tracking one.

For filmmakers and VFX artists, the application is direct. Multi-subject rotoscoping, the frame-by-frame process of isolating moving subjects from their backgrounds, has historically required significant manual labor. SAM-MT automates it in real time, at a speed that allows interactive use during production rather than as a separate post-processing pass.

How SAM-MT Differs From SAM2

SAM2 (Segment Anything Model 2) from Meta is the current standard for interactive video segmentation. It accepts a click or bounding box to define a target, then tracks that object across subsequent frames. At single-target tasks, SAM2 performs well. The limitation appears at scale: tracking five targets takes roughly five times the compute of tracking one, because SAM2 runs a full propagation pass for each object separately.

SAM-MT introduces individual-global representation modeling. Instead of running isolated passes per target, it builds a shared global context across all targets simultaneously, then uses lightweight per-target queries to maintain each subject's individual identity within that shared representation. The result is near-constant compute cost regardless of how many targets are being tracked.

SAM-MT architecture diagram showing individual-global representation modeling for multi-target video segmentation

Architecture overview, SAM-MT paper, Shen et al., ECCV 2026

The architecture separates the shared global context, which models the overall scene and is computed once per frame, from per-target representations, which track each individual subject and are computed at low cost for each additional target.

The shared context also solves a problem that appears in crowded scenes: when subjects overlap or occlude each other, a model processing targets independently can lose track of individual identities. SAM-MT's global representation maintains scene-level awareness, allowing it to recover target identity through occlusion events that would break single-target tracking approaches.

The 36 FPS Benchmark

The speed result requires specific context. SAM-MT achieves over 36 FPS with 10 targets on an NVIDIA RTX A6000, a professional-grade GPU with 48GB of VRAM. With 20 targets, the model sustains 35 FPS on the same hardware. The benchmark is for the GPU inference step, not end-to-end pipeline time including data loading and output rendering.

For comparison: professional video typically runs at 24 FPS (cinema) or 30 FPS (broadcast). A model running at 36 FPS on 10 targets can process footage in real time or slightly faster than real time. It can be used interactively during a review session, with an artist clicking to add targets and seeing the segmentation result immediately rather than waiting for a batch processing job.

The key comparison is how SAM-MT's speed scales with target count relative to SAM2. SAM2's FPS decreases proportionally as targets increase. SAM-MT maintains near-constant FPS whether tracking 5 targets or 20. The gap between the two approaches widens as target count grows, which means SAM-MT's advantage is most pronounced precisely in the scenes that are most difficult for post-production VFX teams: crowded shots with many subjects requiring individual treatment.

FPS benchmark chart comparing SAM-MT and SAM2 performance across increasing target counts

FPS comparison: SAM-MT vs. prior methods, Shen et al., ECCV 2026

VRAM Efficiency

Memory use is the other constraint that typically degrades when tracking multiple subjects. Approaches that maintain a separate memory buffer per target see VRAM requirements grow proportionally with target count. On consumer GPUs with 8–16GB of VRAM, that growth becomes a hard limit at a small number of simultaneous targets.

VRAM usage comparison chart showing SAM-MT memory efficiency across multiple targets versus baseline methods

VRAM scaling: SAM-MT vs. prior methods, Shen et al., ECCV 2026

SAM-MT's shared global context keeps memory overhead low as target count increases. The per-target queries that maintain individual identities are lightweight relative to a full separate context per subject. The result is that VRAM requirements scale sublinearly with target count, unlike approaches where memory use grows in direct proportion.

For production use, the practical benefit is the ability to run a higher target count on the same hardware, or to run the same target count on less expensive GPUs. A workstation with a consumer GPU that cannot run SAM2 on 10 targets may be able to run SAM-MT on the same task within its VRAM budget.

Quantitative Benchmark Results

The paper evaluates SAM-MT on standard video object segmentation benchmarks. Results show competitive performance against prior approaches while maintaining the speed and memory advantages described above.

The benchmarks confirm that the efficiency gains from the individual-global architecture do not come at a cost to segmentation accuracy. SAM-MT achieves competitive scores on standard video object segmentation metrics while delivering the throughput and memory profile needed for practical production use. The combination separates it from prior multi-target approaches, which typically traded one of the three properties, speed, memory, or accuracy, to achieve the others.

SAM-MT quantitative results table on standard video object segmentation benchmarks

Quantitative benchmark results, Shen et al., ECCV 2026

The Filmmaking Application

The specific use case that makes SAM-MT relevant to film production is rotoscoping. Rotoscoping is the process of drawing a mask around a moving subject, frame by frame, to separate them from the background. It is used for background replacement, selective color grading, adding or removing visual elements around a specific performer, and compositing effects that interact with a subject's outline.

Traditional rotoscoping is one of the most labor-intensive tasks in VFX work. A single shot with two or three subjects in motion can require dozens of hours of manual masking. Productions managing crowd scenes or complex multi-character sequences can spend weeks on a single sequence.

SAM-MT changes the input requirement. An artist clicks to identify each target at the start of a clip. The model then tracks all selected subjects through subsequent frames, generating segmentation masks that can be exported to a compositing pipeline. The interactive click interface, rather than a bounding box or text prompt, is specifically suited to VFX workflows where precision in the mask outline matters for a clean composite.

The 36 FPS processing speed allows that workflow to run in real time on standard video frame rates. An artist can refine the initial click selection, preview the multi-subject mask output immediately, and make corrections without waiting for a processing queue. The model also supports additional interactions mid-clip, allowing a user to add or remove targets as new subjects enter or exit the frame.

License and Access

SAM-MT is released under CC-BY-NC-SA-4.0. The non-commercial clause means the weights are available for research, personal projects, and non-commercial production workflows, but commercial productions would need to review the license terms before integrating the model into a revenue-generating pipeline. The code and model checkpoints are available on GitHub and HuggingFace.

The ARDY real-time motion generation model, released earlier in July 2026, addressed a different point in the video production pipeline: generating character motion from sparse control signals. SAM-MT operates upstream of that, in the segmentation and masking stage that determines which elements in a scene can be manipulated independently. Together they represent two of the components that a fully AI assisted VFX pipeline would require: knowing which objects are which, and knowing how to move them.

The approach the SAM-MT authors took, building an architecture specifically for multi-target efficiency rather than adapting a single-target model with repeated passes, is the same design principle that separates purpose-built production tools from general-purpose models. For VFX workflows where the target count is not one but twelve, that distinction is the difference between a tool that fits inside a working pipeline and one that cannot.

SAM-MT was accepted to ECCV 2026. The code, weights, and interactive Gradio demo are available via GitHub and HuggingFace under the CC-BY-NC-SA-4.0 license. Production studios considering commercial integration should review the license terms with their legal teams before deploying in a revenue-generating context.

Generate and composite video in the AI FILMS Studio video workspace.

AI FILMS Studio video generation workspace

Try AI FILMS Studio

Generate text-to-video and image-to-video with the latest AI models in the video workspace.

Nodes Graph Editor

Build custom AI workflows by connecting models visually in the Nodes Graph Editor.

Sources

arXiv: SAM-MT: Real-Time Interactive Multi-Target Video Segmentation GitHub: FudanCVL/SAM-MT HuggingFace: FudanCVL/SAM-MT Project Page: henghuiding.com/SAM-MT

Continue Reading

Jul 16, 2026

Millennium Media's Jonathan Yunger Spent 15 Years Making Action Films. Now He's Building an AI Production Suite.

Millennium Media president Jonathan Yunger built Arcana Labs after 15 years producing The Expendables and Rambo. His $50K Echo Hunter secured a SAG-AFTRA contract.