Aurora: Unified Video Editing Using a VLM Agent

Aurora is an open source framework from the University of Rochester, MIT-IBM Watson AI Lab, and NVIDIA that applies a vision language model agent to video editing. Instead of requiring structured model inputs, Aurora accepts plain language edit requests and converts them into precise editing plans before passing them to a diffusion transformer.

The paper was submitted to arXiv in May 2026 by Yongsheng Yu, Ziyun Zeng, Zhiyuan Xiao, Zhenghong Zhou, Hang Hua, Wei Xiong, and Jiebo Luo.

The Problem with Existing Video Editing Models

Current video editing transformers require precise, structured inputs: reference images, spatial grounding coordinates, and carefully formatted text that matches the model's conditioning channels. That precision requirement creates a gap between what a filmmaker wants to say and what the model can accept.

If you describe an edit in natural language, such as "remove the person on the left and replace the jacket with a red coat", most editing models cannot process that without additional human work to produce structured inputs first. Aurora adds an agent layer that does that work automatically.

How the Agent Works

The VLM agent sits between the user's instruction and the diffusion transformer. When it receives a request, it uses tool calls to identify what reference images are needed, determines where spatial grounding is required, and produces a structured edit plan that aligns with the transformer's conditioning channels.

The agent was trained on supervised data for complete edit planning and on preference pairs for reliable tool use, similar to the RLHF approach that made large language models responsive to human intent rather than just competent at prediction. Flash-GRPO applied the same reinforcement learning logic to video diffusion training, tackling the alignment challenge from the training side where Aurora tackles it from the interface side.

The agent handles underspecification, requests that are vague or incomplete, by inferring the missing structure from the visual context of the input video. This is different from instruction only editing models, which fail when the user's request does not contain all the information the model needs.

AgentEdit-Bench

Aurora introduces a new benchmark, AgentEdit-Bench, designed specifically to test video editing systems under textual and visual underspecification. Existing benchmarks assume well formed inputs; AgentEdit-Bench tests performance when the user's instruction is incomplete.

Aurora outperforms instruction only baselines on all three evaluated benchmarks. The VLM agent transfers to compatible frozen video editing models, meaning the agent frontend is not locked to a single backbone.

The Modular Architecture

The key architectural decision in Aurora is the separation between the edit planner and the executor. The VLM agent produces the edit plan; the diffusion transformer executes it. These are two distinct components.

This separation means the Aurora agent can be wired to future, better video editing transformers without retraining the agent. As video diffusion models improve, the same agent interface remains valid: you upgrade the executor, not the planner. That modularity is not covered in mainstream press reporting on the paper, and it is what makes Aurora a framework rather than just a model.

For Iterative Video Editing

The use case for filmmakers is iterative editing: generating a clip, then making targeted changes to it without restarting from scratch. Describe what needs to change and Aurora handles the translation into model inputs.

The broader shift toward AI filmmaking as a production tool in 2026 has increased demand for this kind of iteration. Generating raw footage is increasingly accessible; editing that footage with precision and without re-prompting from zero is the next gap Aurora targets.

Aurora operates on video clips already generated or shot. It is not a generation tool. The distinction matters for how it fits into a production pipeline: Aurora enters after the initial generation step.

The iterative workflow also changes the review cycle between a director and an editor. Instead of exporting a new cut for every change, a plain language instruction produces the targeted revision in place. The review loop compresses from hours to minutes for the specific edits Aurora handles, which are object removal, replacement, addition, and attribute change in existing video. That speed difference matters most in productions where creative decisions depend on seeing options quickly.

Edit and generate AI video in the AI FILMS Studio workspace.

What Aurora Is and What It Requires

Aurora is a framework that adds an agent layer to a video diffusion transformer. It is not a standalone model that generates or edits video independently. A filmmaker using Aurora needs an underlying video editing transformer alongside the Aurora agent. The paper tests it with InstructVideo and similar backbone models.

The agent is the part that interprets plain language instructions, resolves missing information from visual context, and produces the structured edit plan. The transformer is the part that executes the plan. The Aurora contribution is the agent layer plus AgentEdit-Bench, not a new generative backbone.

The open source framing matters for how Aurora enters production use. The framework is released without a commercial API gate, which means researchers and filmmakers can study the agent architecture directly, connect it to alternative video editing backbones, and run it locally. The paper's code availability is the mechanism that makes the frozen model upgrade path practical for users outside the original research group.

Aurora's public release also creates a benchmark reference point. Any subsequent video editing agent can now be compared against Aurora on AgentEdit-Bench, which raises the evaluation standard for the category. That is a secondary contribution of open release, separate from direct user adoption.

The Research Collaboration Behind Aurora

The University of Rochester's Visual and AI Lab, led by Jiebo Luo, has produced multiple video understanding papers over the past decade. The MIT-IBM Watson AI Lab brings transformer architecture research. NVIDIA contributes compute infrastructure and applied AI expertise. That combination of computer vision research, model architecture knowledge, and production-scale hardware reflects how competitive video AI research is structured in 2026.

The seven-author team on the arXiv submission spans PhD students, postdoctoral researchers, and industry researchers. The University of Rochester to NVIDIA pipeline represents one of the more established academic-to-industry pathways in computer vision, and Aurora's framing as an open source framework fits a strategy of releasing research that practitioners can build on.

Why Underspecification Is the Real Problem

Most technical benchmarks test video editing models on well formed inputs where every parameter is specified. Those benchmarks do not reflect how filmmakers actually work. A director on set describes an edit in plain language: "remove the person in the back" or "make the sky darker". The gap between that instruction and a structured model input is what editing systems have required humans to fill manually until Aurora's agent layer was available.

The underspecification problem is both a user experience issue and an evaluation gap. Systems that are never tested on incomplete instructions are never trained to handle them. Aurora addresses both by introducing AgentEdit-Bench alongside the model.

AgentEdit-Bench Design

AgentEdit-Bench tests video editing systems on inputs that are vague, incomplete, or reference-dependent. It evaluates whether a system can correctly identify what information is missing and resolve it from visual context before producing an edit. Benchmark design matters because models that are not evaluated on underspecification will never be trained to handle it.

The three evaluation categories cover object replacement with unspecified reference images, spatial additions without provided bounding boxes, and referential expressions requiring the system to identify the correct subject from context. Aurora outperforms instruction only baselines on all three.

Tool Use Training vs. Standard Instruction Tuning

Aurora's VLM agent is trained with two distinct objectives. The first is producing complete edit plans from supervised data. The second is reliably selecting the correct tool at each step through preference pair training. The preference pair approach teaches the agent which tool choices lead to correct outputs and which lead to errors.

Tool selection is a skill separate from edit planning, and Aurora trains them independently to maintain reliability in each. Standard instruction tuning would blend these signals and produce an agent that is adequate at both but reliable at neither. The separation is why the Aurora agent generalizes to new video editing backbones without retraining.

What "Frozen" Means for Future Model Upgrades

The Aurora paper specifies that the VLM agent transfers to compatible frozen video editing models. "Frozen" means the backbone model's weights are not modified during agent training. The agent connects to the backbone through input and output interfaces, not by retraining the backbone itself.

When a better video editing transformer is released, the Aurora agent connects to it without retraining. The agent's upgrade path is preserved independently of backbone generations. For a field where video generation models improve every few months, that architectural choice is the difference between a framework that compounds improvements and a model that becomes obsolete.

---

Sources

arXiv: Aurora: Unified Video Editing with a Tool-Using Agent Project Page: yongshengyu.com/Aurora-Page