MolmoMotion: Ai2 Releases Open Source Model That Forecasts 3D Object Motion From Language

Allen Institute for AI (Ai2) has released MolmoMotion, an open source model that predicts where objects will move in 3D space from a short video clip and a plain-language instruction. The release includes model weights, training code, and the MolmoMotion-1M dataset containing 1.16 million annotated videos. License is Apache 2.0.

MolmoMotion introduces PointMotionBench, a new benchmark for evaluating 3D motion forecasting. The model achieves the highest accuracy on that benchmark among all tested approaches, surpassing video generation baselines, parametric 3D methods, and constant velocity estimators. The 3D trajectory outputs can condition downstream video generation models to produce physically grounded motion that follows language instructions more precisely than text alone.

How MolmoMotion Works

The model takes three inputs: a short video clip, a set of 3D points annotating a target object in the first frame, and a natural language instruction describing the intended motion. It then outputs where those 3D points will travel in the seconds that follow.

The output is expressed as real-world 3D coordinates rather than screen space pixel positions. This matters because 2D motion prediction breaks down at any camera angle change or depth variation. When MolmoMotion outputs a predicted arc in 3D, that arc remains stable regardless of how the camera moves. It can be imported directly into 3D applications or used to drive animation without depth calibration after the fact.

The language instruction is the part that distinguishes MolmoMotion from motion estimation models that predict future positions based on past trajectories alone. An instruction like "the bowl slides left and comes to a stop" changes the prediction from extrapolating existing motion to following an intended physical narrative. The model interprets those instructions and produces 3D trajectories consistent with what the instruction describes.

The preproduction uses for this kind of system are direct. Before committing to a physical setup, a filmmaker can describe an object's intended path in plain language and get a 3D prediction of where it will actually go. Models that track across video without this language guidance, like SAM-MT's real-time multi-target segmentation, tell you where objects are frame by frame. MolmoMotion tells you where they will go.

The MolmoMotion-1M Dataset

MolmoMotion-1M contains 1.16 million annotated video clips, each paired with 3D point tracks and corresponding language instructions. This makes it the largest open dataset of its kind for 3D motion forecasting with language conditioning.

The scale of the dataset matters because physical motion varies enormously across object types, surface textures, environmental conditions, and instruction phrasings. A model trained on a smaller dataset generalizes poorly to objects or motion types it has not seen. MolmoMotion-1M is large enough to cover the range of physical behaviors a production environment would reasonably encounter.

The dataset is released under the same Apache 2.0 license as the model, making it usable without restriction for commercial production work. Research teams can also use it to train competing or complementary models without needing to build their own annotation pipeline for 3D motion data.

Benchmark Performance on PointMotionBench

Ai2 created PointMotionBench as part of this release to standardize evaluation of 3D motion forecasting. The benchmark tests models against four types of baselines: video generation methods that predict motion implicitly as part of generating plausible future frames, parametric 3D methods that fit geometric models to observed motion, constant velocity estimators that extrapolate from recent movement, and prior trajectory forecasting models.

MolmoMotion achieves the highest accuracy across all tested conditions. The gap is most pronounced on predictions involving language-specified changes in direction or speed, where video generation and constant velocity approaches struggle because they have no mechanism for incorporating the instruction. Parametric 3D methods can handle geometry but lack the language grounding.

The PointMotionBench numbers give researchers and studios a quantitative reference point for comparing MolmoMotion against methods they may already be using. The benchmark will also serve as a standard evaluation suite for future models in this space.

Downstream Application: Video Generation

The most direct filmmaking application of MolmoMotion is using its 3D trajectory predictions to condition a video generation model. The prediction tells the generator where specific points on an object should end up at each future frame. The generator then produces video frames that satisfy that 3D constraint rather than inventing motion from text alone.

The improvement is clearest in cases where text prompts are ambiguous about direction or magnitude. "A flamingo walks to the right" describes a direction but not a specific path or speed. A text-to-video model without trajectory conditioning has to guess. With MolmoMotion conditioning, the generated video follows a specific 3D arc that was derived from a real observation of how flamingos actually move.

For production, this means that footage of a real reference object performing the intended motion can translate directly into generated footage that follows the same physical trajectory. Meta's SAM 3 text-driven segmentation can identify and isolate the reference object in the source video. MolmoMotion can then extract its 3D trajectory. A video generation model conditioned on that trajectory can produce the final shot.

The comparison below shows three approaches to generating video of a flamingo following a specific instruction. DaS + MolmoMotion uses the 3D trajectory prediction to guide the output. CogVideoX-5B and WAN-14B generate from the text prompt alone.

"A flamingo dips its beak into the water while walking to the right." From left to right: DaS + MolmoMotion, CogVideoX-5B, and WAN-14B.

VFX Planning and Camera Path Simulation

The 3D trajectory outputs from MolmoMotion can be imported directly into a VFX pipeline or previs workflow. A supervisor who needs to know where a prop will land after being thrown, or how a liquid will arc from a pour, can get a 3D prediction from a short reference clip without running a physics simulation from scratch.

Camera path planning benefits from the same output. A camera operator or virtual camera artist who needs to track a moving object can use the predicted 3D arc to compute where the camera needs to point at each moment in the shot. The prediction can be exported as a curve and imported into any 3D application that accepts point data.

The language instruction layer is also useful for iterating on a planned shot. If the initial prediction shows the object moving too fast for the intended framing, the instruction can be modified to describe a slower motion and a new trajectory generated without reshooting reference footage. This puts physics planning inside a prompt driven workflow rather than requiring physical restaging every time parameters change.

AI FILMS Studio provides text-to-video and image-to-video generation for creative production, connecting the kinds of trajectory informed prompting that MolmoMotion enables to actual generated output.

Getting Started With MolmoMotion

The model weights, training code, and MolmoMotion-1M dataset are all available under Apache 2.0. All download links and the arXiv paper are listed in the Sources section below.

The Apache 2.0 license means commercial use, modification, and redistribution are permitted without restriction. Studios integrating MolmoMotion into a production pipeline do not need a separate licensing agreement.

---

Sources

Project Page: allenai.org/blog/molmo-motion GitHub: allenai/MolmoMotion Hugging Face: allenai/MolmoMotion arXiv: MolmoMotion: Learning Motion Forecasting with Language Guidance Dataset: allenai/MolmoMotion-1M