UniAction: Unified Action Conditioning
for World Models

Bowen Xue* Zizhang Li* Hong-Xing (Koven) Yu Jiajun Wu

Stanford University

*Equal contribution

One model, many different actions

Overlays mark the input action.

Compositional Action Generalization

Our model allows multiple action controls simultaneously, even though it only sees one action at a time during training.

Comparison with Conditioning Mechanisms

UniAction follows every kind of action more faithfully than visual or token-based conditioning.

Benefits of Unified Training

One model trained on all actions together outperforms experts trained on each action alone.

Temporal Generalization

UniAction starts the motion exactly when the force is applied, even at times it never saw in training.

Scene Generalization

UniAction carries these actions into scenes far beyond its training data.

Abstract

Actions that condition world models are highly heterogeneous, and thus existing action-conditioned world models are typically built for individual action types, fragmenting already limited action-annotated data across separate models. We introduce UniAction, a unified action-conditioning framework that represents any action as a sparse set of action elements, each with a discrete semantic identity and a continuous spatiotemporal grounding coordinate. By mapping these coordinates directly into the pretrained video model’s native positional space, UniAction allows heterogeneous actions to condition a shared world model within the same spatiotemporal structure. We further introduce cardinality-calibrated attention to address the bias from differing action representation granularities. We evaluate UniAction on four representative action forms: force, camera control, hand pose, and robot pose. UniAction outperforms conditioning alternatives in action following, and a single jointly trained model surpasses the action-specific experts, showing that heterogeneous action data benefit from unified model and joint training.

Overview

UniAction overview. Force, camera control, robot pose and hand pose are represented as action elements. Semantic identities index a codebook of action tokens; spatiotemporal positions are encoded with the backbone's positional encoding; action and video tokens interact in a video diffusion transformer through cardinality-calibrated attention.
UniAction overview. Every action is represented as a sparse set of action elements, each with a discrete semantic identity and a continuous spatiotemporal coordinate in the video model’s native positional space, so heterogeneous actions condition one shared world model through the same mechanism.

BibTeX

@article{xue2026uniaction,
  title   = {UniAction: Unified Action Conditioning for World Models},
  author  = {Xue, Bowen and Li, Zizhang and Yu, Hong-Xing and Wu, Jiajun},
  journal = {arXiv preprint arXiv:2610.32977},
  year    = {2026}
}