When a world model is used to give a VLA policy a “mock exam”, the usual approach is to predict future RGB frames and then re-encode them into the visual tokens the policy reads. Token-World, a preprint from Southern University of Science and Technology, the Hong Kong University of Science and Technology, Peking University, MUKA Robotics and other institutions (arXiv 2610.00575, 2026-09-30, evidence level [R]), changes the interface: it predicts the future directly in the VLM visual-token space the policy consumes and generates no RGB inside the rollout loop. The authors report a Pearson correlation of 0.794 between policy success rates simulated in closed loop and those in the reference environment, against 0.583 for Ctrl-World, and a per-step simulation time of 0.359 s, the lowest among the compared methods.
The problem: the simulator outputs pixels, the policy reads tokens
The paper points out that existing “world model + VLA” pipelines mostly use an indirect interface: the world model predicts human-viewable pixels, and the VLA then encodes those pixels into visual tokens. The authors argue that this mismatch is not just a matter of compute: RGB reconstruction leads the simulator to spend capacity on visual details that may be irrelevant to the policy, whereas a rollout in token space targets the representation actually used to generate actions downstream.
The difficulty is that the VLM tokens a policy uses are high-dimensional and were optimized for perception and action, not for generative dynamics modeling. Token-World compresses first, models dynamics in the compressed space, and restores the original representation only when needed.
Method: compress the channels, keep the spatial layout
- Compact state: a separately trained semantic VAE (S-VAE) compresses frozen Qwen3-VL visual tokens along the feature dimension. In the main configuration, 108 spatial tokens of 2560 channels each are compressed to 16 channels per token; there is no spatial pooling or token fusion, so the spatial arrangement of the 108 tokens is preserved. The codec is trained with MSE, cosine and KL terms and frozen afterwards.
- Action-conditioned dynamics: the world state is the compact visual tokens plus the robot’s proprioceptive state. The model is a spatio-temporal Transformer (factorized spatial and causal temporal attention) with a fixed-size GRU state that carries history beyond the attention window. It is trained with an
x_0-parameterized flow-matching objective plus shortcut forcing, and does not rely on a pretrained video-generation model. - Autoregressive rollout: each prediction is appended to the history and conditions the next step, so the whole rollout stays in the compact space. When the policy needs input, the frozen S-VAE decoder restores the original VLM tokens, which go straight to the policy; there is no RGB reconstruction in the loop.
A separately trained, frozen convolutional decoder renders predicted tokens into 224×224 RGB frames, used only for visualization and manual task-success assessment; it takes no part in the rollout or policy inference.
Results reported by the authors
Data: in simulation, 50 manipulation tasks from RoboTwin with 100 trajectories each (50 successful, 50 failed), 5,000 in total; on real hardware, 600 successful trajectories over 6 tasks collected with a Franka Research 3 arm (Robotiq gripper, two Intel RealSense 435 cameras). Both use a 9:1 train–validation split, and all methods share the same data.
Open-loop evaluation replays recorded action sequences. Feature metrics compare the predicted and ground-truth frozen VLM representations; policy-action consistency feeds each of the two to the same frozen policy and compares the resulting actions. Frames generated by the RGB baselines are first re-encoded by the same VLM before evaluation.
| Method | RoboTwin feature cosine / NMSE | RoboTwin action cosine / NMSE | Real Franka feature cosine / NMSE | Real Franka action cosine / NMSE |
|---|---|---|---|---|
| IRASim | 0.7372 / 0.5685 | 0.9260 / 0.1533 | 0.7061 / 0.6095 | 0.9174 / 0.1702 |
| Ctrl-World | 0.7097 / 0.6437 | 0.9265 / 0.1463 | 0.7213 / 0.5719 | 0.9288 / 0.1410 |
| WorldGym | 0.7146 / 0.6157 | 0.9251 / 0.1602 | 0.6697 / 0.6930 | 0.9089 / 0.1981 |
| Token-World | 0.7714 / 0.4892 | 0.9439 / 0.1145 | 0.7506 / 0.5129 | 0.9402 / 0.1208 |
For long-horizon rollout, Token-World and Ctrl-World are compared at chunks 0, 5, 10 and 15, with 16 steps per chunk. The paper states that Token-World degrades more slowly, but shows this only in a figure (Fig. 5); the text gives no per-chunk values.
Closed-loop evaluation uses three StarVLA variants (StarVLA-OFT, StarVLA-PIv3 and StarVLA-GR00T), which share a Qwen3-VL backbone but have different action heads. Each rollout starts from the same recorded state as its reference-environment counterpart, after which predicted states are fed back to the policy to generate subsequent actions; success is judged manually from the decoded RGB rollouts. Computed over policy–task pairs, the Pearson correlation between simulated and reference success rates rises from 0.583 for Ctrl-World to 0.794. For efficiency, timed at batch size 1 on PPU-ZW810E accelerators, Token-World takes 0.359 s per step, 2.0×, 5.3× and 6.1× faster than IRASim, Ctrl-World and WorldGym respectively. The RGB baselines use 50 denoising steps and their timing includes RGB decoding and VLM re-encoding; Token-World uses 16 steps with shortcut forcing.
Ablations: a larger compressed dimension is not always better
Raising the S-VAE channel count from 8 to 128 improves reconstruction (NMSE 0.1026 → 0.0409, cosine 0.9389 → 0.9762), but worsens the two metrics used to gauge how well a latent suits diffusion modeling: SEC rises from 0.3511 to 0.4740 and LDS falls from 0.3100 to 0.1182. When the same dynamics model is actually trained at 8, 16, 32 and 48 channels, d=16 has the lowest feature NMSE (0.1517) and the highest feature cosine (0.8306) and action cosine (0.9641), while 32 and 48 predict future features worse; the authors therefore choose 16 as the operating point.
Comparing three world-state representations under the same backbone and training protocol: raw Qwen3-VL features give a feature/action NMSE of 0.7851/0.2180, SDXL-VAE latents 0.8164/0.2586, and S-VAE 0.4892/0.1145. Raw VLM features beat SDXL-VAE but are clearly worse than S-VAE; from this the authors conclude that the design of the compact representation strongly affects future-state prediction.
How this relates to earlier articles on this site
“Frontier Radar deep dive | Simulation and evaluation” discussed policy-oriented simulation fidelity; Token-World pushes that idea down to the interface: the simulator’s state is the policy’s input representation, and the evaluation metrics are computed on that representation and on the policy’s actions. “Beyond Binary Success” instead reduces the number of real-robot trials through statistics; note that 0.794 is a correlation between success rates, not a statistical guarantee that can replace real-robot trials. “VLaRL” also uses VLM latents as an interface, but for sim-to-real residual RL rather than as the state of a world model. For where world models fit in the season’s lessons, see the course outline.
Limitations
- This is an arXiv preprint; all numbers are reported by the authors and this site has not reproduced any of them. The paper says code “will be” made available; the material this article is based on contains no usable code repository.
- Open-loop metrics are computed in the frozen VLM’s feature space, which is exactly the space Token-World predicts; the RGB baselines have to generate pixels and be re-encoded before they can be compared in that space.
- Closed-loop success is judged manually from RGB rendered by the auxiliary decoder. The paper text this article is based on does not state the number of policy–task pairs or the specific task set used in closed loop.
- In the latency comparison the baselines use 50 denoising steps and Token-World uses 16, so the speedups do not reflect the difference in representation space alone.
- The RoboTwin data includes failed demonstrations; the real Franka data contains only successful trajectories. The values in ablation Table III (feature NMSE 0.1517 at d=16) differ widely from those in Tables I and IV (0.4892), and the paper does not state which data Table III was evaluated on, so take care when comparing across tables.
- The authors state that the current evaluation covers only manipulation tasks and policies sharing one VLM backbone, and that they have not yet systematically characterized what makes a semantic representation suitable for world modeling; extending to other policy backbones is future work.
Links
- arXiv 2610.00575: Token-World: World Modeling in Vision-Language Model Token Space for Robot Manipulation (corresponding author: Shanghang Zhang, Peking University)
- Full paper in HTML
