A robot world model generates future frames from the current frame and a sequence of robot actions, which makes it useful for policy evaluation and model-based planning. A common way to learn such dynamics from video without action labels is a latent action model: the visual change between two neighbouring frames is encoded as a “proxy action” and learned by reconstructing the next frame. The catch is that a reconstruction objective leads the latent action to encode the robot’s own motion together with object responses and other scene changes, so mapping real robot controls into that space later is not clean. In the preprint SkeLAM (arXiv 2610.31474, 2026-10-22), authors from Harbin Institute of Technology (Shenzhen), Show Lab at the National University of Singapore and Ping An Tech use robot skeleton images as supervision to split the latent action into a “kinematic” part and a “residual” part, and robot actions are mapped only to the kinematic part. The authors report that on 400 DROID evaluation clips, the end-effector average displacement error (ADE) is 41.15 pixels, 19.5% lower than DreamDojo’s 51.11, and the perceptual error LPIPS is 0.1310, 11.1% lower than 0.1473. This article focuses on two things: the skeleton appears only during training, so what connects robot actions at deployment; and how each of these numbers, and the policy-evaluation results, are measured. All numbers below are author-reported [R]; this site has not reproduced them.
The problem: reconstructed latent actions cannot separate “how the arm moves” from “how objects respond”
The paper’s example: similar end-effector motions can produce different object responses under different contact states. If a latent action encodes both, then when robot actions are regressed onto latent actions, the regression target carries scene-dependent variation unrelated to robot motion. The authors do not want to discard that information, because accurate simulation needs to predict exactly how robot motion affects the surrounding scene. So the design has two components: a dedicated kinematic component that serves as the target for action adaptation, and a residual component that keeps complementary visual dynamics for learning the world model.
A skeleton image describes the robot’s configuration and motion without scene appearance. Earlier work (the paper lists OSCAR among others) uses skeleton sequences or kinematic renderings directly as generation conditions. SkeLAM differs: the skeleton only supervises the kinematic latent learned from RGB, and the deployed interface uses only RGB observations and robot actions.
Method: three training stages, with the skeleton gone after the first
The latent action model consists of two Transformer VAEs, with 32 dimensions each for the kinematic and residual latents; the world-model backbone is Cosmos-Predict2.5-2B. The appendix gives this training workflow (K = thousand updates):
| Stage | What it does | Updates |
|---|---|---|
| 1A | Each branch, RGB and skeleton, learns conditional reconstruction: the RGB branch reconstructs the next frame from the current frame and both latents; the skeleton branch reconstructs the next skeleton image from the current one and the skeleton motion latent | 60K |
| 1B | Freeze the skeleton encoder and decoder. Feed the posterior mean of the RGB kinematic latent into the frozen skeleton decoder and require it to reconstruct the next skeleton image (grounding); also align it, after normalisation, with the paired skeleton motion latent | 20K |
| 1C | Cross-modal swap reconstruction: replace the RGB kinematic latent with the paired skeleton’s posterior mean, keep the residual unchanged, and still require reconstruction of the same RGB frame. The whole Stage 1 model is then frozen | 5K |
| 2A / 2B | The two components condition the world model through separate MLP projections; 2A trains only the conditioning projectors, 2B fine-tunes the full DiT backbone, with branches randomly dropped per clip | 5K / 20K |
| 3A / 3B | 3A trains only a 7→256→256→32 MLP adapter that regresses robot actions onto the posterior mean of the kinematic latent; 3B fine-tunes the full world model on the adapter’s predictions while the residual is gradually withdrawn | 2K / 30K |
What the swap does. Aligning RGB and skeleton in latent space alone does not guarantee that the RGB decoder actually accepts motion conditions from the skeleton side. Swap reconstruction requires the same decoder, the same current frame and the same target to work under both kinematic inputs; the authors say this extends cross-modal correspondence from latent space to the reconstruction level.
Branch dropout. The residual latent uses future frames, which are not available at deployment. During world-model training, one mode is sampled per clip: keep both branches, drop both, drop only the kinematic branch, or drop only the residual. The probabilities of the four modes are 0.80, 0.10, 0.05 and 0.05 for the first 1,000 Stage 2 steps and 0.50, 0.10, 0.05 and 0.35 thereafter, raising the share of “drop only the residual”. That mode matches the deployment case of “current frame plus kinematic conditions”; the supervision target is always the complete future clip.
The action interface. DROID actions are defined as the 7-dimensional change between consecutive sampled states: translation in the robot base frame, the rotation vector of the relative rotation, and the next-step gripper openness. They are neither joint-space commands nor quantities relative to the clip start. In Stage 3B, the residual is kept with probability 0.5 for the first 2,000 updates, decays linearly to 0 between updates 2,000 and 7,999, and is 0 thereafter. At inference, the model receives one observed frame and 12 action steps, with the residual and its mask set to zero, and generates 12 future frames.
How to read the DROID numbers
The evaluation set is 400 DROID clips, one from each of 400 trajectories, 100 per motion-magnitude quartile, with additional stratification by camera. Each clip has 1 conditioning frame and 12 future frames at 480×640, 15 FPS, spanning 0.8 seconds. ADE is a 2D end-effector displacement error in the image plane, in pixels, computed on displacement relative to the first frame. The authors state that trajectories with invalid end-effector tracking are excluded from scores but stay in the coverage denominator, so different methods can have different valid subsets; the text this article relies on does not give the number of valid trajectories per method.
| Method | Control interface | LPIPS ↓ | ADE (px) ↓ | Direction F1 ↑ | Magnitude correlation r ↑ |
|---|---|---|---|---|---|
| Ctrl-World | Action embedding | 0.4029 | 42.96 | 0.5888 | 0.5154 |
| OSCAR | Skeleton | 0.4788 | 43.74 | 0.5810 | 0.5029 |
| DreamDojo | Action a (large-scale interaction pretraining) | 0.1473 | 51.11 | 0.4989 | 0.3335 |
| CD-LAM | Action → latent z | 0.1586 | 53.90 | 0.4903 | 0.3179 |
| SkeLAM | Action → kinematic latent | 0.1310 | 41.15 | 0.5465 | 0.5277 |
- Ahead of whom, and by how much. The authors say SkeLAM ranks first on seven of eight metrics among action-conditioned models, including all four visual-fidelity metrics. Against the two latent-action baselines, ADE is lower by 19.5% and 23.7% and LPIPS by 11.1% and 17.4% (this site checked these against the table). Against Ctrl-World, which uses action embeddings directly, however, the ADE gap is only 1.81 pixels, and Ctrl-World and OSCAR have higher direction F1. The main gap between SkeLAM and Ctrl-World is in visual fidelity: LPIPS 0.1310 versus 0.4029.
- Ablations. Removing one component at a time under the same training budget: without skeleton supervision, ADE rises to 64.59 and magnitude correlation falls to 0.2009; without swap reconstruction, ADE rises to 66.74 and LPIPS to 0.1436; without regression supervision for action adaptation, ADE rises to 65.84; without the residual, LPIPS barely changes (0.1305) and ADE rises to 43.37. By this site’s comparison, removing any one of skeleton supervision, swap reconstruction or adapter regression raises ADE above both latent-action baselines (DreamDojo 51.11, CD-LAM 53.90); in other words, the two-component structure alone is not enough.
- What the kinematic component holds. Linear probes trained on frozen representations to predict end-effector displacement give R² of 0.3200 for the 32-dimensional kinematic component, 0.3277 for the 64-dimensional representation that adds the residual, and 0.0078 for DreamDojo; the skeleton teacher, which uses privileged skeleton input, reaches 0.4312. In a separate intervention experiment, replacing the RGB decoder’s kinematic input with the paired skeleton mean raises reconstruction MAE by only 0.53%, whereas a mismatched skeleton raises it by 11.67%; zeroing the kinematic component raises it by 5.78%, zeroing the residual by 0.61%.
Human-to-robot motion transfer and policy evaluation
Transfer has only qualitative evidence. The authors use human demonstration videos from UniSkill as motion prompts: the RGB encoder extracts latent actions from the human video and uses them directly as conditions, with one robot observation as the initial scene and all parameters fixed. The paper’s figure shows two examples, towel pushing (a downward reach of the end effector) and tissue pulling (an upward movement); the compared pretrained DreamDojo checkpoint shows limited end-effector displacement. There is no quantitative metric.
Policy evaluation is open-loop replay. On RoboArena there are 7 policies and 15 sessions, 105 episodes in total. Both world models receive the same initial observations and the actions recorded during real execution; human annotators label whether the task succeeded in the generated rollout, and this is compared with the real outcome. The only comparison is Cosmos-Predict2.5, which is the starting point of SkeLAM’s world-model backbone.
| Model | Balanced accuracy | Success recall | MCC | Spearman ρ | Pearson r | Success-rate MAE (points) |
|---|---|---|---|---|---|---|
| Cosmos-Predict2.5 | 48.44% | 0.00% | −0.0525 | −0.0756 | −0.1936 | 7.62 |
| SkeLAM | 63.02% | 33.33% | 0.2483 | 0.5347 | 0.6091 | 4.76 |
The authors themselves note that the 33.33% success recall is modest and that reproducing successful outcomes in generated rollouts remains challenging: of the episodes that succeeded in reality, only one third were also judged successful in the generated rollout. The policy-level correlations rest on only 7 policies. If each policy has 15 episodes (105 = 7×15, this site’s inference), a single policy’s success rate moves in steps of 6.67 percentage points, so a mean error of 4.76 points amounts to about 5 episodes in total across the 7 policies, and Cosmos-Predict2.5’s 7.62 to about 8 (this site’s conversion). Open-loop replay means the policy never sees the generated frames and does not react to them; the authors list closed-loop policy evaluation as future work.
How this relates to other pages on this site
This site has covered two papers that estimate policy success rates with a world model: Frontier Radar | Token-World rolls out in VLM token space without generating RGB, and Frontier Radar | PhysCanvas adds touch and task state to a canvas. SkeLAM handles an earlier step: how actions enter a world model learned from video. The three papers differ in policies, tasks, labelling and the definition of their correlations, so their numbers are not comparable. Frontier Radar | EgoJEPA also learns latent actions from video, but to train a policy; SkeLAM’s latent actions condition a world model. Differences in action definitions such as a 7-dimensional state change can be read against the action spaces in Physical AI Lab 1.5: Where Does Data Come From?; the correlations and recall over 7 policies and 105 episodes can be read with the methods of Physical AI Lab 1.8: Evaluation.
Limitations
- This is an arXiv preprint; all numbers are author-reported [R] and this site has not reproduced them. The text read gives no confidence intervals or results over multiple random seeds; DROID inference generates each clip only once.
- The motion metrics are 2D pixel quantities in a single camera view and depend on end-effector tracking; trajectories with invalid tracking are not scored, valid subsets can differ between methods, and the text read does not list the valid counts.
- Training needs paired RGB and skeleton images. The skeleton images are pre-rendered from the corresponding OSCAR subsets and depend on robot kinematics and camera projection; whether this supervision can be obtained for another robot or for uncalibrated video is not discussed in the text read.
- Human-to-robot transfer has only two qualitative examples. Policy evaluation is open-loop replay with human judgement, compared only with Cosmos-Predict2.5, with 33.33% success recall; it is neither a closed-loop evaluation nor a hardware safety validation.
- The authors say one complete training run takes about 2 days on NVIDIA A100 GPUs, with Stages 1B/1C trained on eight A100 80GB GPUs. A project page is listed in the author block, but the text read does not say whether code or weights are released, and this site has not verified that page’s content.
Links
- Skeleton-Grounded Latent Action Model for Robot World Models (arXiv 2610.31474)
- Full paper (HTML)
- Frontier Radar | Token-World: rolling out a world model in VLM token space to simulate VLA policies without RGB reconstruction
- Frontier Radar | PhysCanvas: drawing vision, touch and task state on one canvas, then using three frozen reward models to bring world-model outcomes closer to the real robot
- Frontier Radar | EgoJEPA: no human-to-robot retargeting; learn latent actions from egocentric human video, then add one pair of adapters and tune for each robot
- Physical AI Lab 1.5: Where Does Data Come From? Working Out Demonstration Time from SO-101 Metadata, Then Separating the Units and Action Spaces of Three Sources
- Physical AI Lab 1.8: Evaluation — Splitting a “Net Gain of 2” into a Paired Table, Then Checking the Failure Labels and Random Seeds
- Physical AI Lab: a physical AI course where every number can be checked (course outline)
