
One common way to train robot policies on egocentric human video is a latent action model (LAM): infer a “latent action” from two frames, then train the policy to predict it. The authors of arXiv 2611.17110 (2026-11-12) argue that latent actions learned from video reconstruction alone tend to cluster by scene appearance, so the same “grasp the cup” done by a human hand and by a robot hand need not land close together. Their method, SALA, uses language descriptions of atomic actions to estimate semantic similarity between unpaired human and robot clips, and uses that similarity to supervise the latent space. The authors report a mean success rate of 72.5% over 4 tasks on a real robot with a 63-dimensional action space, against 27.5% for a baseline with the same architecture but no latent actions (SALA also used human-video pre-training and task human demonstrations). All numbers below are reported by the authors [R]; this site has not run the code or reproduced any result.
Latent actions learned by reconstruction cluster by scene
A LAM pairs inverse and forward dynamics. An encoder infers a latent action z from observations o_t and o_(t+k), k steps apart; after an information bottleneck compresses it, a decoder reconstructs o_(t+k) from o_t and the compressed code. The objective only asks each latent to explain the change inside its own clip. It does not ask a human and a robot doing the same thing to receive similar latents. The authors argue that such latents end up encoding the scene more than the action.
Unlike world action models (WAMs), LAMs do not predict pixels during policy learning. The authors give this as the reason LAMs can train more efficiently and say inference cost may be lower; the main text gives no measured training or inference time.
Language descriptions define “soft neighbours” for unpaired clips
SALA’s tokenizer is trained on a human dataset and a separately collected robot dataset together. Both share the visual encoder, quantizer and visual decoder, and the loss has three parts:
- Visual reconstruction: human and robot clips both take part; the target is frozen DINOv2 features of the future observation.
- Action reconstruction: only robot clips have action labels. A robot-only action decoder reconstructs the action chunk from the latent code and the initial state, which ties robot latents directly to control.
- Semantic alignment: every clip carries an atomic-action description (for example “grasp the cup”). A frozen text encoder gives the language similarity between a human clip and each robot clip in the batch, and a softmax turns it into a soft neighbourhood P. Cosine similarity between latents gives Q, and a cross-entropy loss pulls Q toward P.
The alignment has two design choices. First, a stop-gradient on the robot latents lets only the human latents move, because only the robot latents are supervised by actions. Second, the quantizer is SoftVQ, not hard VQ: each token is a distribution over the codebook that changes smoothly with z, so it can express graded relations such as “grasping a cup is closer to grasping a bottle than to placing a cup”. The authors report that with hard VQ the alignment loss stays near the value of a uniform prediction, while both quantizers reach similar visual reconstruction losses.
In the implementation each clip becomes 8 latent tokens with a codebook of 256 codes; the text encoder is CLIP ViT-B/32 in simulation and Qwen3-Embedding-4B in the real-robot experiments. EgoJEPA, covered earlier on this site, conditions on hand-action differences to learn latent actions; SALA leaves the cross-embodiment correspondence to language descriptions and needs no paired human-robot data.
A frozen tokenizer labels the videos
Once trained, the tokenizer is frozen and acts as a teacher that labels human and robot clips with latent action distributions y. The future frame is used only for labelling; the policy sees only the current observation, so the teacher is not needed at deployment. The policy follows a π0-style design: Qwen3-VL-4B as the VLM and a separate 0.6B-parameter action expert that generates action chunks by flow matching. The authors add learnable latent-action queries to the action expert, fit the teacher’s distributions with a KL divergence, and let the noisy action tokens attend to those queries. Human video has no states or actions and goes only through the latent-action path, but it trains the same VLM and the same action expert. For this kind of VLA structure, see Physical AI Lab 3.1.
Training has three stages. Pre-training uses human video with the latent-action loss only; labelling needs video frames, with no action labels and no language descriptions. Mid-training uses the target robot’s teleoperation data with both the flow-matching and latent-action losses. An optional post-training stage trains on one task’s teleoperated and human demonstrations together. Language descriptions are used only to train the tokenizer, so the real-robot experiments annotate only a 983-hour subset of the human video plus the robot data.
Simulation: comparison on RoboTwin 2.0
The simulation experiment has no human video. Videos of another arm, ARX-X5 (27,000 trajectories, 68.0 hours), stand in for it, and the target robot Aloha-AgileX has only 2,500 clean-scene trajectories (9.2 hours). Evaluation covers 50 bimanual tasks with 100 rollouts each, in clean scenes (Clean) and domain-randomized scenes (Random). The WAM and LAM baselines are additionally trained on the same videos; π0.5 and X-VLA use the target-robot data only. Selected results (success rate, %):
| Method | Type | Robot-pretrained checkpoint | Clean | Random | Average |
|---|---|---|---|---|---|
| π0.5 | VLA | Yes | 70.7 | 46.6 | 58.7 |
| LingBot-VA | WAM | Yes | 75.8 | 38.4 | 57.1 |
| FastWAM | WAM | No | 79.4 | 3.2 | 41.3 |
| OpenWAM | WAM | No | 86.8 | 3.9 | 45.4 |
| RotVLA | LAM | Yes | 74.4 | 28.0 | 51.2 |
| SALA | LAM | No | 80.2 | 48.6 | 64.4 |
SALA has the highest Random and average scores; its Clean score is below OpenWAM’s. FastWAM and OpenWAM match or exceed SALA in Clean and fall to 3.2 and 3.9 in Random. The authors’ explanation is that video prediction must also model scene appearance, which is exactly what domain randomization changes. In the ablations, removing the alignment loss gives 75.5 / 36.8, and removing the stop-gradient (bidirectional alignment) gives 73.5 / 46.8. The authors also report that with alignment, the median share of ARX-X5 latent variance explained by the action rises from 0.41 to 0.67, and the share explained by the scene falls from 0.14 to 0.04. The paper gives no intervals or multi-seed results for these success rates. SWAM, covered earlier on this site, also reported RoboTwin numbers, but with different training data and settings, so they should not be compared directly with this table.
Real robot: two scaling axes for human video
The real-robot platform is a Sharpa North wheeled humanoid with two 22-DoF Sharpa Wave dexterous hands; its joint-space action has 63 dimensions covering the arms, hands, torso and neck. The data include 2,678 equivalent hours of egocentric human video (71,582 trajectories) and 888 hours of robot teleoperation data. Each of the four downstream tasks has its own teleoperated and human demonstrations; gear assembly, for example, has 1,466 teleoperated (9.0 hours) and 2,715 human demonstrations (12.0 hours). Each method is tried 20 times per task. Success rates (%):
| Method | Gear assembly | Bimanual pouring | Flower insertion | Spoon scooping | Average |
|---|---|---|---|---|---|
| GR00T N1.7 | 45 | 50 | 30 | 40 | 41.25 |
| Vanilla VLA | 40 | 25 | 20 | 25 | 27.5 |
| SALA | 75 | 80 | 55 | 80 | 72.5 |
All three are post-trained on the same task teleoperation demonstrations. The vanilla VLA shares SALA’s architecture without latent actions and can learn only from teleoperation data; SALA additionally uses human-video pre-training and the task’s human demonstrations. The authors therefore say the gap between the two measures what latent actions and the human video they enable contribute together; this comparison does not separate the two.
The two scaling axes are:
- Pre-training axis: human-video budgets of 0, 473, 1,103 and 2,678 equivalent hours, with mid- and post-training data unchanged. The authors report that validation L1 error on held-out teleoperated demonstrations (counting only the joints each task uses) falls with every increase in budget, and that the 0-hour variant is already below the vanilla VLA. This is an offline error, not a success rate, with one run per setting.
- Post-training axis: starting from the model pre-trained on 2,678 hours, with teleoperated demonstrations fixed, using 10%, 50% and 100% of each task’s human demonstrations raises the average success rate from 53.75% to 72.5%.
The paper gives no intervals for the real-robot success rates. A rough Wilson 95% interval, as in Physical AI Lab 2.8 (this site’s calculation, treating each method’s 80 trials over 4 tasks as independent and ignoring differences between tasks): 72.5% is 58/80, about 61.9%–81.1%; 27.5% is 22/80, about 18.9%–38.1%; 41.25% is 33/80, about 31.1%–52.2%. The interval for SALA does not overlap either baseline’s. If the 53.75% on the post-training axis also comes from 80 trials (the paper does not state the trial count for this set separately), it is 43/80, about 42.9%–64.3%, which overlaps the interval for 72.5%; at this sample size the trend is not yet settled.
Limitations
- This is an arXiv preprint and every number is reported by the authors [R], with no independent reproduction. The paper gives no intervals or multi-seed results for success rates; the Wilson intervals above are this site’s rough calculation from counts.
- The authors’ affiliations include Sharpa Robotics, the SJTU-Sharpa Joint Lab for Embodied AI at Shanghai Jiao Tong University, and Tsinghua University; the real-robot platform Sharpa North and the Sharpa Wave hands carry the same brand name as the first affiliation.
- In the simulation experiment, the “human video” is video of another robot arm, not of people.
- Limitations the authors state: tokenizer training requires atomic-action descriptions (the policy itself learns from video frames alone); the scaling analysis measures offline validation error, with one run per setting and pre-training budgets of up to 2,678 hours; whether the trend continues at larger scale and how it translates into real-robot success remain open.
- In the real-robot comparison SALA uses human video and task human demonstrations that the baselines do not, so the comparison is between whole methods, not between architectures at equal data.
- The paper lists a project website; this site has not opened it and has not checked whether code or models are released.
Links
- SALA: Semantically Aligned Latent Actions Enable Dual-Axis Scaling of Embodied Foundation Models (arXiv 2611.17110)
- Full paper in HTML
- Physical AI Lab 1.5: Where Does Data Come From? Working Out Demonstration Time from SO-101 Metadata, Then Separating the Units and Action Spaces of Three Sources
- Physical AI Lab 3.1: Inside VLAs — Encoding One Cup Motion into 256-Bin Tokens, Then Labelling the Scope of Three Models’ Parameter Counts and Rates
- Physical AI Lab 2.8: Reproducible Evaluation — Working Out Wald and Wilson Intervals by Hand, Sizing a Run from 1/√n, Then Bounding the Failure Rate after Zero Failures
- Frontier Radar | EgoJEPA: no human-to-robot retargeting; learn latent actions from egocentric human video, then add one pair of adapters and tune for each robot
- Frontier Radar | SWAM: feeding 47,000 hours of video and demonstrations to a world action model, with ablations of caption detail, UMI action labels and data mixture
- Physical AI Lab: a physical AI course where every number can be checked



