Scaling VLAs runs into two problems: action-labelled robot demonstrations are expensive, and the native controls of different robots disagree in dimension, coordinate frame and meaning. Human and web videos are far more plentiful, but they come with no robot commands. In the preprint RynnVLA-Latent (arXiv 2610.34356, 2026-10-25), authors from the Hong Kong Embodied AI Lab, CUHK, Alibaba DAMO Academy, Hupan Lab and other institutions first train a latent action model, RynnLAM, and use it to label about 30.1K hours of human, robot and simulation video from 25 sources with 745.52 million “latent action” descriptors. A VLA first learns to predict these descriptors, and only at the end is it connected to real actions using demonstrations from the target robot. The authors report 98.8% on LIBERO, 84.7% on LIBERO-Plus and an 88.6% average on RoboTwin 2.0 (Native protocol) for the 4B policy. This article focuses on three things: which constraints RynnLAM uses to keep latent actions close to “control-relevant motion” rather than mere image change; how large each benchmark lead is and what it measures; and the paper’s most informative comparison — on the same subset of about 1,000 hours, where latent-action pretraining and two forms of explicit action alignment each do better. All figures below are reported by the authors [R]; this site has not reproduced them.
RynnLAM: one 608-dimensional descriptor that separates action, local motion and camera
The idea behind latent actions is to do without native control labels and encode the change between two adjacent frames as a proxy action. The authors point out that image change also contains camera motion, static appearance and effects unrelated to the agent’s control, so a representation that reconstructs images well is not necessarily good for predicting low-level manipulation actions. RynnLAM is built on a Depth-Anything-3 backbone. It takes a pair of RGB frames and outputs a continuous descriptor made of three parts:
- 8 local motion tokens (8×64 = 512 dimensions): the decoder’s only cross-frame input, used to predict the residual change of the target-frame features relative to the source frame, not to reconstruct the target frame itself; the error is normalised by the size of the change.
- A global action latent (64 dimensions): used to predict dynamic 3D scene flow in the source camera frame. The pseudo-labels are computed from estimated depth, camera poses and off-the-shelf optical flow; after compensating for camera motion, static regions should have close to zero dynamic flow. A further contrastive loss requires the flow-warped source features to be closer to the target features than the unwarped source features are.
- A camera latent (32 dimensions): decodes the relative 6-DoF camera pose between the two frames. At the same time, gradient reversal makes the same pose decoder unable to read the camera pose from the action latent.
The five losses are weighted 5 each for reconstruction, flow and pose, 3 for feature contrast and 2 for gradient reversal. Training has two stages: 36K steps with the encoder frozen, then 20K steps with it unfrozen. The authors are careful in their wording: these objectives “encourage” camera decoupling but do not guarantee viewpoint invariance or physical identifiability; camera decoupling acts only on the global action latent, and the full descriptor keeps the camera latent, so it is not assumed to be camera-invariant.
Probe results. On the low-level control regression track of the LARY benchmark, a frozen representation plus a lightweight MLP regresses end-effector trajectories. The authors add a “relative” setting that predicts end-effector displacement from the first frame, because with short strides the absolute targets can be partly guessed from the current pose visible in the image. The full descriptor’s relative average MSE is 0.38, lower than DreamDojo 0.55, Moto 0.77 and LAPA 0.93; on relative CALVIN, however, DreamDojo is still better (0.35 vs 0.40). Removing one component at a time raises the relative average MSE to: 0.52 with a frozen encoder, 0.50 without 3D flow, 0.47 without the camera branch, 0.49 without gradient reversal, and 0.57 without residual prediction from the motion tokens. The authors also state that this probe does not directly measure whether the recovered 3D motion is accurate.
Corpus and two-stage training: pretraining uses no native actions or proprioception
The corpus comes from 25 datasets: about 30,117.68 hours, 16.63 million episodes and 3.057 billion frames. By duration, real robots account for about 12,038.97 hours, simulation about 6,139.18 hours and human interaction video about 11,939.53 hours, i.e. roughly 40%, 20% and 40%. HowTo100M alone has more episodes than all real-robot sources combined. Each clip is first captioned by Qwen3-VL, given optical flow by DPFlow and depth by Depth Anything 3. RynnLAM then acts as an offline labeller, taking frame pairs every 4 frames, such as (0, 4) and (4, 8), and producing 745.52 million descriptors. For policy training, every 6 descriptors form one prediction chunk; the training index holds 121.06 million samples, about 946K steps per epoch at a batch size of 128. Sources are sampled in proportion to their configured weight raised to the power 0.7, to balance web video and robot data.
The policy is a Mixture-of-Transformers: the vision-language backbone RynnBrain encodes multi-view images and the instruction, and an action expert with the same depth as the backbone but a hidden width of only 768 attends jointly with it at every layer. Note that “2B” and “4B” refer to the backbone size; total parameters including the action expert are 2.51B and 5.04B.
- Pretraining: flow matching generates chunks of 6 latent actions from Gaussian noise, conditioned on multi-view observations and language; neither native actions nor proprioception are used. Camera configurations are mapped to 6 canonical camera slots, and missing views are masked in both attention and the loss.
- Post-training: the same flow-matching objective is retargeted to the target robot’s action space, the full model is fine-tuned on that embodiment’s demonstrations, and proprioception is introduced only at this step. No separate latent-to-action decoder is added; the pretrained readout trunk is kept and terminated by a thin, newly initialised projection. Different robots share one unified action layout, with unused dimensions masked.
- Inference: Euler steps integrate an action chunk from noise; the whole chunk is executed before replanning. The authors state that the real-robot evaluation measures closed-loop task success and that inference speed is not an evaluation metric.
How to read the benchmark numbers
| Benchmark | RynnVLA-Latent-4B | Strongest comparison in the table | How to read it |
|---|---|---|---|
| LIBERO average | 98.8% | Xiaomi-Robotics-0 98.7% | The authors themselves say the 0.1-point difference is only a reported score difference, not evidence of statistical significance; on the Long suite, 2B (97.8%) beats 4B (97.0%) |
| LIBERO-Plus total | 84.7% | π0.5 84.4% | Evaluated directly with the LIBERO-finetuned checkpoint, with no benchmark-specific adaptation; camera perturbation 72.9% vs π0.5 78.4%, robot initial-state perturbation 64.0% vs 73.6%, and StarVLA is higher on background perturbation |
| VLABench five-track average success rate | 51.9% | π0.5 48.1% | 4B has the highest success rate in the table on every track; π0.5 leads the progress and intention scores on the common-sense track; 2B has a lower average success rate than π0.5 but a higher intention score |
| RoboTwin 2.0 Native average | 88.6% (clean 88.7%, randomized 88.4%) | LingBot-VLA 87.7% | The Native protocol includes randomized-scene demonstrations in training, and the authors state that this is not zero-shot transfer to randomized scenes; baseline numbers are taken from their respective reports |
| Six real-robot tasks | Highest in the table on every task | π0, π0.5, GR00T-N1.7 | Two dual-arm platforms, 35 trials per task per setting; the 2B comparison is mixed |
One number, two values in the paper. The RoboTwin 2.0 average success rate is 88.6% in the abstract and Table 6, but 88.3% in the introduction and conclusion. Table 6 notes that the average in these rows is the arithmetic mean of clean and randomized, and (88.7 + 88.4) ÷ 2 = 88.55, which matches 88.6% (this site’s check); the text read does not explain where 88.3% comes from.
Converting real-robot percentages back to trials. With 35 trials per task, one success is about 2.86 percentage points. On that basis (this site’s calculation), Table-Cleaning going from π0.5’s 25.71% to 4B’s 40.00% is 9 successes vs 14; Steak-Serving going from 40.00% to 57.14% is 14 vs 20; and on Plate-Placement, 4B’s 85.71% vs π0.5’s 82.86% is a difference of a single trial. The text read gives no confidence intervals or repeated runs, and the number of demonstrations per task is in appendix Table 11, which this article did not read. The real-robot platforms are the stationary Marvin-WUJI (dual 7-DoF arms with dual 20-DoF dexterous hands, 54 DoF in total) and the wheeled mobile Astribot-S1 (dual 7-DoF arms with parallel grippers). The results reflect performance after post-training on the target robot, not zero-shot deployment on an unseen embodiment.
One subset of about 1,000 hours: latent actions and explicit action alignment each have an edge
The full-scale results mix the effects of the backbone, data volume and supervision type. To separate them, the authors trained a set of variants on a subset of about 1,000 hours (RoboCOIN, EgoDex, EPIC-KITCHENS, Something-Something). Every row in Table 9 uses this subset, so the rows cannot be compared directly with full-corpus results. For human videos without native actions, the two explicit-alignment baselines use motion-capture-derived wrist poses and fingertip positions as pseudo-actions.
| Pretraining supervision | 2B LIBERO | 2B LIBERO-Plus | 4B LIBERO | 4B LIBERO-Plus |
|---|---|---|---|---|
| Shared canonical action vector with masking | 98.7 | 80.5 | 99.2 | 83.5 |
| Embodiment-specific input and output projections | 99.0 | 82.1 | 99.1 | 83.6 |
| LAPA latent-action targets | 98.2 | 80.2 | 98.0 | 81.3 |
| RynnLAM latent-action targets | 98.1 | 82.3 | 98.7 | 84.2 |
- The direction is consistent; the margins are small. At both sizes, RynnLAM is highest on LIBERO-Plus and below both explicit-alignment methods on LIBERO. The authors’ reading is that latent-action supervision may favour robustness under distribution shift, while explicit native-action alignment remains competitive for in-distribution execution. But RynnLAM is only 0.2 (2B) and 0.6 (4B) points above embodiment-specific projections, and the text read gives no number of repeats or intervals.
- Other variables. On the same subset, the RynnBrain backbone beats the Qwen-VL backbone on LIBERO-Plus by 3.5 (2B) and 4.5 (4B) points; removing the camera slots costs 0.4 and 1.3 points; replacing RynnLAM descriptors with LAPA targets costs 2.1 and 2.9 points. On LIBERO-Plus, in this set of comparisons, the backbone makes a larger difference than the choice between supervision types (this site’s comparison).
- Data scale. The authors pretrained with 10%, 30%, 50%, 70% and 100% of the corpus and evaluated only on LIBERO-Plus: both sizes are highest at 100%, 2B rises at every step, and 4B dips slightly from 10% to 30% before recovering. The exact values appear only in Figure 8, not in the text. Incidentally, 4B scores 84.2% on LIBERO-Plus with the subset of about 1,000 hours and 84.7% with the full corpus; the two differ in data composition, so this 0.5-point gap cannot be treated as a measurement of the scale effect (this site’s comparison).
How this relates to other content on this site
This site recently covered two preprints that learn latent actions from video. Frontier Radar | EgoJEPA uses only egocentric human video, learns latent actions conditioned on hand-motion differences, and adds one pair of adapters per robot; Frontier Radar | SkeLAM uses robot skeletons to split latent actions so that they can serve as action conditions for a world model. What RynnVLA-Latent adds is putting human, robot and simulation video under one latent-action target for policy pretraining, and comparing that directly with explicit action alignment on the same subset. The three papers differ in data, evaluation and baselines, so their numbers cannot be compared with one another. The paper’s starting point — native controls that differ in dimension, coordinate frame and meaning — can be read alongside the action spaces discussed in Physical AI Lab 1.5: Where Does Data Come From?; “generate one action chunk and execute all of it before replanning” relates to the action chunking covered in Physical AI Lab 1.6: Imitation Learning; and gaps of 0.1 to 0.6 points, or real-robot results from 35 trials per task, can be read with the methods of Physical AI Lab 1.8: Evaluation.
Limitations
- This is an arXiv preprint; all figures are reported by the authors [R], and this site has not reproduced them. The text read gives no confidence intervals or multi-seed results; baseline numbers on the simulation benchmarks are taken from their respective reports, and evaluation conditions may not match exactly.
- The paper gives the RoboTwin 2.0 average success rate as both 88.6% and 88.3%; this article uses 88.6%, following Table 6 and the abstract.
- The 3D-flow pseudo-labels come from depth, pose and optical flow estimated by off-the-shelf models; the LARY probe measures control predictability, not the metric accuracy of 3D motion. The authors also describe their visualisation of policy-predicted latent actions through the flow decoder as a qualitative interpretation only.
- Camera decoupling acts only on the global action latent, and the full descriptor keeps camera information; on LIBERO-Plus, camera and robot initial-state perturbations remain clearly weaker than π0.5.
- Post-training still needs demonstrations from the target robot; the real-robot results are closed-loop success rates after post-training, 35 trials per task, with no evaluation of inference speed, and they are not a hardware safety validation.
- This article is based on the paper’s HTML body text up to the conclusion, not the appendices (data collection, pseudo-action implementation, Table 11 and so on); the text read does not give the compute or time used for training. The paper lists a project page and GitHub, Hugging Face and ModelScope links; this site has not checked whether code or weights have been released there.
Links
- RynnVLA-Latent: Scaling Robotic Manipulation Foundation Models with 3D-Grounded Latent Actions (arXiv 2610.34356)
- Full paper (HTML)
- Frontier Radar | EgoJEPA: no human-to-robot retargeting; learn latent actions from egocentric human video, then add one pair of adapters and tune for each robot
- Frontier Radar | SkeLAM: robot skeletons split latent actions into “how the arm moves” and “how the scene changes”, and robot actions connect only to the first part
- Physical AI Lab 1.5: Where Does Data Come From? Working Out Demonstration Time from SO-101 Metadata, Then Separating the Units and Action Spaces of Three Sources
- Physical AI Lab 1.6: Imitation Learning — Working Out How an Error Rolls Out of the Demonstrations’ Coverage, Then Putting the ACT, Diffusion Policy and Data Scaling Numbers Back in Their Own Conditions
- Physical AI Lab 1.8: Evaluation — Splitting a “Net Gain of 2” into a Paired Table, Then Checking the Failure Labels and Random Seeds
- Physical AI Lab: a physical AI course where every number can be checked (course outline)
