Frontier Radar | EgoJEPA: no human-to-robot retargeting; learn latent actions from egocentric human video, then add one pair of adapters and tune for each robot

Comparison diagram: two filled circles of the same size and color side by side, with no arrow between them; below each circle is a horizontal rule and a text label. On the left, the retargeting pipeline: a hand-crafted mapping, with calibration and world modeling redone for each robot. On the right, EgoJEPA: Stages 1–2 learn latent actions on human video and are reused, and only Stage 3 is done for each robot. — What each of the two pipelines has to redo when a new robot is added. Drawn from Figure 1(a) of arXiv 2610.20904 and the cross-embodiment efficiency paragraph of Section 4.3 (author-reported); the two items are parallel approaches, their left and right positions, circles and rules indicate no order, ranking or measurement, and the diagram contains no time figures.

A common way to learn dexterous manipulation from egocentric human video is to retarget the person’s wrist and finger poses into a particular robot’s actions with geometric rules, then pre-train on those. The preprint EgoJEPA (arXiv 2610.20904, 2026-10-14) skips that step. It conditions a JEPA-style visual prediction on the differences between hand annotations in adjacent frames to learn a latent action space that is independent of the robot body, trains a world-action model on human video, and, for each new robot, adds a pair of adapters to the action branch and runs a third stage of tuning on robot demonstrations. The authors report a 74% average out-of-domain success rate on five real-robot tasks across three embodiments, against 65% for a retargeting baseline with the same architecture, and a cumulative cross-embodiment wall-clock time of 392.7 hours instead of 743.2 (47.2% less) across the five tasks. Two accounts deserve a close read: where exactly the cost of retargeting goes, and how many trials each improvement rests on. All numbers below are author-reported [R]; this site has not reproduced them.

Two costs of retargeting

The authors put the retargeting pipeline’s problems in two groups. First, the mapping is tied to the embodiment: a hand-crafted geometric or kinematic mapping is written for one robot morphology and has to be recalibrated when the morphology changes. In the paper’s Figure 1(a), human–robot calibration alone takes 52.1 hours, 25.2% of the total pipeline time; the text read here does not say which task’s pipeline this is. Second, noise is passed straight down: hand-pose estimates suffer from occlusion, viewpoint changes and temporal jitter, and retargeting turns those errors into physically infeasible robot actions. In a stress test that adds Gaussian position noise of σ = 10 mm per axis to the human action annotations only, with robot training unchanged, the retargeting pipeline’s success on bottle cap unscrewing drops from 60% to 30%.

The other route skips hand annotations and learns latent actions directly from visual change between frames (the baseline LAPA is of this kind). The authors argue that such latents may mix controllable hand motion with camera motion and passive scene dynamics. EgoJEPA sits between the two: it uses explicit hand motion as supervision but never maps it into any robot’s coordinates.

In its related work the paper places EgoScale, EgoMimic, DexWild and similar methods in the retargeting group, and notes that the concurrent Being-H0 discretises MANO trajectories into motion tokens with a VQ-VAE; the authors’ criticism is that this tokenizer sees only hand-pose sequences, not the visual context.

Three stages: from hand-motion differences to per-robot adapters

Stage 1, human-action pre-training: the “human action” is the difference between hand annotations in adjacent frames; the annotations can be 2D/3D keypoints, MANO parameters or another hand-pose representation, and a sequence of differences is taken over a horizon H. An action encoder turns this sequence into a latent action, and a predictor combines it with the visual encodings of the preceding frames to predict the target representation of frame t+H. The target encoder is an exponential moving average of the online encoder, no gradient flows through the target branch, and the loss is L1. The authors call the resulting latent actions “effect-aware”, because they must explain what the hand’s motion did to the image.
Stage 2, latent-action-based world modeling: the Stage 1 action encoder is frozen and used to label human video with latent actions; a world-action model, conditioned on a language instruction, predicts future visual representations and latent actions together through joint flow matching. The action branch is supervised by these latent actions, not by retargeted coordinate-level robot actions.
Stage 3, embodiment-specific tuning: starting from Stage 2, an input and an output adapter (each a two-layer MLP with a ReLU in between) are inserted into the action branch, mapping robot action tokens into the shared action space and the output back into the target robot’s action space. Robot actions are also written as differences between consecutive time steps, to match the relative hand motion of Stage 1; the visual encoder is kept unchanged.

In the implementation, the Stage 1 visual encoders are initialised from the Wan2.2 VAE encoder and the predictor is a 24-layer frame-causal Transformer; the Stage 2 video Transformer is initialised from Wan2.2-TI2V-5B, and the latent actions are average-pooled along the feature dimension to 128 dimensions (1024 in Stage 1). Appendix Table A3 holds a detail that is easy to miss: the trainable modules in Stage 3 are the video DiT, the action DiT and the adapters, not the adapters alone. The three stages use 32, 64 and 64 GPUs for 3,500, 6,000 and 3,000 steps; Stages 1–2 use a single camera view without proprioception, while Stage 3 uses three views and adds proprioceptive conditioning. “Lightweight adapters” therefore describes the modules added per robot, not the only part trained per robot.

Evaluation setup and main results

Human data: a commercially acquired egocentric dataset of 1,111,943 episodes from 62,518 recording sessions, about 20,106.6 hours and 2.17 billion frames, with 64,686 task instructions and 608 scene labels; monocular RGB at about 30 Hz, with wrist poses and 21 three-dimensional keypoints per hand. The appendix states that it is licensed for non-commercial academic research.

Robot tasks: plush toy grasping with the Sharpa dexterous hand (56-dimensional actions, 750 demonstrations); cup stacking (52 dimensions, 916), zipper pulling (623) and bottle cap unscrewing (635) with the Wuji dexterous hand; cardboard box folding with the Xense Tac UMI gripper (14 dimensions, 843 demonstrations averaging 297.6 s each). Each task is tested 20 times in-domain and 20 times out-of-domain, and success means completing the whole task; out-of-domain covers unseen object instances and positions and orientations outside the training distribution.

Baselines: LAPA (latent actions without action annotations); DexWild (co-training on human video and robot demonstrations); a retargeting pipeline (EgoJEPA’s architecture with the latent actions replaced by geometrically retargeted robot targets, using the official retargeting SDKs for Sharpa and Wuji and a thumb–index fingertip distance mapping for the gripper opening); a robot-only ablation; and a variant whose Stage 1 reconstructs hand motion with a VQ-VAE. Because the retargeting baseline shares the architecture, the main difference between it and EgoJEPA is the action interface.

Method	In-domain average	Out-of-domain average
EgoJEPA	94%	74%
Retargeting pipeline	80% (derived by this site from the “14-point” margin)	65%
EgoJEPA (VQ-VAE Stage 1)	not given in the text read	63%
DexWild	not given in the text read	40%
EgoJEPA (robot-only)	not given in the text read	40%

The authors report that EgoJEPA is highest on all five tasks; LAPA’s average is not stated in the text read. The two lowest out-of-domain tasks are bottle cap unscrewing and box folding, which the authors attribute to the coordinated bimanual control needed to steady the bottle while turning the cap, and to the multi-step operations with strict geometric alignment. Without human pre-training, the out-of-domain average drops from 74% to 40%, and bottle cap unscrewing from 70% to 30%; replacing the JEPA prediction with VQ-VAE reconstruction lowers it from 74% to 63%, from which the authors conclude that learning the visual consequences of hand motion is more useful than reconstructing the motion itself.

Keep the denominators in mind: 20 trials per task per setting, so 5 percentage points on one task is one trial. With the five tasks weighted equally, the gap between 74% and 65% amounts to 9 more successes in 100 out-of-domain trials (this site’s arithmetic). The main results come without confidence intervals or significance tests.

Noise and cross-embodiment time: how to read the two accounts

Robot data efficiency (bottle cap unscrewing, out-of-domain): with only 10% of the robot demonstrations EgoJEPA reaches 35%, above the robot-only model trained on all of them (30%); with 50% it reaches 65%, above the retargeting baseline trained on all of them (60%). The paper’s Figure 5 shades 95% Wilson intervals for 20 trials. Converted to 20 trials, 35% vs 30% is 7 vs 6 successes and 65% vs 60% is 13 vs 12, one trial apart in each case; by this site’s Wilson calculation, 7/20 has an interval of roughly 18%–57% and 6/20 roughly 15%–52%, which overlap heavily. “Better at every data budget” holds for the point estimates, but these comparisons alone cannot tell the methods apart.

Annotation noise: perturbations are applied only to the human annotations in Stages 1–2, Stage 3 uses clean robot data, and evaluation is on bottle cap unscrewing out-of-domain. Position noise σ is 0, 2, 5 or 10 mm; temporal jitter has a maximum offset of 0, 3, 6 or 12 frames (about 0, 100, 200 or 400 ms at 30 Hz); annotation dropout masks 0%, 10%, 20% or 40% of frames (missing segments interpolated linearly, rotations spherically). At σ = 10 mm EgoJEPA drops from 70% to 60% (14/20 → 12/20) and retargeting from 60% to 30% (12/20 → 6/20). Of the three perturbations, temporal jitter hurts most, leaving EgoJEPA at 50% with a 12-frame offset; dropout hurts least, leaving 65% at 40% missing.

Cross-embodiment time: the authors report wall-clock time rather than GPU hours, because the retargeting baseline’s human–robot calibration runs entirely on CPUs. For box folding with the gripper, EgoJEPA’s Stage 3 takes 37.5 hours and the retargeting pipeline 193.4 hours, including 46.7 hours of CPU calibration and 109.2 hours of GPU world modeling. Subtracting, this site finds that the rest of the retargeting time is also 37.5 hours, the same as EgoJEPA’s Stage 3; the paper does not label that part, but on these numbers the saving comes from not redoing calibration and world modeling for each new embodiment. Across the five tasks, cumulative time falls from 743.2 to 392.7 hours, and (743.2 − 392.7) / 743.2 ≈ 47.2%, matching the paper.

Three points apply when reading this 47.2%. First, wall-clock time is not compute: Stage 3 uses 64 GPUs and calibration uses CPUs, so adding their hours does not count one kind of resource. Second, the text read does not say whether the 392.7 hours include the one-off Stage 1–2 pre-training. Third, the paper says retargeting redoes calibration and world modeling “whenever a new robot embodiment is introduced”, while Figure 7 lists time per task; whether the three Wuji tasks shared one calibration is not stated in the text read.

How this relates to other pages on this site

Physical AI Lab 1.5: Where Does Data Come From? covered the difficulty of human video: a human hand is not a robot hand, so it first has to be “translated” into one. EgoJEPA’s answer is to skip the explicit translation and defer it to each robot’s Stage 3, where robot demonstrations learn it. It still needs per-frame hand annotations, and each task still used between roughly 600 and 900 robot demonstrations.

Physical AI Lab 0.2: Why Now? cited EgoScale’s 20,854 hours of egocentric video; the EgoJEPA paper places EgoScale on the retargeting route. EgoJEPA’s own roughly 20,106.6 hours come from a different, commercial dataset, so the two hour counts describe different data and cannot be compared.

Frontier Radar | DexStitch takes the opposite route, explicitly converting human video into dexterous-hand data with contact-guided inpainting. TDV, from the same day, uses hand or object pose trajectories as the intermediate representation, and Being-M0.7 feeds human video and motion data into humanoid whole-body control. These papers differ in robots, tasks and metrics, so their success rates cannot be compared across papers; what separates them is the interface placed between human data and robot actions.

Physical AI Lab 1.8: Evaluation practised asking for the denominator and the conditions first. Every percentage in this article should be passed on together with “20 trials, which task, in-domain or out-of-domain”.

Limitations

This is an arXiv preprint; all numbers are author-reported [R] and this site has not reproduced them. The paper’s first page lists a project page address that this site has not opened or verified, so it is not linked here; the text read contains no information about a code release.
The human data for Stages 1–2 is a commercially licensed dataset that the appendix says is licensed for non-commercial academic research; the text read does not say whether it will be released, so outside groups would find it hard to repeat the pre-training on the same data.
There are only 20 trials per task per setting, and the main results have no confidence intervals or significance tests; the data-efficiency and noise-robustness experiments use a single task, bottle cap unscrewing.
Beyond the “lightweight adapters”, Stage 3 still fine-tunes the video and action DiTs and switches to three camera views with proprioception; each task used 623–916 robot demonstrations.
The cross-embodiment saving is counted in wall-clock time and mixes CPU calibration with 64-GPU training; whether the one-off Stage 1–2 cost is included is not stated in the text read.
Limitations the authors list themselves: severe annotation degradation (severe temporal misalignment, prolonged occlusion) can still distort the latent representations; performance drops on out-of-domain tasks that need fine-grained contact dynamics, such as bottle cap unscrewing, and the authors name explicit interaction modelling or force-feedback priors as future work.
The paper’s broader-impact statement notes that real-world deployment needs safety and physical containment protocols; the results described here are not a hardware safety validation.