
The usual way to teach a dexterous hand from human video is to mask out the person’s arm or overlay a rendered robot on the frame, then retarget the reconstructed human hand pose into robot joint commands. The preprint RoboTryOn (arXiv 2611.05163, 2026-11-03) changes the order. Like virtual try-on, it keeps the demonstrated motion, task, objects and background and swaps only the person for the target robot, producing a robot video; an inverse dynamics model (IDM) then reads joint-action labels from the generated video. The authors report that, on two bimanual robots with dexterous hands, the average success rate on 7 tasks with no teleoperated demonstrations rises from 2.1% to 60.0%. What makes the paper worth a close read is that it measures the “visual gap” and the “action gap” separately, while its “about a third of the collection cost” counts only collection person-hours. All numbers below are author-reported [R]; this site has not reproduced them.
The idea: keep the motion, swap the body
The paper sees two gaps between a human video and a robot demonstration. Visually, the human video shows a human arm, while the deployed policy sees a robot; earlier methods narrow this gap by masking the arm or overlaying a rendered robot, but must apply the same edit to the robot’s camera stream during both training and deployment. In action, the reconstructed hand pose is itself imprecise, and human and robot hands differ in morphology and kinematics, so no exact retargeting map exists; the paper adds that even a perfect mapping would reproduce only the shape of the hand, not how firmly to grip.
RoboTryOn has three steps. The three models, and the policy trained afterwards, are all trained as conditional flow-matching models:
- Keyframe translation: keyframes are selected from the reconstructed trajectories of both wrists, at frames where the speed of both wrists reaches a local minimum (the hands are not blurred) and one wrist has moved more than a threshold from the previous keyframe; the first and last frames are always included. An image model masks the human arm, draws the hand skeleton, and fills in the target robot guided by a reference image. Each keyframe uses the previously translated robot keyframe as its reference, and the reference image for the first one selects the robot. The mask is enlarged and smoothed so its outline does not reveal whose hand it covers.
- Video synthesis: a video model interpolates between consecutive keyframes, with motion guided by the human hand-skeleton sequence. During training the skeleton is perturbed and randomly dropped, so it gives only coarse hand position and motion; finger detail comes from the model’s learned prior over robot hands.
- Pseudo action labels: the IDM predicts the chunk of joint-space actions between two frames K steps apart and is trained on real actions from teleoperated data; states are derived from the actions using the lag measured on the teleoperated data. Because the IDM estimates wrist positions less accurately than the hand reconstruction, the authors use inverse kinematics to replace the wrist position with the reconstructed one, keeping the IDM’s predicted finger joints.
The three generation and labelling models are trained on robot data only: the teleoperated demonstrations of the four in-domain tasks plus additional robot videos (not used to train any policy). At inference the same mask and skeleton conditions are extracted from human video, so no paired human–robot demonstrations are needed.
Setup: two robots, 11 tabletop tasks
Both robots are bimanual with dexterous hands: the OpenArm has two 7-DoF arms and two 6-DoF Inspire RH56F1 hands; the RB-Y1 has two 7-DoF arms and two 20-DoF WujiHand2 hands. Each robot is tested on 11 tasks:
- In-domain (4 pick-and-place): PnP Bottle, Doll, Box and Ball, with 50 teleoperated demonstrations per task per robot and 50 human videos.
- Out-of-domain (7): 50 human videos per task and no robot demonstrations: one novel object (PnP Coffee Pot), one novel goal (PnP Bottle to Shelf), two novel skills (Pouring, Brushing) and three bimanual tasks (Box Convey, Air Fryer, PnP Ball in Cup). Every teleoperated demonstration is a single-arm task.
The policy is a fine-tuned π0.5, trained for the same number of steps on each data mixture: 200 teleoperated episodes (50 per in-domain task), 550 RoboTryOn-synthesized episodes (50 for each of the 11 tasks), or all 750 together. Each task is evaluated over 28 trials with randomized object positions. The baseline EgoMimic masks the arm out of human frames; Masquerade inpaints the arm out and composites a rendering of the robot in the retargeted pose. Both use retargeted hand actions and, following their own protocols, apply the same edit to the live camera image at deployment. RoboTryOn runs on the raw camera view.
Main results: the 7 out-of-domain tasks go from almost zero to 73.0% and 46.9%
| Data (OpenArm / RB-Y1) | In-domain, 4 tasks | Out-of-domain, 7-task average | Of which bimanual, 3 tasks |
|---|---|---|---|
| Teleop only | 47.3% / 69.6% | 3.1% / 1.0% | 0.0% / 0.0% |
| Teleop + EgoMimic | 50.0% / 63.4% | 5.1% / 5.1% | 0.0% / 0.0% |
| Teleop + Masquerade | 67.9% / 63.4% | 8.2% / 7.7% | 0.0% / 0.0% |
| Teleop + RoboTryOn | 75.9% / 79.5% | 73.0% / 46.9% | 63.1% / 28.6% |
| RoboTryOn synthetic data only | 57.1% / 60.7% | 48.5% / 40.3% | 38.1% / 16.7% |
The abstract’s “2% to 60%” is the mean of the two robots’ out-of-domain averages: (3.1 + 1.0) / 2 ≈ 2.1 and (73.0 + 46.9) / 2 ≈ 60.0; the 45.9% for bimanual tasks is computed the same way. The two robots differ considerably: on the RB-Y1 the bimanual tasks reach only 28.6%. Keep the denominators in mind too. The 7 out-of-domain tasks amount to 196 trials per robot; by this site’s arithmetic, OpenArm’s 3.1% and 73.0% are about 6 and 143 successes. A column covering a single task (novel object, novel goal) has only 28 trials per robot: the 89.3% for the novel goal, for example, is 25/28. Table 1 in the text read gives no confidence intervals; per-task results are in Appendix F.2, which this site did not see.
The authors run two further checks. Ablation (OpenArm): removing the hand-skeleton guidance drops out-of-domain success from 73.0% to 21.4%, the largest drop; conditioning only on the first keyframe gives 39.3%; removing the wrist correction gives 48.0%. A different policy backbone: on RLDX-1-Img, a VLA with a different architecture, out-of-domain success rises from 0.0% to 40.3% and in-domain from 53.6% to 60.7%.
How each gap is measured, and how far each measurement goes
Visual gap: frames of each kind are encoded with π0.5’s SigLIP visual encoder and compared with real robot frames by Fréchet SigLIP distance. RoboTryOn’s synthesized video reduces it from 125.5 for raw human video to 32.7; the authors say overlay roughly halves it and masking leaves it largely unchanged. This shows that synthesized frames lie close to real robot frames in the policy’s own feature space. It does not show that contact and object motion in the synthesized video are physically correct.
Action gap: the generated videos have no ground-truth actions, so the authors replay the action labels open loop in a simulated replica of each scene and check task completion. Over 66 episodes of three pick-and-place tasks: IDM labels 69.7%, without wrist correction 63.6%, retargeted labels 34.8%. According to the authors, the difference arises mainly at the grasp: retargeting brings the hand near the object but does not grip it securely. This tests the labels themselves in simulation; it is not a real-robot policy success rate.
Whether the data “points the same way” as teleoperation: at π0.5’s pretrained weights, the authors compute the gradient of the flow-matching loss on teleoperated in-domain data and on each out-of-domain task’s data, then take the cosine similarity between them. RoboTryOn data scores 0.53, the two baselines 0.17 and 0.14; within the action expert, RoboTryOn scores 0.51 and retargeted data is slightly negative (−0.03, −0.05). From this the authors suggest that retargeted labels push the policy in a direction that conflicts with the teleoperated data, which “may explain” why the baselines gain little out of domain. This is a correlational explanation, not causal proof.
What “about a third of the collection cost” counts
The cost experiment uses only the 4 in-domain tasks: it varies the number of demonstrations per task and plots success rate against collection person-hours. The authors report that recording a human video takes about a fifth of the time of a teleoperated episode; translated human videos alone reach the success rate of 50 teleoperated demonstrations per task at about a third of the collection cost; and adding an equal number of human videos to the teleoperated data raises success from 47.3% to 71.4% with less than one extra hour of recording. The 71.4% comes from this scaling experiment; the 75.9% in Table 1 uses the 750-episode mixture covering all 11 tasks, so the two are not the same setting.
Three items are left out of this account. The first two are in the paper’s limitations section (Appendix A); the third is in Section 4.1, Implementation details. First, generation and labelling need compute, and the authors say plainly that synthesis trades collection time for compute. Second, each new robot still needs a few hours of that robot’s teleoperated video (on any tasks) to train the generative models and the IDM. Third, the three models themselves were trained on the in-domain teleoperated demonstrations. So “no robot demonstrations for the out-of-domain tasks” holds; “the system needs no robot data” does not.
The policy also inherits speed. Human demonstrators move at roughly 1.25× the teleoperation speed. Retiming the skeleton sequence to teleoperation speed before synthesis raises in-domain success from 75.9% to 85.7%, with 70.4% out of domain; synthesizing at 2× gives the fastest policy, but success falls on both task groups (57.1%, 42.9%).
How this relates to existing pages on this site
Physical AI Lab 1.5: Where Does Data Come From? says the trouble with human video is that a human hand is not a robot hand: there are no exact joint angles and no forces. RoboTryOn does not translate the human hand directly; it first swaps in a robot hand at the pixel level, then lets the IDM read off the joints. It does not solve the force problem, and the authors state that the pseudo labels cannot capture forces. Physical AI Lab 1.6: Imitation Learning covered demonstration coverage; the authors explain the in-domain gains in exactly those terms, saying the synthesized demonstrations cover more object placements and arm motions. Physical AI Lab 1.8: Evaluation practised asking for the denominator first; every percentage in this article should be quoted together with “which robot, which task group, 28 trials per task”.
Frontier Radar | EgoJEPA also tries to avoid retargeting. It learns latent actions from egocentric human video, then adds a pair of adapters and fine-tunes for each new robot. Frontier Radar | DexStitch also converts human video into dexterous-hand data, using contact-guided inpainting. The three papers differ in robots, tasks and trial counts, so their success rates cannot be compared across papers.
Limitations
- This is an arXiv preprint; every number is author-reported [R] and this site has not reproduced any of them. The authors are from KAIST and RLWRLD, and the paper was supported by RLWRLD Inc.; the author list of the RLDX-1 technical report, used as the second backbone, also includes this paper’s author Suhyeok Jang. The text read contains no information about a code or data release.
- The human videos were recorded from the position of the robot’s head camera, and the teleoperated data keep the robot’s neck and base fixed. The authors state that a moving camera, as in in-the-wild egocentric video, falls outside the training distribution of the video model and the IDM, and that the IDM always predicts a fixed neck and base.
- The actions are pseudo labels: the IDM reads joints from pixels and the wrist position comes from the hand reconstruction. They cannot capture forces, and their precision is bounded by what stereo video and hand reconstruction resolve, so tasks that depend on force or fine precision, such as tight insertion, are not yet labelled accurately enough. The wrist correction relies on the head camera calibration.
- The demonstrated motion must be one the robot can execute: the pipeline can retime it but cannot correct a position beyond the arm’s reach or a grasp the robot hand cannot form.
- The evaluation covers tabletop manipulation on two robots, 28 trials per task, and Table 1 has no confidence intervals. The authors also translated human videos into the GR-1 but could not evaluate policies on it without the physical robot.
- Cost is counted in collection person-hours only; it excludes compute for generation and labelling and the few hours of teleoperated video each new robot needs. These results are not a hardware safety validation.
Links
- RoboTryOn: Synthesizing Dexterous Robot Demonstrations from Human Videos via Virtual Try-On (arXiv 2611.05163)
- Full paper (HTML)
- Physical AI Lab 1.5: Where Does Data Come From? Working Out Demonstration Time from SO-101 Metadata, Then Separating the Units and Action Spaces of Three Sources
- Physical AI Lab 1.6: Imitation Learning — Working Out How an Error Rolls Out of the Demonstrations’ Coverage, Then Putting the ACT, Diffusion Policy and Data Scaling Numbers Back in Their Own Conditions
- Physical AI Lab 1.8: Evaluation — Splitting a “Net Gain of 2” into a Paired Table, Then Checking the Failure Labels and Random Seeds
- Frontier Radar | EgoJEPA: no human-to-robot retargeting; learn latent actions from egocentric human video, then add one pair of adapters and tune for each robot
- Frontier Radar | DexStitch: label the transition as a skill too, then cut, convert and stitch dexterous-hand skills from human videos
- Physical AI Lab: a physical AI course where every number can be checked (course outline)



