Teaching a robot from human video means crossing two gaps at once: the frames show a human hand rather than a robot, and the video carries no robot action labels. The preprint TDV (arXiv 2610.21792, 2026-10-15) answers by never showing the policy an image. From a single human demonstration it extracts only the pose trajectory of the hand or the object, writes that trajectory in the frame of a reference object, expands one demonstration into 15 or 50 by “changing the starting point”, and trains a diffusion policy whose inputs and outputs are trajectories. The authors report a 91.8% average success rate over 12 RLBench tasks, and on a real robot, after augmenting one human demonstration to 50 trajectories, 38/39 position-generalization trials across three object tasks. Two distinctions come first when reading the paper: the simulation results use ground-truth poses from the environment and skip the “extract from human video” step entirely; only the real-robot results test the whole chain, and each cell there has a denominator of just 5 or 13 trials. All numbers below are author-reported [R]; this site has not reproduced them.
The idea: let pose trajectories sidestep both the visual gap and the missing actions
The paper sorts existing approaches into two kinds and names a weakness for each: image editing that swaps the human in the frame for a robot, or erases the human, depends on the quality of the edit; flow as an intermediate representation is robust to visual differences between embodiments, but requires the deployment camera viewpoint to match the one used during data collection. DexStitch, covered earlier on this site, belongs to the first kind: it inpaints the human hand into a dexterous robot hand and hands the result to a visual policy. TDV does not give the policy images at all. The authors argue that a manipulation task is about the state changes of the task-relevant objects rather than about who performs them, so a trajectory made of 6D object poses records the state transitions of the manipulation independently of visual domain and embodiment.
A complete demonstration is first cut into subtask segments wherever the moving entity or the reference object changes, and each segment is assigned to one of two types. In pouring, for example, grasping the kettle is a hand task and the pouring that follows is an object task:
- Hand tasks: HaMeR reconstructs a MANO hand model with 21 joints. Because the experiments use a two-finger gripper, the authors take three joints, the thumb MCP, the thumb IP and the index MCP, and build with cross products a frame aligned with the gripper’s end-effector pose; its position is the midpoint of the thumb MCP and index MCP.
- Object tasks: TRELLIS first generates a 3D mesh from one close-up image of the object; the mesh has no scale, so a global scale factor is estimated by similarity ICP against a point cloud segmented with SAM2. With the task description as a text prompt, GroundingSAM gives the first-frame mask, and SAM2’s video predictor propagates it to later frames; finally FoundationPose uses the mesh and the mask to estimate the object’s 6D pose from RGB-D input.
Both kinds of trajectory are written in the reference object’s coordinate frame to remove the effect of viewpoint changes. This is the point made in Physical AI Lab 1.1: Coordinate Frames and Kinematics: change the origin and every number for the same position changes; attach the trajectory to the reference object and wherever the camera moves, the trajectory itself is unaffected. Note that the object branch needs depth and a close-up image of the object; not just any ordinary video can be fed in directly.
How one demonstration becomes 50: trajectory generation by changing the start
Physical AI Lab 1.5: Where Does Data Come From? covered synthetic routes such as MimicGen, which automatically amplify a few demonstrations; TDV also cites MimicGen for the advantages of trajectories as an intermediate representation in computational efficiency and data augmentation. Its augmentation rule is restrained: to keep the original trajectory feasible, it only randomizes the initial pose of the target object within the workspace, and the endpoint stays fixed. Because the trajectory is referenced to the reference object’s frame, a change in the reference object’s pose can be treated as an equivalent change in the target object’s pose.
- Position: compute the vectors from the original start and from the new start to the fixed endpoint, and project them onto the XY plane; normalize the original trajectory by the length of the original projected vector, rescale it by the length of the new projected vector, and rotate it by the angle r between the two projected vectors, giving a trajectory that starts at the origin; finally translate it by the offset vector from the original start to the new start.
- Orientation: keep the original orientation over a fixed proportion α∈(0,1) at the end of the trajectory so that the final operation stays consistent; between the new start orientation and that terminal region, interpolate smoothly with spherical linear interpolation (Slerp). The paper does not give the value of α.
An example with hypothetical numbers (not data from the paper): if the original start lies 0.30 m from the endpoint in the XY plane and the new start lies 0.15 m away with a 40° difference in bearing, the whole trajectory is scaled by 0.15 ÷ 0.30 = 0.5, rotated by 40° and translated to the new start, while the orientation near the endpoint follows the original demonstration. The generated trajectory keeps the shape of the original demonstration; only its start changes. The text read here does not separately explain how the height direction is handled in this step.
Policy and closed-loop execution: a trajectory diffusion transformer plus a switch policy
The policy learns f(the next n steps of the target trajectory | the past h steps of the target trajectory, task description l, subtask signal s), and a DDPM approximates this conditional distribution. The history trajectory is mapped to a latent space by a two-layer MLP, the task description is tokenized with BERT, and the subtask signal is a learnable embedding; with modality-specific positional encodings, the three are fused by a transformer encoder, and the decoder uses DiT blocks conditioned on the fused features and the timestep embedding. The authors stress two design choices: the input is a stretch of trajectory history rather than just the current pose, which matters for tasks that revisit the same pose; and the encoder is decoupled from the denoising process to make inference more efficient. The paper does not give values for h, n or the number of denoising steps.
Multi-step tasks advance through a switch policy: an MLP classification network that judges from the history trajectory and the task description whether the current subtask is complete. When it outputs 1, the subtask signal is incremented and the gripper state changes; when the signal reaches the maximum number of subtasks, the task is complete. In training, only the final frame of each subtask segment is labeled 1 and every earlier frame 0; to handle the class imbalance, the loss weight ratio is set to 1:100.
At deployment, object tasks use FoundationPose to estimate the poses of the moving object and the reference object in real time, while hand tasks use the gripper’s end-effector pose as the pose of the moving entity; these poses are fed back into the diffusion policy’s history sequence, closing the loop. The authors report that the whole pipeline runs in real time at 30 Hz on a single NVIDIA RTX 4090 and keeps executing even when an object is dropped during manipulation. The text read here does not state how the object trajectory is converted into end-effector commands for the arm.
RLBench: a test of “trajectories as observations”, not of learning from human video
The simulation experiments use 12 RLBench tasks, each with 2 to 60 variants (differing in color, shape and target position), with 100 demonstrations per task for training and 25 reserved for evaluation; results are averaged over 3 random seeds of 25 trials each. The paper states that no trajectory extraction from human video is performed here: observations of both the moving and the reference objects come directly from the environment’s ground truth. In other words, these numbers test whether pose trajectories work as policy observations; they contain no pose-estimation error and involve no human-to-robot transfer.
| Task | OpenVLA | RVT2 | 3D-DA | SPOT | TDV |
|---|---|---|---|---|---|
| Close Jar | 1.3 | 100.0 | 49.3 | 98.7 | 100.0 |
| Drag Stick | 1.3 | 100.0 | 100.0 | 80.0 | 100.0 |
| Insert Peg | 0.0 | 41.3 | 64.0 | 78.7 | 98.7 |
| Meat off Grill | 4.0 | 94.6 | 94.6 | 100.0 | 100.0 |
| Place Cups | 2.0 | 40.0 | 22.6 | 62.7 | 73.3 |
| Place Wine | 4.6 | 86.6 | 92.0 | 100.0 | 98.7 |
| Put in Cupboard | 0.0 | 65.3 | 86.6 | 42.7 | 98.7 |
| Put in Safe | 4.6 | 96.0 | 100.0 | 100.0 | 98.7 |
| Screw Bulb | 0.0 | 88.0 | 76.0 | 48.0 | 81.3 |
| Sort Shape | 0.0 | 48.0 | 49.3 | 32.0 | 69.3 |
| Stack Blocks | 0.0 | 68.0 | 71.0 | 94.0 | 88.0 |
| Stack Cups | 0.0 | 76.0 | 53.3 | 96.0 | 94.7 |
| Average | 1.5 | 71.6 | 75.3 | 79.4 | 91.8 |
The table shows mean success rates (%) and omits the standard deviations given in the paper. The comparison conditions are not fully matched: RVT2 and 3D-DA were retrained on these 12 tasks with their official pipelines; OpenVLA is the released checkpoint fine-tuned with LoRA; SPOT’s implementation is not public, so its numbers are taken directly from its original paper. The text read here also does not describe the baselines’ inputs; when comparing, keep in mind that TDV receives ground-truth poses here and does not have to estimate them from images itself.
Checking the paper’s sentences against the table: the abstract says “near 100% across seven tasks”, and exactly 7 tasks show TDV at 98.7 or above. The text says TDV ranks first in 8 of 12 tasks, but by the table TDV is alone at the top in 4 (Insert Peg, Place Cups, Put in Cupboard, Sort Shape), and only 7 even when ties for first are counted. The text says TDV leads by over 20% on Insert Peg and Sort Shape; in the table both lead the next-best method by exactly 20.0 percentage points. The text says the average surpasses the previous state of the art by 13.4%; against SPOT’s 79.4 the table gives 12.4 percentage points, or about 15.6% relative (this site’s calculation), neither of which is 13.4, and the paper does not say how that figure was computed. The authors attribute the weaker results on Screw Bulb and Stack Blocks to infeasible motion planning in RLBench and to errors from unseen relative spatial configurations.
Real robot: how many trajectories one demonstration must be augmented to
The real setup is a UR5 arm with a Robotiq 2F-85 gripper, with an Intel RealSense D435 camera on an adjustable 6-DoF mount. The six tasks fall into two types: object tasks Cup-Coaster (put a cup on a coaster), Stitch-Card (put a Stitch toy on a card) and Pour-Water (pour water); hand tasks Pull-Drawer (pull a drawer open), Unplug-Charger (unplug a charger) and Pull-Tissue (pull a tissue).
Spatial generalization was tested only on the three object tasks; the authors reason that the relative motion in hand tasks is usually consistent and the reference object’s pose can be estimated in real time, so one demonstration is often enough. In a 45 cm × 50 cm workspace, 13 random positions test position generalization (PG), and random orientation perturbations at those 13 positions test pose generalization (PoG):
| Training trajectories | Cup-Coaster PG / PoG | Stitch-Card PG / PoG | Pour-Water PG / PoG | Total PG / PoG (this site’s sum) |
|---|---|---|---|---|
| 1 (single demonstration) | 6/13 / 6/13 | 4/13 / 5/13 | 4/13 / 1/13 | 14/39 / 12/39 |
| 15 (augmented from 1) | 7/13 / 11/13 | 10/13 / 8/13 | 12/13 / 9/13 | 29/39 / 28/39 |
| 50 (further augmented) | 13/13 / 12/13 | 12/13 / 12/13 | 13/13 / 9/13 | 38/39 / 33/39 |
With a single trajectory, the authors describe the policy as overfitting: at the start of every execution it first returns to the initial pose seen in training, which is inefficient and makes collisions more likely (the paper’s Fig. 3). This is the coverage problem from Physical AI Lab 1.6: Imitation Learning: once the start falls outside what the demonstrations cover, the policy can only pull toward states it has seen. At 15 trajectories the coverage is still sparse, and starts far from the augmented distribution lead to failed executions. The text says that at 50 trajectories “the success rate approaches 100%”; that holds for position generalization (38/39, about 97.4%), while pose generalization totals 33/39 (about 84.6%), and Pour-Water’s PoG is 9/13 at both 15 and 50 trajectories, with no gain. One pattern in the table goes unexplained in the paper: Cup-Coaster at 15 trajectories scores 7/13 on PG, lower than the supposedly harder PoG at 11/13.
Visual domain generalization covers background and viewpoint: the human demonstrations were recorded against backgrounds different from the robot workspace, and four kinds of background perturbation were added at deployment; three camera viewpoints different from the one used in data collection were also tested. Each condition and each type of generalization got 5 trials with randomized poses, across all six tasks:
| Task | PG (7 conditions combined) | PoG (7 conditions combined) |
|---|---|---|
| Cup-Coaster | 35/35 | 34/35 |
| Stitch-Card | 34/35 | 32/35 |
| Pour-Water | 35/35 | 33/35 |
| Pull-Drawer | 34/35 | 33/35 |
| Unplug-Charger | 33/35 | 34/35 |
| Pull-Tissue | 33/35 | 34/35 |
Failures occurred mainly in two situations: at a steep top-down camera angle, small objects such as the Stitch toy were hard to localize; and when an object’s color closely resembled the background, pose tracking failed. The text read here does not say how many augmented trajectories were used in this experiment. Physical AI Lab 1.8: Evaluation practised looking at the denominator first: here each cell is 5 trials and each task has 35 per type, so 34/35 and 32/35 differ by two trials, not enough to rank the tasks by difficulty. The paper lists “cross-embodiment transfer” as one aspect of visual domain generalization, but the experiments use only one robot (a UR5 with a two-finger gripper); cross-embodiment here means from the human hand to this one arm, and there is no separate cross-embodiment results table.
Limitations
- This is an arXiv preprint; all numbers are author-reported [R], and this site has not reproduced them. The authors are from Tsinghua Shenzhen International Graduate School, Zerith Robotics and the University of Science and Technology of China, and the work was supported in part by Zerith Robotics. The text read here contains no link to code or a project page.
- The authors state that the method relies heavily on accurate 6D pose estimation, which excludes objects with complex geometry or deformable structure that are hard to track reliably.
- The introduction uses the abundance of human videos on the internet as motivation, but the object branch needs a close-up image of the object and RGB-D input, and the experiments use only self-recorded demonstrations; the paper does not test on internet video.
- The RLBench results use ground-truth poses and include neither human-video extraction nor pose-estimation error; SPOT’s numbers are taken from its original paper rather than rerun in the same pipeline. The text’s “first in 8 of 12” and “13.4% improvement” do not match Table I; go by the table.
- The real-robot evaluation is small: 13 trials per cell for spatial generalization and 5 per cell for visual domain, with no confidence intervals reported. Augmentation changes only the start and keeps the original trajectory’s shape and endpoint; cases with a different endpoint or grasp were not tested.
- The hand-task mapping targets only a two-finger gripper, using three thumb and index joints; the paper does not discuss multi-fingered hands. Nor does it discuss safety on the real robot, and these results do not constitute a hardware safety validation.
Links
- Trajectory-based Diffusion from One-shot Human Video for Generalizable Manipulation (arXiv 2610.21792)
- Full paper (HTML)
- Frontier Radar | DexStitch: label the transition as a skill too, then cut, convert and stitch dexterous-hand skills from human videos
- Physical AI Lab 1.1: Coordinate Frames and Kinematics — Working Out by Hand How a Two-Link Arm Reaches the Cup
- Physical AI Lab 1.5: Where Does Data Come From? Working Out Demonstration Time from SO-101 Metadata, Then Separating the Units and Action Spaces of Three Sources
- Physical AI Lab 1.6: Imitation Learning — Working Out How an Error Rolls Out of the Demonstrations’ Coverage, Then Putting the ACT, Diffusion Policy and Data Scaling Numbers Back in Their Own Conditions
- Physical AI Lab 1.8: Evaluation — Splitting a “Net Gain of 2” into a Paired Table, Then Checking the Failure Labels and Random Seeds
- Physical AI Lab: a physical AI course where every number can be checked (course outline)
