When several skills are chained, the hard part is often not any single skill but the transition between them: the state one skill leaves behind may not be a starting point the next skill saw during training. The preprint DexStitch (arXiv 2610.17789, 2026-10-13) defines the “transition” as an ordinary skill label. Users annotate part of the long-horizon human videos (about 240 in all) frame by frame, and a classifier segments the rest into skills and transitions; per-finger contact estimates are then used to replace the human hand with a dexterous robot hand, yielding robot observations and actions that carry skill labels. Transitions are supervised entirely by human video; teleoperation contributes only about 32 single-skill demonstrations. The authors report 80.0% single-skill success and 77.5% within four-skill sequences across four household skills, and 92.4% frame-level agreement between the classifier and manual annotations. When reading the paper, it pays to establish first what the 77.5% counts, and that it was measured on only one fixed order. All numbers below are author-reported [R]; this site has not reproduced them.
The problem: who supervises the transitions between skills
The authors treat long-horizon manipulation as “primitive skills plus the transitions that connect them.” For skills to run in sequence, both need supervision, and collecting it on a robot through teleoperation and manual annotation is expensive; human videos are easy to record but come with no robot actions and no skill labels. The paper sorts existing work into two lines. One learns a shared latent skill space across humans and robots (XSkill, UniSkill): skill structure passes through a continuous latent, and the policy is not told which skill comes next. The other retargets human motion and replaces the human with a rendered robot (Phantom, Masquerade, H2R, EgoEngine), but each demonstration is treated as one trajectory, with no skill or transition boundaries marked.
DexStitch takes the opposite route: the user fixes a skill set in advance, with the transition as one of its labels, annotates part of the videos frame by frame, and trains a classifier to segment the rest. Because every ordered pair of skills needs its own transition, the authors record long-horizon human videos that contain many transitions, rather than collecting transitions one at a time.
From human video to robot data: contact-guided retargeting and four-layer compositing
Conversion has three stages and runs frame by frame, so each converted frame keeps the skill label of its source frame:
- Hand pose and contact: HaWoR fits the MANO hand model in camera coordinates, giving hand keypoints and the wrist pose; HaCo predicts contact probabilities over the MANO vertices, which are pooled by finger and thresholded with hysteresis into temporally stable per-finger contact states, and the contact points of each contacting finger are kept.
- Contact-aware retargeting: DexPilot first matches wrist-to-fingertip vectors for a kinematic initialization. Because human and robot hands differ in shape, matching fingertips alone may miss the points the human actually touched, so a second stage uses L-BFGS-B to refine the joint angles, minimizing the distance from each human contact point to the nearest point on the robot fingertip surface (only surface points whose normal is aligned with the human contact normal), with a quadratic regularizer keeping the solution close to the first stage. For the arm, the camera-to-base transform is calibrated once with EasyHeC and inverse kinematics is solved; per-frame results are smoothed with a median filter and a Savitzky–Golay filter.
- Inpainting and compositing: SAM2 tracks two sets of masks, the human hand and arm and the manipulated objects; ProPainter inpaints the human region while protecting the object mask; the robot is then rendered at the retargeted configuration and the calibrated pose.
A direct overlay would draw fingers that should curl behind the object in front of it. The authors’ fix rests on an assumption: when the hand wraps an object, the thumb lies in front and the other fingers curl behind, forming an antipodal grasp. The 3D contact points of the non-thumb contacting fingers are therefore projected into the image and dilated into a connected region; wherever it overlaps the object mask, the object is drawn in front of the robot. Each frame is composited in four layers, from back to front:
- The inpainted background
- The robot fingers hidden by the object
- The object copied from the source frame (not from the inpainted plate, because inpainting alters exactly the region where hand met object)
- The remaining robot pixels
The paper states that the pipeline uses RGB video only, with no object mesh, depth or physics simulator. Conversion quality is compared only qualitatively (Fig. 4: holding a sponge, lifting a snack carton, hanging a mug on a rack). As the authors describe it, Masquerade’s parallel-jaw gripper cannot express the dexterous grasps and is drawn over the object; without contact-guided occlusion handling, the hand is drawn in front of the object and the grasp reads as hovering. The paper gives no quantitative metric for conversion quality.
The skill classifier and a policy conditioned on the next skill
Classifier: a frozen V-JEPA 2 video encoder with a small MLP head, only 296K trainable parameters. The encoder samples at 4 FPS and emits one token per pair of frames, so the classifier reads the video at 2 Hz; it looks at a causal window of the eight most recent tokens (four seconds of context) and predicts the skill at the latest one. The authors say that freezing the encoder is what makes it possible to fit the classifier from the handful of annotated sessions available. Frame-wise predictions are then temporally filtered and merged into an ordered sequence of skill segments.
Policy: one shared Diffusion Policy conditioned on the current observation, the current skill and the next skill. It outputs an action chunk and, alongside it, predicts the skill labels over that span. The current skill determines what to do; the next skill tells the policy how the behavior should end and connect to what follows. At inference, the current skill is considered complete only when the policy consistently predicts the next skill over consecutive time steps, at which point the conditions advance by one; aggregating over several steps is meant to keep isolated predictions from triggering premature switches. The skill sequence can come from a script, or be extracted by the same classifier from a human prompt video.
Implementation details: the observation is a 224×224 RGB image plus a 33-D proprioceptive-and-skill state (3-D wrist position and 6-D rotation, 12 hand joint angles, and 6 + 6 one-hot vectors for the current and next skills). At execution, the 6-DoF arm command comes from the predicted wrist pose via inverse kinematics and the 12 hand joints are commanded directly, 18 DoF in total. The observation history is 2 steps and the prediction horizon 32, of which the first 16 actions are executed, running at 20 Hz with 10 DDIM denoising steps. The policy is pretrained for 120k steps on about 240 converted human demonstrations, then finetuned for 70k steps with about 32 teleoperated demonstrations mixed in (about 8 per skill, each performing a single skill), on a single NVIDIA A100. The hardware is an RB5-850E arm with an xHand dexterous hand and an Intel RealSense D435i camera, of which only RGB is used.
Results: what the 77.5% counts
The four skills are: pick-and-place of a green box, pick-and-place of a red box, a cup task, and stacking bowls. The baseline, UniSkill, uses the same converted data, the same 120k-step pretraining and 70k-step finetuning, and differs only in the skill representation.
| Method | Single skill: green / red / cup / bowls | Single-skill average | In sequence: green / red / cup / bowls | Sequence total |
|---|---|---|---|---|
| UniSkill | 80 / 20 / 20 / 60 | 45.0 | 0 / 0 / 0 / 0 | 0.0 |
| DexStitch | 100 / 100 / 40 / 80 | 80.0 | 80 / 70 / 80 / 80 | 77.5 |
The denominators are small. Each skill runs 5 times in the single-skill setting, so 100% is 5/5 and 40% is 2/5; the 80.0% average is 16 of 20. The sequence setting is 10 four-skill sequences, i.e. 40 skill executions, 10 per skill; 77.5% corresponds to 31 of 40 (this site’s conversion). In other words, 77.5% is the success rate of each skill execution within a sequence, not the share of sequences in which all four skills were completed. The paper does not say how many of the 10 sequences succeeded at all four steps, nor whether later skills were still executed, or reset, after an earlier one failed. The cup task is also named inconsistently: the task list says “handing over a cup,” while the caption of Table I says “hanging a cup.”
The authors attribute the gap to UniSkill to the skill representation, in particular whether transitions are modeled, on the grounds that both methods share data and training budget; this attribution is the authors’ interpretation and is not tested separately.
Classifier: of the 240 human demonstrations, 216 train the classifier and 24 (18,792 frames) are held out for evaluation. The comparisons are two vision-language models prompted zero-shot, answering the same closed-set question at the same 2 Hz decision times and from the same causal window:
| Method | Accuracy | Macro-F1 | Edit | F1@50 |
|---|---|---|---|---|
| Qwen2.5-VL-7B (zero-shot) | 65.8 | 66.0 | 70.3 | 41.0 |
| InternVL3-8B (zero-shot) | 57.8 | 56.7 | 47.5 | 28.5 |
| V-JEPA 2 + MLP (this paper) | 92.4 | 92.7 | 99.6 | 99.0 |
The authors report that their classifier recovers 242 segments against 241 in the ground truth and matches the skill order in every episode but one; the gap concentrates on transitions, with transition F1 of 0.89 for this paper, 0.41 for Qwen and 0.18 for InternVL. Note that this compares a classifier trained on data from the same setup with zero-shot models; it shows that segmentation is usable in this setting, not a general comparison of the two kinds of model.
Ablations (four-skill sequence setting, %):
| Variant | Success rate |
|---|---|
| Without human data | 35.0 |
| Without teleoperation data | 5.0 |
| Without the transition skill | 10.0 |
| Without next-skill conditioning | 7.5 |
| Teleoperation at 50% / 75% / 100% of finetuning | 10.0 / 20.0 / 50.0 |
| Full method (90% teleoperation) | 77.5 |
The authors’ reading: neither data source suffices alone; without the transition skill or next-skill conditioning, failures appear mostly at transitions, because the same transition has to lead to different states depending on the skill that follows; keeping a small share of synthetic data in finetuning (90% teleoperation) beats teleoperated data alone (50.0%). The paper says “every variant uses the same 90% teleoperation finetuning ratio,” but the ratio variants and “without teleoperation data” plainly do not; the table is the reference. If the ablations also use 40 skill executions as the denominator, 5.0% is 2 executions and 7.5% is 3 (this site’s inference; the paper does not state the ablation denominator). Only four ratios were tried, 50%, 75%, 90% and 100%, so “90% is the peak” holds only among those four.
How this relates to existing pages on this site
Physical AI Lab 1.5: Where Does Data Come From? covered the difficulty with human video: a human hand is not a robot hand, and the video has no precise joint angles and no force. DexStitch supplies joint angles through retargeting and makes the frames look like a robot through inpainting and compositing; what it produces is still a kinematic trajectory with no force or tactile information, and the gap in real execution is left to finetuning on about 32 teleoperated demonstrations.
Physical AI Lab 1.6: Imitation Learning showed how errors carry a robot out of the demonstrations’ coverage and put Diffusion Policy’s reported numbers back in their experimental conditions. DexStitch’s policy is a Diffusion Policy; it looks at the same problem at the skill boundary: the state one skill leaves behind may not be among the next skill’s training starting points, so it makes the transition itself a labeled, supervised skill.
Physical AI Lab 1.8: Evaluation practiced asking first about the denominator and the conditions. Here 100% is 5 of 5, and the sequence result is counted in skill executions rather than whole sequences; both points belong in any retelling.
Frontier Radar | Cracking Mobile Manipulation also looks at performance once skills are chained, and reports success over whole long-horizon tasks; DexStitch’s 77.5% is the success rate per skill execution within a sequence. The two papers differ in robot, tasks and metric, so their numbers cannot be compared.
Limitations
- This is an arXiv preprint; all numbers are author-reported [R], and this site has reproduced none of them. The text this site read contains no code or project-page link, and none is linked here.
- The authors state two limitations themselves: compositing assumes a wrap grasp (thumb in front, other fingers behind), so pinch and top-down grasps fall outside that rule; and composition is evaluated on a single fixed four-skill order, with orderings held out from training not yet tested, so the compositional claim has not been tested directly.
- The real-robot evaluation is small: 5 trials per skill in the single-skill setting and 10 sequences. The paper reports no confidence intervals or significance tests, and not the number of sequences completed in full.
- Conversion quality is shown only qualitatively, with no quantitative metric; the classifier is compared with zero-shot vision-language models, whereas this paper’s classifier was trained on 216 videos from the same setup.
- All experiments use one hardware setup (RB5-850E with xHand, one D435i) and four household skills; the text read does not detail how the human videos correspond to the robot scene, how many people recorded them, or how scenes varied.
- Retargeting aligns contact points kinematically only and includes no force; the safety of real-robot execution is outside the paper’s scope, and these results do not constitute hardware safety validation.
Links
- DexStitch: Learning and Composing Dexterous Skills from Human Videos via Contact-Aware Robotic Inpainting (arXiv 2610.17789)
- Full paper in HTML
- Physical AI Lab 1.5: Where Does Data Come From? Working Out Demonstration Time from SO-101 Metadata, Then Separating the Units and Action Spaces of Three Sources
- Physical AI Lab 1.6: Imitation Learning — Working Out How an Error Rolls Out of the Demonstrations’ Coverage, Then Putting the ACT, Diffusion Policy and Data Scaling Numbers Back in Their Own Conditions
- Physical AI Lab 1.8: Evaluation — Splitting a “Net Gain of 2” into a Paired Table, Then Checking the Failure Labels and Random Seeds
- Frontier Radar | Cracking Mobile Manipulation: 80% on subtasks, 19% once chained into mobile manipulation; attribute each failed episode first, then intervene stage by stage
- Physical AI Lab: a physical AI course where every number can be checked (course outline)
