Two common ways of collecting demonstrations for a multi-fingered dexterous hand each have a drawback. Teleoperation records actions the robot can execute, but it needs a complete robot setup and the operator has to coordinate many finger joints. Human-hand demonstrations are easy to collect, but the motion has to be estimated and retargeted to the robot hand, and the data contain no robot actions. The preprint DexPort (arXiv 2610.32969, 2026-10-23) takes a third route: the same 20-DoF dexterous hand used at deployment and a fisheye camera form a detachable module that mounts on a hand-held handle for collection and on the robot wrist for deployment. The operator controls the fingers through a small set of skills, and the data record both the 20-dimensional commanded targets sent to the hand and the measured joint states. The authors report that, on the same data, using the commanded targets rather than the measured joint states as action labels raises average final-task success across the three tasks scored by success rate from 11% to 64%, and that co-training with 1,000 in-the-wild demonstrations raises average pick-and-place success across three evaluation conditions from 36% to 78%. This article focuses on what the “command labels versus state labels” comparison does and does not show, and on how to read the in-the-wild and collection-throughput numbers. All numbers below are author-reported [R]; this site has not reproduced them.
The problem: why dexterous-hand demonstrations are hard to collect
For parallel-jaw grippers, UMI already collects demonstrations naturally and at scale with a hand-held gripper in many environments. The authors argue that extending this approach to multi-fingered hands is not straightforward. DexUMI, DEXOP and similar systems collect contact-rich behaviour with wearable hand exoskeletons, but more degrees of freedom mean more mechanical complexity and a heavier burden on the operator; and they record exoskeleton configurations, not the commands received and states produced by the robot hand used at deployment. RealDexUMI records executable commands with a physical robot hand, but demonstrates the approach on a hand with only six active DoFs.
From this the authors derive three requirements for a portable collection interface: use the same end effector for collection and deployment (Same Emb.), so hand motion needs no retargeting; record the commands sent to the hand (Action Data) rather than only inferring actions from later measured states, because the commanded target and the measured joint state may differ during contact; and expose enough controllable degrees of freedom (DoF), since otherwise a simpler gripper would do. Table I of the paper compares existing systems on these three criteria and on weight; a few rows are excerpted below (the authors’ compilation):
| System | Controllable DoF | Weight | Same embodiment | Records commands |
|---|---|---|---|---|
| UMI | 1 | 780 g | ✓ | ✗ |
| DexUMI | 12/6 | 850 g | ✓ | ✗ |
| DEX-Mouse | 6 | > 1000 g | ✓ | ✓ |
| RealDexUMI | 6 | 680 g | ✓ | ✓ |
| DexPort | 20 | 955 g | ✓ | ✓ |
Hardware: one hand–camera module, two mountings
The module joins a five-fingered, 20-DoF Wuji robot hand and a wide-angle fisheye camera (the appendix specifies a USB board camera with a 200° fisheye lens) through a rigid 3D-printed flange. For collection it attaches to a handle built around a Meta Quest 2 controller; for deployment, the same physical module mounts to the robot wrist, so the hand embodiment and the camera-to-hand extrinsics stay the same in both settings. The authors designed flanges for both the Wuji 1 and the Wuji 2 hands.
- What is recorded. The Quest 2 controller provides the wrist’s 6-DoF trajectory, the fisheye camera captures the hand–object interaction and the surrounding scene, and the hand controller records both the 20-dimensional commanded joint targets and the resulting joint states. A single recording loop runs at 30 Hz, with the camera, hand and controller each read in a background thread. Because collection and deployment use the same 30 Hz loop, the authors expect latencies to roughly match, so unlike UMI they do not measure stream latencies or align the streams in post-processing.
- Weight. The complete hand-held assembly weighs 0.955 kg, of which the non-hand hardware contributes 0.220 kg. A strap across the back of the operator’s hand carries part of the load. The initial design placed the controller behind the hand, offsetting the grip from the assembly’s centre of mass and creating a long moment arm; the final design places the controller above the hand so the grip lies near the centre of mass, reducing the wrist torque the operator must counteract.
- Deployment calibration. The collected end-effector actions are expressed in the Quest controller frame and must be converted to the robot end-effector frame at deployment. The authors bolt the module to the robot flange, jog the arm through about 30 poses that tilt the flange axis by at least 30°, pause 1.5 s at each before recording the paired flange and controller poses, initialise with a closed-form hand–eye solution and refine by nonlinear least squares to obtain the fixed controller-to-flange transform, once for each arm. The 20-dimensional hand targets need no conversion.
- Deployment platform. Policies are deployed on Tianji Marvin, a bimanual robot with two 7-DoF arms. Pick-and-place and screwing use the Wuji 1; drawer stowing and bimanual scanning use the Wuji 2. On the Wuji 1, hand targets pass through a 5 Hz low-pass filter and are interpolated by the hand firmware at 1 kHz, with a per-joint torque limit of 0.5 N·m.
Skills help only the operator; the policy learns the raw commands
The Quest 2 controller has only two buttons, two continuous triggers and a two-axis joystick. To drive 20 joints with it, the authors package finger motion into a few parameterised skills:
- Thumb control: the two joystick axes control thumb flexion–extension and abduction–adduction, following HATO.
- Power grasp: one joystick axis controls coordinated closure of the four non-thumb fingers; pushing the joystick incrementally tightens or loosens the grasp, and releasing it holds the current target, so the operator does not have to keep pressing.
- Pinch grasp: the same incremental command moves only the index finger, closing it against the independently positioned thumb, for small objects and grasps that need precise fingertip contact.
- Two-finger grasp: the index and middle fingers open or close together while the other fingers keep their targets, for multi-object grasps, two-finger grasp-and-pull and other multi-stage motions.
- In-hand rotation: a button invokes a learned closed-loop policy that produces the repeated finger gaiting needed to rotate a tool in hand, while the operator keeps controlling the hand’s overall motion.
The in-hand rotation skill is trained with PPO in Isaac Lab. The screwdriver is modelled as a 12-sided prism rotating on a revolute joint; the hand base is fixed in simulation; the policy controls 12 joints, with the ring and pinky fingers fixed. Object mass, centre of mass, friction, restitution, finger PD gains, initial grasp and random disturbance forces are randomised; training uses 4096 parallel environments and 1.18 × 10⁸ environment steps in total. During collection this three-layer MLP runs at 20 Hz on the hand’s joint encoders.
The skills only assist the operator. The learned policy is a diffusion policy (a 1D convolutional U-Net denoiser with a fine-tuned DINOv2 ViT-S/14 visual encoder). It observes the two most recent time steps, each consisting of a 224 × 224 RGB image, the end-effector position relative to the start of the current chunk, and the measured 20-DoF hand state. It outputs a 16-step action chunk; each step is 29-dimensional: 9 for the wrist (3 translation plus a 6D rotation) and 20 for the hand (absolute joint targets). The policy receives no skill identity, skill parameter or task-phase label; it predicts the recorded commanded joint targets directly. At deployment the policy runs asynchronously at 30 Hz: each new chunk keeps its first 8 steps, steps whose time has passed are dropped, and the command at each control tick averages all chunks that cover it. Wrist motions go through the calibrated transform and are solved for the 7-DoF arm with Levenberg–Marquardt inverse kinematics in PyRoki.
The authors state explicitly that switching the action label from the commanded target to the measured joint state is the single change that defines the “state-as-action” baseline; every other part of the training recipe is shared.
The core comparison: the same demonstrations, a different action label
The four evaluation tasks exercise different kinds of dexterity: vegetable pick-and-place (placing two vegetables on a plate in turn, choosing between pinch and power grasps by shape and size); screwing (rotating a screwdriver in hand while keeping a stable grasp); drawer stowing (without releasing a held object, opening a drawer with the same hand, placing the object inside and closing it, with an added third-person camera because the grasped object hides much of the view); and bimanual barcode scanning (one hand grasps and presents an object while the other positions and operates a scanner). Each policy is evaluated for 15 episodes per task or evaluation condition. Drawer stowing and bimanual scanning are scored by phase, a phase counting as successful only if all preceding phases also succeed; screwing reports average hold duration in seconds and rotations per minute.
The authors report:
- Across the three tasks scored by success rate, the command-label policy reaches an average final-task success of 64% and the state-label baseline 11%, a gap of 53 percentage points (this site’s calculation). Going by the evaluation protocol, these three should be pick-and-place, drawer stowing and bimanual scanning; the text we read does not list the final success rate for each task (they are in Figure 4), nor does it say which pick-and-place condition is used.
- On bimanual scanning, the baseline grasps the first object in 80% of episodes, holds both in 13%, and never completes a scan. The authors note that because each stage requires the previous one to succeed, the deficit compounds from stage to stage instead of appearing as a constant offset.
- On screwing, the baseline holds the tool more than twice as long as the command-label policy (31.7 s vs 15.6 s), but rotates it at a third of the rate.
- The failure modes the authors describe: the baseline’s grasps are too weak, it drops the package when presenting it to the scanner, and the laundry slips out of the hand once it starts pulling the drawer open (Figure 6).
A few points to keep in mind when reading this comparison:
- It is a label ablation on the same DexPort data, not a comparison between collection interfaces. The two policies share observations, network and training recipe; only the action label differs. From this the authors infer that exoskeleton-type interfaces without action data have a major limitation; that is an extrapolation, and the paper runs no experiment on data collected with an exoskeleton.
- Why the gap shows up in contact. The paper’s reasons are that the commanded target and the measured state can differ during contact, and that with commands the policy can exert the force needed to hold the object tightly. This site’s reading: when a finger is blocked by the object, its measured joint position stops at the object’s surface while the commanded target can go further; reproducing the measured state as the target loses that difference. The text we read gives no force measurements and does not report how large the command–state difference is.
- “Holding longer” is not the same as doing better. In the screwing task the baseline is higher on hold duration and lower on rotation rate. Looking at a single metric would lead to the opposite conclusion, which is why the authors report both.
- Sample size. Each task uses 15 episodes; 80% is about 12 episodes and 13% about 2 (this site’s conversion). The text we read reports no confidence intervals, repeat counts or random seeds.
- Which command is recorded. On the Wuji 1, hand targets pass through a 5 Hz low-pass filter before the firmware interpolates them; the text we read does not say whether the recorded commanded target is taken before or after the filter.
In-the-wild data and collection throughput
In the laboratory the authors collect 1,605 demonstrations: 700 for vegetable pick-and-place, 200 for the screwdriver, 380 for drawer stowing and 325 for bimanual scanning. They also collect 1,000 pick-and-place demonstrations in unstructured environments such as an apartment and offices, for 2,605 in total (this site’s sum). The in-the-wild data use the same training objects as the laboratory data; what differs is the scene, background and lighting, not the objects themselves. Collection needs neither a robot arm nor external tracking infrastructure. For co-training, 50% of each batch is drawn from in-the-wild data, and every policy is trained for the same number of epochs over its laboratory demonstrations (about 250).
Table III of the paper gives pick-and-place full success (both objects placed on the plate, %) for two amounts of laboratory data, with 15 episodes per condition:
| Lab demos | In-the-wild demos | Seen objects | Unseen objects | Distractors |
|---|---|---|---|---|
| 500 | 0 | 0 | 20 | 0 |
| 500 | 500 | 33 | 13 | 53 |
| 500 | 1,000 | 47 | 7 | 40 |
| 700 | 0 | 47 | 20 | 40 |
| 700 | 500 | 53 | 53 | 53 |
| 700 | 1,000 | 73 | 73 | 87 |
- 36% to 78% is the average of three cells with 700 laboratory demonstrations. (47 + 20 + 40) ÷ 3 ≈ 36 and (73 + 73 + 87) ÷ 3 ≈ 78 (this site’s check). At 15 episodes per cell, the distractor column goes from about 6 to about 13 successes, and unseen objects from about 3 to about 11.
- “Every condition improves as in-the-wild data increase” holds only for the 700-demonstration group. In the 500 group, unseen objects fall from 20% to 13% and then 7%, and distractors rise from 0% to 53% before dropping back to 40%. The authors explain that the in-the-wild data add scene diversity but not object geometry, and hypothesise that transferring this robustness to new objects requires a sufficiently large laboratory dataset; this is the authors’ hypothesis, and the paper does not test it separately.
- Network size is not identical. In appendix Table VI, the “pick-and-place in-the-wild” configuration uses U-Net channels of (512, 1024, 2048) and the “pick-and-place baseline” (256, 512, 1024). The text we read does not say exactly which policies “pick-and-place baseline” refers to; if the pick-and-place policy trained only on laboratory data used the smaller network, the gain from co-training would mix in a difference in model capacity.
Collection throughput is counted as the number of trajectories collected in 30 minutes of continuous operation on the unimanual pick-and-place task (Table II). The operator is expert-level for each method:
| Method | Trajectories / 30 min | Portable | Same hand embodiment | Action data |
|---|---|---|---|---|
| DexPort | 122 | ✓ | ✓ | ✓ |
| Skill teleoperation (same skills drive the hand, VR tracking drives the end effector) | 68 | ✗ | ✓ | ✓ |
| Glove teleoperation (Wuji Glove tracks the hand, VR controller drives the end effector) | 86 | ✗ | ✓ | ✓ |
| Human hand | 275 | ✓ | ✗ | ✗ |
The resulting ratios, about 1.4× (versus glove teleoperation), about 1.8× (versus skill teleoperation) and human hands about 2.3× faster, are all ratios of this single 30-minute count. The authors attribute the slower teleoperation to the operator being outside the scene and losing direct visual and haptic cues; that is the authors’ explanation, and the paper does not test it separately. The authors treat human-hand demonstrations as an upper bound on speed rather than a competing data source, since they provide neither the deployed hand’s embodiment nor its commanded actions. The text we read does not say how many operators performed each method or how many rounds were counted, and does not compare the success rate or quality of the trajectories collected by each method.
How this relates to other content on this site
Physical AI Lab 1.5: Where Does Data Come From? covers three sources, teleoperation, human video and synthesis, and stresses that action spaces with the same number of dimensions can mean different things. DexPort sits between teleoperation and human-hand demonstrations: the operator carries the robot hand itself into the environment, so the action space is that of the deployed hand. Its label ablation shows that even for the same hand and the same 20-dimensional action, “the command sent” and “the state measured” are two different labels. The policy itself is the diffusion policy discussed in Physical AI Lab 1.6: Imitation Learning, and the in-the-wild co-training result relates to that lesson’s discussion of diversity mattering more than sheer quantity, although here only scene diversity is added. The 15-episode evaluations, stage-chained success rates, and hold duration and rotation rate pointing in opposite directions can be read with the methods of Physical AI Lab 1.8: Evaluation. The in-hand rotation skill is trained in simulation with dynamics randomisation; for the concepts, see Physical AI Lab 2.5: Sim2real with Domain Randomization and System Identification. The text we read reports no separate success rate for the skill itself.
The earlier Frontier Radar | DexStitch takes another route: it cuts dexterous-hand skills out of human video and converts them into robot data. DexPort avoids the human-to-robot conversion, at the cost of every operator carrying a real robot hand, with a hand-held assembly weighing nearly 1 kg. The two papers differ in tasks, hardware and metrics, so their numbers cannot be compared.
Limitations
- This is an arXiv preprint; all numbers are author-reported [R], and this site has not reproduced them. Each task or condition is evaluated over 15 episodes, and the text we read has no confidence intervals, repeat counts or random seeds.
- This article is based on the paper’s HTML text and table text; it did not read values from figures such as Figure 4. The final success rate of each of the four tasks and the exact screwing rotation rates are in the figures and are not quoted here.
- 64% versus 11% is an action-label ablation on the same data; the paper has no comparison with data collected by exoskeletons or other interfaces, and “a major limitation of exoskeleton-type interfaces” is the authors’ inference.
- In-the-wild data are used only for pick-and-place, with the same objects as in the laboratory; drawer stowing also uses a third-person camera. The operator is occasionally visible in the recorded frames, and the authors do not remove them.
- Limitations stated by the authors: despite the ergonomic design, the device’s weight limits uninterrupted use and rotation-heavy motions; the skills cover only part of the space of dexterous behaviours.
- The experiments run on a real robot, but the text we read contains no safety evaluation; these results are not a hardware safety validation.
- The paper gives a video page, amberxie88.github.io/dexport/; the text we read does not say whether code, hardware designs or the dataset have been released. This site did not open that page to check its contents while writing.
- “A gap of 53 percentage points”, “2,605 in total”, the episode counts such as “about 12” and “about 2”, and the checks of ratios and averages are this site’s conversions from the paper’s numbers.
Links
- DexPort: A Hand-Held Skill Interface for Dexterous Hand Data Collection (arXiv 2610.32969)
- Full paper in HTML
- Physical AI Lab 1.5: Where Does Data Come From? Working Out Demonstration Time from SO-101 Metadata, Then Separating the Units and Action Spaces of Three Sources
- Physical AI Lab 1.6: Imitation Learning — Working Out How an Error Rolls Out of the Demonstrations’ Coverage, Then Putting the ACT, Diffusion Policy and Data Scaling Numbers Back in Their Own Conditions
- Physical AI Lab 1.8: Evaluation — Splitting a “Net Gain of 2” into a Paired Table, Then Checking the Failure Labels and Random Seeds
- Physical AI Lab 2.5: Sim2real with Domain Randomization and System Identification — Putting Two Papers’ Numbers Back in Their Setups, Then Working Out Friction by Hand from One Push of a Cup
- Frontier Radar | DexStitch: label the transition as a skill too, then cut, convert and stitch dexterous-hand skills from human videos
- Physical AI Lab: a physical AI course where every number can be checked (course outline)
