
Training a dexterous robot hand from human demonstrations requires knowing where the hand is in real space: in metres, and in a world frame that does not move with the wearer’s head. Monocular video leaves depth and absolute scale ambiguous, while optical motion capture and data gloves depend on a laboratory or on hardware worn on the hands. arXiv 2611.21143 (2026-11-16, City University of Hong Kong and CyberOrigin) presents WinDEX: everyday manipulation is recorded with a head-worn fisheye stereo camera and an IMU, geometric optimization produces metric 3D hand annotations, an end-to-end network is trained on them, and two distilled models serve reinforcement learning (offline) and teleoperation (real time). The authors report a world-frame joint error (W-MPJPE) of 37.29 mm against 97.65 mm for fine-tuned HaWoR; 43.61 FPS for the offline model and 171 FPS for the streaming model; and a 61% mean simulated grasp–lift–place success rate for reinforcement learning policies trained with WinDEX trajectories as references, against 8% and 17% with WiLoR and HaWoR trajectories. All numbers below are reported by the authors [R]. This site did not run the code or reproduce any result, and the paper text read here stops partway through Section VII-C.
Why metric scale and a world frame
The paper writes the state of one hand in two parts: MANO pose parameters θ, and the wrist translation w in metres in the reference camera frame; within a recording each hand shares one shape vector β. Camera poses from visual–inertial SLAM then move the hand from the camera frame into a gravity-aligned world frame. A head-worn camera moves by itself, so a hand trajectory left in the camera frame mixes in head motion. For how conversions between coordinate frames work, see Physical AI Lab 1.1.
The capture device, CyberCap, is a head-worn fisheye stereo RGB camera: a 60.7 mm baseline, hardware synchronization, 1920 × 1200 pixels per eye at 30 fps, and a rigidly mounted IMU recording at 300 Hz on the same clock. Nothing is worn on the hands, and no external cameras or markers are needed. The known baseline lets stereo triangulation constrain absolute depth, which is exactly what monocular methods lack. The paper calls CyberCap low-cost; the text read here gives no price.
Physical AI Lab 1.5 explains that human video is plentiful, but a human hand is not a robot hand, so it has to be annotated and aligned first. WinDEX addresses the geometric accuracy of that annotation step. Table I of the paper compares how several egocentric datasets obtain scale and how they are validated; in the “accuracy validated” column, Ego4D, EgoDex and EgoScale (wild subset) are all marked “not reported”. This is the WinDEX authors’ summary. Lesson 1.5 cites EgoScale’s data scale; this site has not gone back to the EgoScale paper to check this entry.
Annotation: human-reviewed 2D keypoints, geometric 3D fitting, a 15-pixel filter
- Fisheye periphery: hands often appear near the edge of the fisheye image. For each frame the authors render 5 virtual pinhole views (centred on the image centre and the four quadrant centres), detect keypoints with YOLO11-pose and ViTPose and fuse them joint by joint, discarding candidates with confidence below 0.3. Human review then selects the reliable 2D keypoints.
- Initialization and fitting: WiLoR supplies initial pose and shape, and stereo triangulation supplies wrist depth. Stage one fits frame by frame, with the shape taken as the average of independent fits on the 64 highest-confidence frames. Stage two is temporal optimization, adjusting the strength of the temporal constraint across three modes: smooth motion, occlusion and rapid motion. During occlusion the last high-confidence pose is held; the authors state that this avoids unstable fits but does not recover unobserved finger articulation.
- Filtering: the fitted 3D hand is projected back into the image and compared with the manually accepted 2D keypoints; a frame is kept only if the mean error is at most 15 pixels.
- World frame: cuVSLAM combines stereo and inertial data to estimate the camera trajectory. Against optical motion capture, the camera trajectory has an RMS absolute trajectory error of 0.150 m and an RMS orientation error of 11.69°.
Against optical motion-capture ground truth, the retained annotations have mean errors across both hands of 9.6 mm for wrist position, 10.1 mm MPJPE and 6.1 mm PA-MPJPE. Pooling the in-the-wild annotations (both those from optimization alone and those from model-initialized optimization), the mean reprojection error against the image observations is 7.9 pixels.
The data–model flywheel starts from 20 hours annotated by optimization alone. WinDEX-Net provides initial estimates for unlabelled recordings, reviewers screen independently detected 2D keypoints, the pipeline above refines the estimates and applies the same 15-pixel threshold, and accepted samples enter the next training round. In their related-work section the authors note that such schemes risk propagating prediction errors across rounds. Their constraints are threefold: the threshold is fixed across rounds; the pose prior is trained outside this dataset; and an optical motion-capture set that is never trained on audits each round, with iteration stopping once the audit error stops improving. The text read here does not give the number of rounds.
The resulting WinDEX-Hand training corpus holds 1,000 hours of in-the-wild pseudo-ground-truth annotations from more than 200 operators, more than 1,000 scenes and more than 5,000 distinct short-range tasks, organized into 13 scene categories using O*NET occupational and work-activity information. Evaluation uses two sets excluded from training: a separately collected, human-annotated 20-hour in-the-wild set with 2D labels only and no 3D ground truth; and a 20-hour optical motion-capture set of 10 volunteers performing 38 task types, recorded in 6 everyday settings recreated inside the capture volume. In the training corpus the median wrist depth is 318 mm, with the central 80% of hand-frames between 215 and 455 mm; half of hand-frames are more than 25° off the optical axis, and one in ten exceeds 43°.
One teacher, two students: trading accuracy for speed
The teacher, WinDEX-Net, works directly on the native fisheye stereo images without an external hand detector. It combines DINO appearance features with intermediate features from the depth branch of VGGT-Omega, conditioned on per-patch viewing rays and the stereo calibration; a local crop branch re-crops the hand region from the previous estimate; and a temporal encoder conditioned on the SLAM camera trajectory helps separate head motion from hand motion. The paper illustrates the crop with one example: in a 512 × 288 full-frame view a hand spans only 12 patch tokens, and resampling it to a 224 × 224 crop gives 196.
Two students are distilled for deployment. WinDEX-Window has 116M parameters, processes 32 frames at a time and uses both past and future frames, for offline reconstruction. WinDEX-Stream has 8.92M parameters, uses MobileNetV3-Large encoders and a two-layer causal GRU, sees only the current stereo pair and its retained past state, and outputs 21 metric 3D joints per hand directly rather than MANO parameters, for live control. Table III of the paper compares speed and accuracy on three motion-capture recordings (RTX 4090, FP16, batch size 1, offline models with 32-frame windows at stride 16; errors in mm):
| Model, crop iterations | FPS | Wrist error | MPJPE | PA-MPJPE |
|---|---|---|---|---|
| WinDEX-Net, 2 | 9.50 | 25.9 | 17.8 | 7.8 |
| WinDEX-Net, 3 | 8.17 | 20.4 | 15.5 | 7.8 |
| WinDEX-Net, 4 | 7.38 | 18.9 | 15.4 | 7.8 |
| WinDEX-Window, 2 | 43.61 | 27.1 | 20.1 | 8.1 |
| WinDEX-Window, 3 | 35.94 | 23.8 | 18.7 | 8.1 |
| WinDEX-Window, 4 | 31.77 | 20.3 | 18.7 | 8.1 |
| WinDEX-Stream | 171.00 | 42.7 | 33.2 | 12.5 |
The 43.61 FPS in the abstract is the offline throughput of WinDEX-Window with 2 crop iterations; 171 FPS is the causal per-frame throughput of WinDEX-Stream, which has the largest error on all three metrics in this table. More iterations lower the wrist error and also lower throughput. For the later reconstruction comparisons the authors fix WinDEX-Window at 2 iterations.
Comparison with prior methods: camera frame, world frame and 2D detection
Camera frame (Table IV): each of nine baselines is reported with pretrained (P) and fine-tuned (FT) weights; fine-tuning uses only the WinDEX-Hand training corpus, and both settings share the same input adapter and test recordings. Single-image methods use the calibrated left view, multi-view methods use stereo; UmeTrack uses 18 corresponding joints and the others 21. WinDEX-Window has a P row only: wrist error 15.70 mm, MPJPE 16.68 mm, PA-MPJPE 7.82 mm and jitter 3.15 m/s². The best fine-tuned baseline in each column is UmeTrack for wrist error at 27.45 mm, and HaWoR for MPJPE at 23.48 mm, PA-MPJPE at 9.45 mm and jitter at 5.50 m/s². On this basis the authors claim state-of-the-art camera-frame results on this benchmark.
World frame (Table V): the comparison is between HaWoR’s complete hand-reconstruction and camera-trajectory pipeline and WinDEX-Window composed with cuVSLAM camera poses. Position errors are in mm and RTE is a percentage:
| Method | W-MPJPE | WA-MPJPE | RTE |
|---|---|---|---|
| HaWoR (P) | 100.92 | 44.19 | 3.10 |
| HaWoR (FT) | 97.65 | 40.91 | 2.83 |
| WinDEX-Window + cuVSLAM | 37.29 | 17.69 | 2.00 |
The authors report reductions of 61.8%, 56.8% and 29.3% relative to fine-tuned HaWoR. Reading the 37.29 mm requires the metric definitions. W-MPJPE splits a recording into 100-frame segments and fits a similarity transform to the first two frames of each segment before measuring error; WA-MPJPE fits it to all frames of the segment; RTE is the root-trajectory error after rigid alignment without scaling, divided by the total ground-truth path length. At CyberCap’s 30 fps, a segment is about 3.3 seconds (this site’s arithmetic). So 37.29 mm measures error after per-segment alignment, not drift accumulated over a whole recording; for the whole camera trajectory, see the 0.150 m in the previous section.
2D: on the held-out human-annotated in-the-wild set, WinDEX-Window’s reprojection error is 14.5 pixels; the baseline values in Figure 10 were not in the text read here. Table VI compares hand bounding-box detection. WinDEX-Net reaches an F1 of 80.88% in the wild and 87.53% on motion capture; Sapiens2-1B reaches 79.69% and 81.39%, and its in-the-wild recall of 77.43% is slightly higher than WinDEX-Net’s 77.09%.
Two numbers should be cited with their table. In Table III, WinDEX-Window with 2 iterations has a wrist error of 27.1 mm; in Table IV the same model has 15.70 mm. Table III states that it uses three motion-capture recordings, and the text read here does not say how Table IV’s test recordings relate to them. In addition, in Table II (the ablation of which depth-branch layer WinDEX-Net taps), the layer 11 (production) row has the values 15.70, 16.68 and 7.82, identical to the WinDEX-Window row of Table IV; the text read here does not explain this. When quoting 15.70 mm, name Table IV as its source.
Downstream: Hand2Policy and Hand2Teleop
Hand2Policy uses WinDEX-Window to reconstruct recorded demonstration trajectories and retargets each clip once, producing a reference specific to one robot that a tracking policy can reuse for replay, reset or training; rewards are based on pose tracking. The simulation uses the trajectory imitator of ManipTrans (reference [7] in the paper) with Inspire and Wuji dexterous hands. The authors report that, under matched training settings and budgets, the mean simulated grasp–lift–place success rate is 61%, against 8% and 17% when training with WiLoR and HaWoR motion references. The 7.6× and 3.6× in the abstract are these two ratios (61/8 ≈ 7.6 and 61/17 ≈ 3.6, this site’s arithmetic). The authors also state that the policies transferred to physical robots; the task count, number of trials and real-robot results are in Section VII-D, which this site did not read.
Hand2Teleop runs WinDEX-Stream and real-time VIO on a local server, and the operator is tracked only by CyberCap’s stereo cameras and IMU. Arm targets come from the global wrist displacement relative to a session anchor, solved with damped least-squares inverse kinematics; hand targets come from wrist-relative finger articulation through analytical retargeting of the dex-retargeting family, with the vendor implementation for the live Wuji hand. Filtering, rejection of transient jumps and holding the last accepted target reduce the effect of unreliable observations on commands. The physical platform is a 20-joint Wuji five-finger hand on a seven-joint robot arm. 171 FPS is model throughput; end-to-end teleoperation latency is in Section VII-E, which this site did not read.
Limitations
- This is an arXiv preprint. All numbers are reported by the authors [R] and have not been independently reproduced. The paper text read here stops partway through Section VII-C: the baseline values in Figure 10, Section VII-D (Hand2Policy), Section VII-E (Hand2Teleop) and Section VIII (limitations) were not read, so the reinforcement learning task count and number of trials, the real-robot transfer results and the teleoperation latency are unknown here.
- All evaluation data were collected by the authors with CyberCap. 3D ground truth comes only from 6 settings recreated inside the motion-capture volume; the in-the-wild evaluation has 2D labels only and no 3D ground truth. Baselines were fine-tuned only on WinDEX’s own pseudo-ground-truth training corpus, and single-image baselines received only the left view. The text read here contains no comparison on a third-party public benchmark.
- None of the tables read gives a standard deviation, confidence interval or multi-seed result.
- W-MPJPE and WA-MPJPE are computed after aligning each 100-frame segment and do not measure drift accumulated over a whole recording; the camera trajectory itself has RMS errors of 0.150 m and 11.69° against motion capture.
- During occlusion the annotation pipeline holds the last high-confidence pose, so finger articulation while occluded is not recovered.
- Table III and Table IV give different wrist errors for the same WinDEX-Window configuration (27.1 mm and 15.70 mm), and one row of Table II is identical to a row of Table IV; the text read here does not explain either.
- The paper calls CyberCap low-cost, but the text read here gives no price. One author is affiliated with CyberOrigin, and the text read here does not state how CyberCap relates to that company. The paper lists a project website, which this site did not open; whether code or data are released, and under what license, is unverified.
Links
- WinDEX: Metric 3D Hand Motion Reconstruction in the Wild for Dexterous Teleoperation and Policy Learning (arXiv 2611.21143)
- Full paper in HTML
- Physical AI Lab 1.5: Where Does Data Come From? Working Out Demonstration Time from SO-101 Metadata, Then Separating the Units and Action Spaces of Three Sources
- Physical AI Lab 1.1: Coordinate Frames and Kinematics — Working Out by Hand How a Two-Link Arm Reaches the Cup
- Physical AI Lab: a physical AI course where every number can be checked



