Frontier Radar | Ego2X: record the contacts in first-person human video in the object’s frame, then transfer them to a robot; authors report that post-training X-VLA only on simulation-verified converted data raises success from 40.5% to 67.0%

Ego2X maps egocentric video to robot data via object-frame contacts. Authors report 62.9% contact realization (HuRo 27.9%) and X-VLA 40.5%→67.0% in sim.

Data table on a light background. The title reads "Authors report: Ego2X is higher in every row shown; X-VLA rises from 40.5% to 67.0%", and the subtitle says all values are percentages reported by the authors, the metric and comparison vary by row, the top two rows come from EgoDex and the bottom two from simulation. The table has four rows and two columns, Ego2X and Comparison: dexterous-hand contact realization against HuRo, 62.9% vs 27.9%; parallel-gripper two-pad fit against Ego2Robot, 63.2% vs 9.7%; closed-loop conversion against HuRo and Ego2Robot, 59.6% vs 1.9%; X-VLA bread-in-basket success against zero-shot, 67.0% vs 40.5%, with this row in a darker color. The source line at the bottom cites arXiv 2611.24138 Tables 4, 5, 7 and notes that HOT3D depth error and the bell task are not drawn.
Drawn from arXiv 2611.24138 Tables 4, 5 and 7 (reported by the authors). Each of the four rows has its own metric and comparison, and Ego2X is higher in every row shown. The closed-loop conversion row covers 52 matched videos of the Place bread in basket task in RoboTwin 2.0. The darker X-VLA row is the downstream result discussed in the article, evaluated on 200 paired scenes in RoboTwin 2.0 simulation; the paper reports a Holm-adjusted McNemar p = 5.0 × 10⁻⁷ against zero-shot. In the gripper row, Ego2Robot uses device-tracked hand data, a different input from Ego2X's monocular video. Rows follow the paper's order from contact to policy and are not a ranking. Not measured by this site.

Pipelines that turn first-person human video into robot data mostly carry over the motion of the hand and arm. The authors of the preprint Ego2X (arXiv 2611.24138, 2026-11-18) point out that a monocular hand estimate drifts by centimeters along the camera ray, while stable manipulation needs millimeter precision on the object surface, so retargeted fingers hover beside objects and parallel grippers close on empty space. Ego2X records every contact in the object’s frame instead, uses those contacts to drive a dexterous hand or a parallel gripper, and finally replays the result in closed loop in the RoboTwin 2.0 simulator, keeping only clips that actually complete the task. All numbers below are reported by the authors [R] and have not been reproduced by this site; the paper states that real-robot deployment is not yet complete.

This site’s lesson Physical AI Lab 1.5 explained that human video is plentiful but “a human hand is not a robot hand”, so it has to be translated into a robot hand before it is usable. EgoJEPA, covered earlier, avoids retargeting and learns latent actions directly; DexStitch turns human video into dexterous-hand data with contact-guided inpainting. Ego2X stays with retargeting. What changes is the thing being transferred, which is now the contact on the object, and a physics check in simulation decides whether a clip may enter the training set.

What CORE records for each contact

The core of Ego2X is CORE (Contact-centric Object-interaction REpresentation). Every frame and every contact gets a four-part record:

  • Position: the MANO hand region in contact, the surface anchor in the object frame and its outward normal.
  • Force prior: each region’s relative share of the load, the push direction, and a lower bound on the normal support force.
  • Grasp mode: one of seven modes, set by which palm and finger regions touch the object: palm carry, power grasp, intermediate grasp, fingertip hold, precision pinch, hook grasp and unknown. Each mode maps to the robot contact pads that must realize it.
  • Interaction state: one of Approach, Acquire, Hold, Transport and Release, used to synchronize gripper actuation, protection of object pixels in the video and simulation waypoints.

The way contact positions are found rests on one observation: monocular hand depth drifts along the viewing ray, but the 2D projection stays well localized. Ego2X first uses EgoTac to detect contact timing from the tactile appearance of the image, then casts a ray from the camera center through the pixel of each hand vertex and takes its first intersection with the object’s signed distance field, computed from a mesh reconstructed by SAM 3D. Each anchor is graded Certified, Trusted, Hypothesis or Uncertain from palmar orientation, non-penetration of the adjacent phalanx, agreement with rigid object motion and 2D occlusion order. Only the first two grades constrain the later depth correction and inverse kinematics; the other two only keep the sequence continuous in time.

The paper says plainly that without tactile sensors, monocular RGB cannot uniquely determine internal grasp forces in Newtons. The force in CORE is therefore a prior. The relative shares come from normalizing EgoTac’s predicted intensities. For the lower bound on support force, the object’s mass is estimated from mesh volume times one of three reference densities, 0.5, 1.0 and 1.5 g/mL, multiplied by the magnitude of the center-of-mass acceleration minus gravity, and then allocated by multiplying with the hand’s share and each region’s share.

From one video to one LeRobot episode

The input is one monocular egocentric video and the target robot’s URDF. The output is a verified LeRobot episode with robot actions, training masks and a video that keeps the held object. There are four steps in between:

  1. Decoupled depth correction: using the hand to set the object’s scale and then the object to correct the hand’s depth couples their errors in a loop. Ego2X splits the object and the hand into two paths. The hand only shifts along its root camera ray, within −150 to +50 mm, and the shift is chosen with a Cauchy objective over Certified and Trusted vertex–anchor pairs.
  2. Cross-morphology retargeting: a URDF compiler samples the robot’s link collision meshes into contact pads that match the grasp modes. For a dexterous hand, inverse kinematics pulls the pads onto the CORE anchors. For a parallel gripper, Ego2X searches the object’s distance field for antipodal contact pairs that fit the jaw aperture and locks one object-frame grasp per contact interval.
  3. Video compositing guided by contact state: during Acquire, Hold and Transport, a protected object zone is subtracted from the erasure mask so the held object is not inpainted away with the arm. Along the contact edge, leftover skin pixels are then stripped by Lab color before the rendered robot is composited in.
  4. Closed-loop verification in simulation: the episode is replayed in RoboTwin 2.0, and only rollouts that pass the physical checks are kept for VLA post-training.

Geometry and contact metrics, each with its own conditions

The abstract’s “in-contact hand depth error reduced by 33.7%” comes from a narrow setting on HOT3D: HaWoR hand estimates, 6 clips and the dataset’s object poses, with the error along the ray falling from 14.35 mm to 9.51 mm. With EgoHand estimates instead (10 clips), the same error falls from 51.71 mm to 44.41 mm, a 14.1% reduction. When object poses are estimated, the median in-contact object translation error falls from 104.6 mm to 67.3 mm on the 8-clip confirmation split and from 141.9 mm to 99.1 mm on the 11-clip held-out split, mean reductions of 30.4 mm and 47.3 mm.

The source of the object pose has a large effect on contact recovery. With dataset poses, CORE reaches a frame F1 of 0.924 on 31 clips. With poses estimated by FoundationPose (19 clips), the raw frame F1 is only 0.191, and 0.386 after object depth fusion.

Retargeting comparisons on EgoDex (Tables 4 and 5 of the paper):

  • Dexterous hands: 79 takes with 25,549 contact slots, where a contact counts as realized if the pad’s signed distance is within −5 to +10 mm. Ego2X realizes 62.9%, official HuRo stage-8 27.9%, paired test p = 2.6 × 10⁻¹⁴. A wrist-following ablation reaches 29.6%, close to HuRo; it shares Ego2X’s front end, wrist targets and solver and only drops the contact terms, so the gap can be attributed to the contact terms (this site’s inference from the ablation design). The estimated human MANO hand itself realizes only 35.8% over all slots. Joint speed stays within 7.5 rad/s (HuRo: 63.9 rad/s), and whole-hand penetration is 4.4% vs 4.1%, p = 0.393.
  • Parallel grippers: on 28 hand sides and 1,272 holding frames, the two-pad fit is 63.2%, against 9.7% for Ego2Robot. The inputs differ: Ego2X and HuRo use hands reconstructed from monocular RGB, while the official Ego2Robot uses device-tracked 3D hand data.
  • Five grippers: compiled from the same CORE records to Panda + 2F-85, ARX5, Franka Hand, UR5 + WSG50 and Piper, end-effector realization is 0.796–0.820. Whole-arm inverse kinematics with a fixed table-edge mount ranges from 0.032 (Piper) to 0.695 (UR5 + WSG50).

For video compositing, on 3 pixel-audited takes, erasing the arm without object protection removes 19.4%–55.0% of the visible held-object pixels; protection by CORE interval lowers that to 3.0%–11.2%. On 14 unseen EgoDex takes (663 frames), skin bleed along contact edges falls from 12.55% to 0.00%.

Simulation verification and X-VLA post-training

In RoboTwin 2.0, under the same simulator checks, Ego2X converts 31 of 52 matched Place bread in basket videos (59.6%), while HuRo and Ego2Robot convert 1 each (1.9%). On Place bell on scale, Ego2X yields 45 of 53 specs (84.9%) and both baselines 0/53. The paper calls these matched videos and does not describe their origin in that section.

The authors then post-train X-VLA on Ego2X-converted clips alone, with no simulator demonstrations, and evaluate Place bread in basket on 200 paired scenes (Table 7 of the paper; p values are Holm-adjusted exact McNemar tests against zero-shot):

Training dataClipsSuccess rateAgainst zero-shot
Zero-shot X-VLA040.5%baseline
Ego2X, two unfiltered pools55 / 7341.0% / 44.5%not significant
Ego2X, passed simulation verification (P70)7063.5%p = 7.4 × 10⁻⁶
Ego2X, passed simulation verification, layout-weighted (55 lw)5567.0%p = 5.0 × 10⁻⁷
Simulator expert demonstrations (reference)2072.0%reference

Two things can be read from this table. First, converted data from the same source stays near zero-shot when it is not verified, and rises to 63.5% and 67.0% only after the simulator check. Second, 67.0% is still below the 72.0% from 20 simulator expert demonstrations, and the paper gives no test between the two. The “26.5%” improvement in the conclusion is the difference between 67.0% and 40.5%, in percentage points.

Other results: on the first 50 scenes (44.0% zero-shot), verified Ego2X reaches 70.0%, while training on the single verified clip each from HuRo and Ego2Robot gives 2.0% and 4.0%. In 3-seed co-training (20 simulator demonstrations plus 24 unfiltered clips), adding HuRo data lowers the simulator-only success rate by 7.5 percentage points, and Ego2X beats HuRo by 5.5 percentage points, 95% interval [+0.7, +10.3]. On Place bell on scale (200 scenes, 17.0% zero-shot), 40 unfiltered clips give 27.0%, and 40 zero-retry clips give 55.0% with task-aligned prompts and 48.0% with a fixed placeholder prompt (195 scenes).

Limitations

  • This is an arXiv preprint; all numbers are reported by the authors [R] and have not been reproduced by this site. The authors say they “will” release the conversion pipeline and the CORE-annotated dataset; this site has not checked whether that has happened.
  • There are no real-robot results. The paper’s Future Work section says real-robot deployment is still underway because of hardware and schedule constraints. Every policy success rate comes from two tasks in RoboTwin 2.0 simulation; a simulated success rate is not a hardware safety validation.
  • CORE’s force is a visual and kinematic prior, not a sensor-calibrated force in Newtons; the authors plan to address this with tactile integration.
  • The authors acknowledge that contact localization under heavy occlusion is limited by object pose estimation error. With estimated poses, the frame F1 above is 0.386, far from the 0.924 with dataset poses.
  • The 33.7% applies only to the HaWoR, 6-clip, dataset-object-pose setting. Samples for the geometry metrics are small (6–31 clips, 3 audited takes for compositing), and the baselines have different inputs; in particular, Ego2Robot uses device-tracked hand data.
  • This article is based on the paper’s arXiv HTML text; supplementary material was not checked.

Links