Frontier Radar | Positioning a one-end-grasped rope with RL: GPU-parallel training stacks, relative keypoint observations and failure breakdowns, then 9/10, 10/10 and 8/10 on a real robot with no fine-tuning

Preprint: PPO for a one-end-grasped rope. GPU-parallel MuJoCo Playground converges faster than CPU stacks; authors report 9/10, 10/10, 8/10 real trials.

Cycle diagram: light background, the title “One-end-grasped rope: the perception–action loop” at the top and a source line at the bottom. Four numbered solid circles sit on a ring, joined in turn by arrows along the ring, with the fourth step leading back to the first; the four steps are listed by number beside it. The steps are: RGB-D mask and skeleton give 5 rope keypoints; actor: keypoint positions and velocities relative to the EE, and the target error; a 10 Hz policy outputs EE xy and yaw deltas; the Cartesian controller moves the EE and the rope deforms.
The deployment loop described in Fig. 2 and Section II of the paper (arXiv 2611.07261, authors' description): on the real robot, rope keypoints are estimated from RGB-D and combined with proprioception, target error and the previous action into the actor observation; the action becomes EE motion through the Cartesian pose controller, and the new robot and rope states form the next observation. In training, keypoints are read directly from the simulator and the critic also receives privileged simulator information; deployment needs only the actor-observable inputs, so the critic is not drawn in the loop. Numbers and arrows only show order within one pass; the four circles have the same size and colour and indicate no measurement or ranking. The 5 keypoints and 10 Hz are settings from the paper, not performance results.

The preprint An Empirical Study of Reinforcement Learning for Keypoint Positioning of Deformable Linear Objects (arXiv 2611.07261, 2026-11-05, from the HRII Laboratory at the Istituto Italiano di Tecnologia, IIT) studies a narrow but common problem: a robot holds one end of a deformable linear object (DLO) such as a rope or hose and moves chosen points on it to targets on a table. The paper is an empirical study: it uses PPO to test the training stack, observation design, robustness training and transfer to a real robot one by one. The authors report that, without real-world fine-tuning and using markerless RGB-D keypoint estimation, the policies succeed in 9/10, 10/10 and 8/10 trials for single-point tracking, disturbance recovery and two-point tracking. All numbers below are author-reported [R]; this site has not reproduced them.

The task: one end grasped, planar, slow

One end of the rope is rigidly attached to the end effector (EE) of a Franka Emika Panda; the other end is free. The Cartesian controller only allows the EE to translate in the xy-plane and rotate about the z-axis. The paper restricts the task to slow planar motion and treats it as quasi-static at the control level, neglecting dynamic effects. At each step the policy outputs a normalized three-dimensional action, two translation increments and one yaw increment, which are multiplied by fixed scaling factors. The policy runs at 10 Hz, the simulation integrates at 1 kHz, and each training episode lasts at most 200 control steps (20 s).

There are two tasks. Single-point tracking moves the keypoint at the rope’s free tip to a target; two-point tracking also moves the keypoint attached to the EE to its own target. A keypoint counts as reached within ε = 0.03 m. The two-point reward is staged: the tip error is penalized throughout, while the error of the EE-attached keypoint is penalized only after the tip has entered its target region. If the tip leaves that region, the reward again prioritizes recovering the tip. This design connects directly to the failure types discussed below.

Training stacks: parallel simulation mattered more than a faster learner

The paper compares three complete PPO training stacks: PyTorch Stable-Baselines3 with CPU-based MuJoCo (SB3+mj, 4 parallel environments); the same simulation with only the learner replaced by the JAX-based Stable-Baselines Jax (SBX+mj); and MuJoCo Playground (MP), which runs JAX PPO together with the JAX-native MJX simulator on the GPU across 1024 environments. Each stack was tuned separately with Optuna (three tuning seeds per trial, on 8 CPU cores and one A100 80 GB), then trained from scratch on 5 held-out seeds; all later runs used 8 CPU cores and one V100 32 GB.

Training stackFinal success rateArea under the success-rate curve
SB3+mj0.860 ± 0.1080.529 ± 0.048
SBX+mj0.880 ± 0.1100.532 ± 0.069
MP0.950 ± 0.0320.735 ± 0.077

Within a common wall-clock budget of about 240 minutes, the authors report that MP reaches a mean success rate of 0.95 while both CPU stacks reach only moderate success; over the same time MP collects about 13× more environment interactions. SBX+mj, which only swaps the learner, differs little from SB3+mj; the authors attribute this to the DLO physics simulation remaining the bottleneck. The environment count differs by 256× (1024 versus 4, this site’s arithmetic) while interactions differ by about 13×, so the number of environments should not be read as throughput. The authors also state that the three stacks differ at once in learning framework, simulation backend, parallelism and hardware, so this compares whole systems, not individual algorithms or simulators under control.

Observation design: relative keypoints and an asymmetric actor–critic

At deployment the policy only has vision-estimated keypoints, so the actor sees 5 keypoints, while during training the critic sees 12 keypoints plus privileged simulator information (the command-clipping magnitude and indicators for leaving the robot workspace and the work area). Keypoints are given as positions and velocities relative to the EE, together with the absolute position of the free tip in the robot-base frame. From the paper’s equations (1) and (3), the two-point actor observation has 37 dimensions and the critic observation 70 (this site’s arithmetic).

The ablation used the single-point task, with each of the four combinations trained for about 20 million steps on 2 seeds. The authors report that the EE-relative representation learns better than the robot-base-frame one, that the asymmetric actor–critic beats the symmetric one, and that “relative + asymmetric” performs best overall, so it is used for all later simulation and real-robot experiments. The text gives only learning curves for this ablation, not numerical values.

Robustness: paired comparisons and failure breakdowns say more than one success rate

Robust training (N+DR) adds zero-mean Gaussian noise at every step to the actor-observable positions, velocities, yaw and yaw rate, plus a random delay of at most one control step. Each episode randomizes the sliding-friction coefficient between rope and table, μ ∈ [0.5, 0.8], and scale factors for joint stiffness, actuator damping and passive damping (each between 0.95 and 1.05); disturbance-recovery training also applies an external force at a random time. Bending and torsional stiffness are not randomized, because the MP plugin used does not support changing them at runtime. Each condition is evaluated over 250 episodes (5 evaluation seeds × 50 episodes):

TaskConditionSuccesses / 250Success rate
Single-pointBasic24999.6%
Single-pointN+DR24196.4%
Single-point (disturbance recovery)N+DR+D24598.0%
Two-pointBasic23192.4%
Two-pointN+DR22288.8%

The slightly higher rate with disturbances is not taken as evidence that disturbances help. Both conditions use the same targets and initial states, so episodes can be paired: 238 outcomes are unchanged, 8 change from failure to success and 4 from success to failure. The authors note that during training the force occasionally pushed the tip toward the target, which may partly explain the difference, and limit the conclusion to “robustness is retained”. The two-point task pairs the same way: 213 target pairs succeed under both conditions, 18 only under Basic, 9 only under N+DR and 10 under neither, so N+DR causes a modest net drop rather than a uniform loss of capability.

All 28 two-point failures under N+DR are timeouts. By type: the tip never reached its target (Type-1) in 19, or 67.9%; the tip reached it but the EE-attached keypoint did not (Type-2) in 2, or 7.1%; the tip reached its target and had left it by termination (Type-3) in 7, or 25.0%. Failures cluster at the edge of the workspace: “peripheral” pairs, where either target lies within 0.05 m of its admissible boundary, make up 44.4% of the 250 episodes but 78.6% of failures, and the failure rate is 19.8% at the periphery against 4.3% in the interior. Wider target separations are harder: success is 92.7%, 90.9% and 83.1% for separations below 0.10 m, 0.10–0.20 m and at least 0.20 m, which the authors call a descriptive association.

A separate single-point policy trained jointly on rope lengths of 0.300, 0.375 and 0.450 m reaches 84.4%, 86.0% and 86.4% at those lengths. All three lengths were in training, so this measures in-distribution robustness, not extrapolation to an unseen length. The text does not state the evaluation condition for this policy, so this site does not compare it directly with the fixed-length 96.4%.

Real robot: 30 trials with no fine-tuning

An Intel RealSense D435i mounted above the workspace supplies images at 30 Hz; the camera is registered to the robot-base frame with ArUco-based extrinsic calibration, and the perception and control pipeline is integrated through ROS. A fine-tuned YOLO26-seg model segments the rope, the mask is skeletonized, and 5 approximately equally spaced keypoints are taken along the centerline and projected onto the table. The authors report a mask precision of 98.81%, recall of 99.13%, F1 of 98.97%, mAP@0.5 of 99.48% and mAP@0.5:0.95 of 79.30%, with inference at 29.8 FPS. The policies run with no change to the network and no real-world fine-tuning, still at 10 Hz.

The primary evaluation uses a silicone catheter 0.004 m in diameter, with 10 trials per task, and all trials are included:

TaskSuccessMedian completion time of successful trials [IQR]Failure type
Single-point tracking9/109.30 s [6.70, 14.40]Type-1
Disturbance recovery10/106.55 s [4.85, 8.32]—
Two-point tracking8/10tip reached 8.55 s [7.19, 12.30]; task completed 9.05 s [7.80, 12.47]Type-3

In the two-point task, completing the whole objective after the tip arrived took a median of only 0.56 s more [IQR 0.17–2.02 s]. Both failures were cases where the tip reached its target and then left it, the same mechanism as Type-3 in simulation. The authors group the real-world failures into three causes. The robot itself sometimes occluded the rope: after short occlusions the policy could recover once valid observations returned near the goal, but longer ones led to incorrect actions. The catheter is very compliant and sometimes bent sharply and released stored elastic energy, causing sudden deflections in compact two-point configurations. Perception delay sometimes made the policy overshoot the target; the authors say training with noise and delay reduced this and allowed recovery in several trials.

How this relates to the course on this site

Physical AI Lab 0.4 and 1.4 covered speed-up figures for GPU-parallel simulation and why every multiple needs its baseline. This paper adds a deformable-object case: accelerating only the learner barely helped, and the gap opened once simulation was parallelized as well. What it reports is success rate and interaction count under one wall-clock budget, not a single speed-up factor.

Physical AI Lab 2.5 argued that domain randomization should state which model quantity it changes and over what range. This paper states its friction range, stiffness and damping scales, and observation noise and delay, and also says that bending and torsional stiffness could not be randomized. The real-world failures where the catheter bent sharply and sprang back can be read alongside that gap; this is this site’s reading, and the paper does not attribute those failures to the unrandomized stiffness. A delay of at most one control step at 10 Hz corresponds to 0.1 s (this site’s conversion); the link between latency and control rate is covered in 2.6.

Physical AI Lab 1.8 practised splitting a “net gain” into a paired table. The 8-versus-4 and 18-versus-9 counts in this paper are that reading in practice: when overall success rates differ by a few points, first check how many episodes actually changed and in which direction. For how a staged reward shapes behaviour, compare 1.7.

Limitations

  • This is an arXiv preprint; all numbers are author-reported [R] and this site has not reproduced them. The text this site read contains no code or dataset link.
  • Each real-world task has only 10 trials with a single silicone catheter; the paper gives no confidence intervals for the real-world results, and the text this site read does not state the catheter’s length. The default simulated rope length is 0.30 m.
  • The task is limited to planar, slow, one-end-grasped manipulation treated as quasi-static. The authors list shape control, three-dimensional manipulation and generalization to unseen DLOs, environments and task configurations as future work.
  • The stack comparison changes the learning framework, simulation backend, parallelism and hardware at once, and the authors state explicitly that it is not a controlled comparison of single components. The observation ablation uses only 2 seeds, and the text gives curves but no numbers.
  • The policies are task-specific: the observation size depends on the number of tracked points M, so a separate policy is instantiated for each M. The variable-length experiment covers only the three lengths seen in training.
  • The text this site read does not state the size or split of the test set behind the segmentation metrics.
  • The real-robot trials validate task success, not safety; simulated success rates do not replace real-world testing.

Links