When a robot policy drifts during execution, correcting it with repeated VLM checks or online optimization can incur substantial latency. RESETTLE, a preprint from HKUST and collaborating institutions (arXiv 2610.12185, 2026-10-08), puts recovery at the action-execution interface of a frozen policy. It samples two action chunks independently from the same observation and intervenes only when their disagreement exceeds a threshold at two consecutive decisions. An adapted V-JEPA 2 encoder then retrieves the nearest frame from successful demonstrations of the same task, and a “state-servo prior plus guarded visual residual” executes a single control action before control returns to the base policy. The authors report absolute success-rate gains of up to 8.70, 6.28 and 6.83 percentage points on LIBERO-Plus, Meta-World and RoboCasa Tabletop across six base policies, further gains on four real-robot tasks, and a measured monitoring-plus-recovery computation time of 83.6 ms on QwenPI. All numbers below are author-reported [R]; this site has not reproduced them.
The paper sorts existing recovery methods into three groups. The first trains corrective behaviour with extra supervision: DAgger collects expert labels on states the learner visits, and IntervenGen generates recovery trajectories from human interventions; both need additional data and policy updates. The second uses VLM supervision: Agentic Robot triggers corrections and retries through visual verification, and VoLoAgent switches between policy execution and recovery tools; repeated verification and recovery computation add latency. The third corrects actions at inference time without changing weights, as LPB and VLA-Corrector do, but relies on gradient guidance and its extra computation. The authors also report that on generalized VLAs/WAMs, the expert-feature distance LPB uses overlaps more between successful and failed executions. RESETTLE belongs to the third group, but uses no gradient guidance or online trajectory optimization, makes no extra VLM calls, and needs no access to the base policy’s internal representations.
When to intervene: two samples under identical conditioning, and the disagreement has to persist
Many generative policies sample action chunks from noise with diffusion or flow matching, so the same observation and instruction can yield a different chunk under a different noise sample. At each nominal decision, RESETTLE generates two action chunks from two independent noise samples and computes a temporally weighted disagreement using an embodiment-specific action distance; weights can decrease for later actions, and for GR1, γ = 0.8 gives earlier actions more weight. For policies with Cartesian actions, the distance is the Euclidean distance between translational components only, excluding rotation and gripper commands; for the GR1 in RoboCasa it is a distance computed from squared differences over all 29 normalized coordinates (7 per arm, 6 per hand, 3 for the waist).
The threshold is the 95th percentile (P95) of disagreement scores from offline inference on that base policy’s training data; evaluation outcomes are not used, and each base policy is calibrated separately. LIBERO-Plus uses one threshold for all perturbation types and tasks; the other benchmarks use a benchmark-wide threshold shared across tasks. Recovery triggers only when the threshold is exceeded at two consecutive decisions, which suppresses isolated sampling fluctuations. Without a trigger, the first sample is executed; the exceedance count resets after every intervention.
The authors’ rationale: on QwenPI, disagreement from training-data inference and from successful executions sits in the dominant low-disagreement region, while failed executions have heavier tails. Under the “two consecutive P95 exceedances” rule, across seven LIBERO-Plus perturbation categories, 77.89%–100.00% of failed trajectories trigger, but so do 9.22%–42.81% of successful ones. In the authors’ words, persistent disagreement is informative but not a definitive failure label, which is why the design applies a brief correction and then re-observes. The paper also reports that calibrating on just 5 training trajectories gives a disagreement distribution similar to 100 trajectories or the full training set, and that single- and two-seed inference have the same measured latency at the reported precision, about 297 ms for FastWAM and about 109 ms for QwenPI, so monitoring adds negligible measured latency in these implementations.
How to correct: retrieve a demonstration frame, then a servo prior plus a guarded residual
After a trigger, the adapted encoder maps the current image to a pooled, normalized representation and finds the nearest frame by cosine distance among successful training demonstrations of the same task; that frame supplies both a reference image and the associated robot state. If no same-task reference is available, control stays with the base policy. The encoder is a ViT-g/16 of roughly 1B parameters, initialized from the V-JEPA 2-AC checkpoint with action-conditioned pretraining on DROID and then trained jointly on expert demonstrations with objectives including residual-action supervision and dynamics prediction; the dynamics predictor does not run at deployment. One recovery module is trained per setting (LIBERO-Plus, Meta-World, RoboCasa and the real robot) and shared by the base policies in that setting.
The corrective action has two parts. The displacement from the current to the reference robot state gives a bounded state-servo prior in native control coordinates. A visual residual head, conditioned on the current and reference image representations, robot state, previous action and the prior, adjusts that prior. The guard requires the residual’s cosine similarity with the prior to stay above a minimum and the residual-to-prior norm ratio to stay below a maximum; on LIBERO, translation and rotation are checked separately. An accepted residual is added to the prior before clipping to native command limits; a rejected one leaves only the bounded servo action. Each intervention discards the base policy’s pending actions, executes one native control step, and returns control to the base policy with a fresh observation.
The appendix states that demonstration references are empirical anchors of successful behaviour, not certificates of physical reachability, and do not define the base policy’s full feasible range.
Success rates: three simulation benchmarks, hierarchical planning and real robots
Each comparison uses identical frozen weights, evaluation conditions and execution budgets, and gains are absolute differences between two percentages (percentage points). LIBERO-Plus has 10,030 cases across seven perturbation types; all three base policies were trained on standard LIBERO without further fine-tuning, and the average weights each type by its number of cases. Meta-World covers all 50 tasks at 10 episodes each, 500 episodes in total, and the average is the unweighted mean of four difficulty groups. RoboCasa is the 24-task GR1 Tabletop suite. LIBERO-Pro Swap is the same set of 100 trials, 10 tasks × 10 seeds.
| Benchmark | Base policy | Baseline (%) | +RESETTLE (%) | Gain (points) |
|---|---|---|---|---|
| LIBERO-Plus | FastWAM | 49.3 | 58.0 | +8.7 |
| LIBERO-Plus | π0.5 | 70.0 | 73.7 | +3.7 |
| LIBERO-Plus | QwenPI | 80.2 | 84.6 | +4.4 |
| Meta-World | QwenPI | 60.66 | 65.99 | +5.33 |
| Meta-World | SmolVLA | 59.20 | 65.48 | +6.28 |
| RoboCasa Tabletop | LDA | 50.33 | 52.42 | +2.09 |
| RoboCasa Tabletop | QwenGR00T | 53.67 | 60.50 | +6.83 |
| LIBERO-Pro Swap | π0.5 | 15.00 | 24.00 | +9.00 |
| LIBERO-Pro Swap | GPT-5.5 + π0.5 (Harness VLA) | 42.00 | 50.00 | +8.00 |
On LIBERO-Plus the largest gains come under robot-initialization perturbations: +16.7, +17.8 and +21.4 points for FastWAM, π0.5 and QwenPI; π0.5 and QwenPI each drop 0.2 points under lighting perturbations. With the servo prior alone and no residual, overall success is 55.9%, 72.9% and 83.1%; adding the guarded residual gives 58.0%, 73.7% and 84.6%. The OpenVLA and other baselines in the upper part of the paper’s Table 1 are taken from their own papers, not paired evaluations in this work.
The last two LIBERO-Pro Swap rows come from Harness VLA: a GPT-5.5 planner coordinates execution from instructions, observations and memory, π0.5 handles local manipulation, and RESETTLE is added at the low-level policy’s action-execution interface. The authors take this as evidence that local correction complements high-level planning.
Real robot: four tasks on a Franka Research 3
The real-robot experiments use a Franka Research 3 with external and wrist-mounted cameras. The base policies are QwenPI and VLAct (both on a Qwen3-VL backbone); they share one real-world recovery module, with thresholds calibrated separately. Each configuration runs 20 trials per task with randomized initial arm configurations and object layouts, 320 trials in total.
| Task (metric) | QwenPI baseline | QwenPI +RESETTLE | VLAct baseline | VLAct +RESETTLE |
|---|---|---|---|---|
| Press-button (success rate) | 90 | 100 | 90 | 100 |
| Cube-up (success rate) | 55 | 70 | 85 | 90 |
| Carrot-in-pot (success rate) | 85 | 95 | 90 | 95 |
| Use Spoon (stage completion) | 36.7 | 50 | 80 | 86.7 |
| Four-task average | 66.67 | 78.75 | 86.25 | 92.92 |
The first three tasks report binary success rate; Use Spoon is a long-horizon task scored by normalized stage completion, and the four-task average simply averages the two kinds of metric. With 20 trials per task, one trial is worth 5 percentage points (this site’s arithmetic): VLAct’s 5-point gains on Cube-up and Carrot-in-pot each amount to one more success out of 20. The text this site read gives no confidence intervals for the real-robot results.
Latency: 83.6 ms measures computation, not response time
Table 6 of the paper implements several methods on QwenPI and measures them; the paper states that local policy and recovery inference runs on NVIDIA RTX 4090D GPUs. “Monitor” is latency beyond nominal policy inference; “Total” is monitor plus recovery, excluding nominal inference, polling delays and physical execution. Both agentic methods use Qwen3-VL-4B-Instruct for monitoring; LPB is a version the authors adapted to their own QwenPI+V-JEPA framework.
| Method | Monitor (ms) | Recovery (ms) | Total (ms) | Relative to RESETTLE |
|---|---|---|---|---|
| RESETTLE | ≈ 0.0 | 83.6 | ≈ 83.6 | 1.0× |
| Agentic Robot | 107.0 | 109.0 | 216.0 | 2.6× |
| VoLoAgent | 107.0 | 215.0–1193.0 | 322.0–1300.0 | 3.9–15.6× |
| LPB (authors’ adaptation) | 56.2 | 423.5 | 479.7 | 5.7× |
Total latency is 61.30% lower than Agentic Robot’s, 82.57% lower than the adapted LPB’s, and 74.04%–93.57% lower than VoLoAgent’s place and grasp branches; the lower end of VoLoAgent’s range is the place branch, the upper end the grasp branch. The methods produce different things: RESETTLE produces one action, while VoLoAgent prepares a trajectory. The authors also note that RESETTLE checks only at nominal decisions, so its response is limited by the decision rate and the two-consecutive-exceedance rule; the totals in the table measure computation, not end-to-end response time.
How this relates to earlier pages on this site
The course lesson Physical AI Lab 1.6: Imitation Learning showed how behaviour-cloning errors accumulate and carry a robot outside the demonstrations’ coverage. RESETTLE targets exactly such states, but without retraining the policy: at run time it treats persistently high disagreement as a sign the policy may be entering unfamiliar states, then pushes the robot one step toward the nearest demonstrated state. The recovery benchmarks in Frontier Radar deep dive | Simulation and evaluation measure whether a policy itself can recover from deviations; RESETTLE adds a recovery module outside the policy. One is an evaluation, the other a method.
Two points help when reading its evaluation. RESETTLE uses LIBERO-Plus, the same perturbation benchmark as Frontier Radar | Is Success All You Need?; that article cautioned that similar success rates do not mean similar successful trajectories, and RESETTLE reports only success rates: the text this site read reports nothing on the duration or smoothness of corrected trajectories. Physical AI Lab 1.8: Evaluation showed how a net total can cancel gains against losses; RESETTLE’s trigger statistics show that successful trajectories are also intervened on, and the text this site read gives no paired count of gains and losses.
Limitations
- This is an arXiv preprint and all numbers are author-reported [R]. The authors are from The Hong Kong University of Science and Technology, The Chinese University of Hong Kong, SmartMore, Harbin Institute of Technology (Shenzhen) and the Shenzhen Loop Area Institute. This site has reproduced none of the results; the paper says code is available, which this site has not checked.
- Disagreement measures unstable action selection; it does not directly test physical feasibility or task correctness. The paper states that spatial or instruction-following errors may not trigger intervention when the policy’s proposals are consistent, and that recovery can still fail after a trigger because of unresolved physical configurations or mismatched goals.
- The guard only limits the residual’s direction and size relative to the prior. The authors say the results support visual refinement without establishing collision avoidance; neither the simulation results nor the 320 real-robot trials are hardware safety validation.
- Recovery depends on successful same-task demonstrations and a task identifier, and does not intervene without a reference. Recovery modules are trained per setting, so a new task or robot needs corresponding demonstration data.
- 9.22%–42.81% of successful trajectories trigger, depending on the perturbation category. The paper’s HTML text that this site read ends in Appendix B and does not include the training details, trigger statistics, ablations, latency details and failure cases in Appendices C–E.
- The real-robot experiments use 20 trials per task, the text gives no confidence intervals, and the four-task average mixes two metrics. Latency was measured only for QwenPI-based implementations (the paper states local inference runs on RTX 4090D GPUs), and the compared methods are the authors’ implementations or adaptations.
Links
- RESETTLE: Robotic Recovery through Disagreement-Triggered Retrieval and Efficient Corrective Control (arXiv 2610.12185)
- Full paper (HTML)
- Physical AI Lab 1.6: Imitation Learning — Working Out How an Error Rolls Out of the Demonstrations’ Coverage, Then Putting the ACT, Diffusion Policy and Data Scaling Numbers Back in Their Own Conditions
- Physical AI Lab 1.8: Evaluation — Splitting a “Net Gain of 2” into a Paired Table, Then Checking the Failure Labels and Random Seeds
- Frontier Radar deep dive | Simulation and evaluation: what VLA evaluation is missing beyond initial-state success rate
- Frontier Radar | Is Success All You Need?: two VLAs with similar success rates can succeed in very different ways
- Physical AI Lab: a physical AI course where every number can be checked (course outline)
