Most VLA robustness evaluations report a single number: task success rate (TSR) under perturbation. A preprint from teams at the University of Edinburgh and Politecnico di Milano, Is Success All You Need? (arXiv 2610.01351, 2026-10-01), asks a different question: once a task counts as a success, how did the robot get there? On LIBERO and LIBERO-Plus, the authors look only at successful trajectories and compare duration, jerk, path length and gripper motion before and after perturbation. The paper reports that under the LIBERO-Spatial camera perturbation, π0.5 and VLANeXt reach success rates of 69.7% and 68.0%, yet π0.5’s successful trajectories grow 22.5% longer in duration, 24.1% higher in mean Cartesian jerk and 37.8% higher in gripper movement, against only 4.2%, 3.2% and 2.5% for VLANeXt. All numbers below are author-reported [R], all come from simulation, and this site has not reproduced them.
The problem: a success rate cannot record “dropped it and picked it up again”
Figure 1 of the paper gives a concrete case. On the LIBERO-Spatial task “pick up the black bowl on the ramekin and place it on the plate”, π0.5 under the LIBERO-Plus camera-angle perturbation eventually meets the goal condition, but drops the bowl and re-picks it along the way. That trajectory counts as one success in TSR; the change shows up only in the mean Cartesian jerk and the cumulative gripper movement.
The authors argue that high jerk means abrupt changes in acceleration, which can bring higher mechanical stress and wear, and that completion speed and predictability affect how far users trust a robot. The paper surveys prior work: RoboEval and BEHAVIOR report behavioural metrics beyond TSR but do not apply robustness perturbations; LIBERO-Plus, LIBERO-PRO, Colosseum V2 and others apply perturbations but still measure the drop in TSR. The authors state that, to their knowledge, the two practices have not yet been combined.
Method: successful trajectories only, compared task by task with the unperturbed baseline
Each successful trajectory yields seven values: duration (simulator time, excluding inference latency), total gripper movement (cumulative absolute change in the sum of the two finger positions), mean and P95 Cartesian jerk, mean joint jerk, Cartesian path length and joint path length. All derivatives are computed at the simulator’s 10 Hz control frequency.
- Typical behaviour: for each task, take the mean of a metric over successful trajectories under perturbation and compute its percentage change from the original LIBERO baseline; then take the median over the suite’s 10 tasks. The paper also reports how many tasks show a significant increase (two-sided Mann-Whitney U test with Benjamini-Hochberg FDR correction across tasks, α = 0.05).
- Consistency: for each task, compute the median absolute deviation (MAD), and again report the median of its percentage change from baseline; a positive value means successful trajectories are more spread out.
The experiments cover four suites, LIBERO-Spatial, -Long, -Goal and -Object, and LIBERO-Plus’s seven perturbation classes: object layout, camera viewpoint, robot initial state, language instruction, lighting, background texture and sensor noise. The baseline runs 50 rollouts per task; each perturbed variant is run twice (the object perturbation provides only one initial state). The models are π0.5, VLANeXt and OpenVLA-OFT, all using checkpoints fine-tuned jointly on every LIBERO suite. OpenVLA-OFT was evaluated on the Spatial suite only, because its mean TSR across that suite’s seven perturbations had already fallen to 67.9%, and the LIBERO-Plus paper’s results indicate further degradation on the Goal and Long suites. Evaluation builds on the open-source vla-eval harness, which the authors extended to output end-effector position, joint position and gripper trajectories.
Results: camera, initial-state and sensor perturbations change successful trajectories
The authors report that successful trajectories are not invariant to perturbation; the largest effects come from the camera, robot initial state and sensor noise classes, and their size depends on the model. Under the camera perturbation on the Spatial suite, π0.5’s mean Cartesian jerk rises by a median 24.1% (significant in 7 of 10 tasks) and OpenVLA-OFT’s by 35.3% (10/10). VLANeXt generally shows smaller shifts. The two path-length metrics change little under perturbation.
A lower success rate goes together with behavioural change, but the two are not the same thing. Running a Spearman rank correlation over 63 model × suite × perturbation combinations, the authors find moderate positive correlations between TSR degradation and the change in total gripper movement (ρ = 0.56), P95 Cartesian jerk (0.54), duration (0.52), mean Cartesian jerk (0.43) and mean joint jerk (0.4). Cartesian and joint path length show no significant correlation (ρ = −0.03, p = 0.80; ρ = 0.2, p = 0.11). A moderate correlation means TSR hints at some of the behavioural change but cannot replace behavioural metrics.
Similar success rates, behavioural changes several times apart
The paper’s most direct evidence is a comparison of two models under the same perturbation (LIBERO-Spatial, camera-viewpoint perturbation; values are the median percentage change over 10 tasks, with the number of tasks showing a significant increase in parentheses):
| Model | TSR (successes / total) | Duration | Mean Cartesian jerk | Total gripper movement |
|---|---|---|---|---|
| π0.5 | 69.7% (524/752) | +22.5% (10/10) | +24.1% (7/10) | +37.8% (9/10) |
| VLANeXt | 68.0% (511/752) | +4.2% (7/10) | +3.2% (5/10) | +2.5% (0/10) |
On TSR alone the two are nearly tied; on successful trajectories, π0.5’s execution becomes clearly slower and jerkier, with more gripper opening and closing. On the Long suite the ranking even flips: under the background, camera, initial-state and sensor perturbations, π0.5 has the higher TSR in every case, yet shows larger increases in mean Cartesian jerk, mean joint jerk and P95 Cartesian jerk in each of them. Under the background perturbation, for example, π0.5’s TSR is 96.9% against VLANeXt’s 79.4%, while P95 Cartesian jerk rises by 11.6% and 6.2% respectively.
The third question in the course lesson Physical AI Lab 0.6: How to Judge Robot News — Tracing Each Question Back to a Sentence in the Source is “a success rate on what?”. This paper adds the next step: the same success rate can sit on top of very different executions.
Consistency: some perturbations spread successes out, others narrow them
The authors report that the camera perturbation increased the MAD of duration, mean Cartesian jerk and gripper movement for every evaluated model and suite: successful executions not only moved away from the baseline but also became less consistent with each other. The background, language, lighting and object perturbations, by contrast, often reduced MAD; under the lighting perturbation on LIBERO-Goal, for example, gripper-movement MAD fell by 28% for π0.5 and by 21% for VLANeXt.
The authors offer two possible explanations and state that they have not yet distinguished them. One is a selection effect: perturbation makes longer, jerkier trajectories such as “recover after a misgrasp” more likely to fail, so the remaining successes skew towards clean, direct ones. The other is that the perturbation itself narrows the range of the policy’s responses. This matters for interpretation: when metrics are computed only on successful trajectories, the perturbation also changes which trajectories are counted, so a smaller MAD cannot be read directly as “more stable”.
Applying it to your own evaluation: a few more columns next to TSR
The steps below are this site’s summary of the paper’s method, not a checklist given by the paper. They assume your evaluation harness can export per-step end-effector position, joint position and gripper state (the paper notes that beyond LIBERO, benchmarks such as MetaWorld that expose trajectory-level state and action information also qualify).
- Report the number of successes and the total next to TSR, because the sample size of behavioural metrics changes with the success rate.
- Compute duration, jerk and gripper movement per task on successful trajectories only, compare against the same task’s unperturbed baseline, then take the median across tasks and mark how many tasks are significant.
- Report variability (for example MAD) separately, keeping the selection effect above in mind.
- When comparing two models, first find conditions where their TSR is similar, then check whether the behavioural metrics separate them.
This differs from Frontier Radar | Beyond Binary Success: sequential tests that separate robot policies with fewer real-world rollouts, which is about telling two policies apart statistically with fewer evaluation runs; this paper is about the quality of a success itself. The two can complement each other. For more on evaluation gaps, see Frontier Radar deep dive | Simulation and evaluation: what VLA evaluation is missing beyond initial-state success rate; the course plan is in Physical AI Lab: a physical AI course where every number can be checked.
Limitations
- This is an arXiv preprint, evidence level R; all numbers are author-reported and this site has not reproduced them.
- All experiments run in LIBERO / LIBERO-Plus simulation; the authors state explicitly that whether these behavioural effects extend to physical robot deployment remains to be established.
- Only successful trajectories are analysed; failed trajectories are out of scope. Full evaluation covers only π0.5 and VLANeXt; OpenVLA-OFT has results for the Spatial suite only and cannot be compared with the other two on the other suites.
- The metrics are generic and task-agnostic, so their meaning is not specific: more gripper movement may correspond to repeated grasps or drop-and-regrasp, but confirming such events requires task-specific analysis.
- Duration uses simulator time and deliberately excludes inference latency; the authors note that using wall-clock time instead could favour smaller or faster models.
- The percentages in the table are medians over 10 tasks of within-task mean changes, not changes in a single trajectory, and cannot be converted directly into wear or safety risk.
- The paper says the evaluation framework has been integrated into vla-eval and gives a code repository in a footnote; this article has not examined that code.
Links
- Is Success All You Need? Investigating the Impact of Input Perturbations on VLA Behaviour in Tabletop Manipulation Tasks (arXiv 2610.01351)
- arXiv 2610.01351 full HTML text (Tables II and III)
- Physical AI Lab 0.6: How to Judge Robot News — Tracing Each Question Back to a Sentence in the Source
- Frontier Radar | Beyond Binary Success: sequential tests that separate robot policies with fewer real-world rollouts
- Frontier Radar deep dive | Simulation and evaluation: what VLA evaluation is missing beyond initial-state success rate
- Physical AI Lab: a physical AI course where every number can be checked
