This article is based on the September 2026 archive of the Physical AI Radar. Of the radar’s eight tracks, “Simulation and evaluation” collected the most items in September, with 11 in week W39 alone. After removing LLM evaluation articles and misclassified items, the remaining papers and official documents all address the same question: what does a success rate above 95% on LIBERO actually tell us, and what measurements are still missing?
The trend: initial-state success rate is no longer enough
The main metric in VLA papers today is the success rate of running a full trajectory from a preset initial state. Several works from September supplement this metric from four directions: first, testing whether a policy can recover after an execution deviation; second, reaching statistically sound conclusions with fewer trials when the real-robot budget is limited; third, measuring how much closed-loop performance survives deployment transformations such as quantization; and fourth, checking whether the simulator itself is faithful to the features the policy actually relies on.
Recovery: RoboRecover
RoboRecover ([R], 2026-09-24) selects deviated states from real execution trajectories, reconstructs the scene by replaying the action prefix, and then has the policy continue the original task. The benchmark has 2,000 scenarios, 1,000 each from RoboTwin and LIBERO, each with a fixed 800/200 train/test split. The numbers in the body of the paper speak for themselves:
| Platform | Policy | Initial-state success rate | Recovery success rate |
|---|---|---|---|
| LIBERO | UnifoLM | 98.83% | 48.00% |
| LIBERO | π0.5 | 94.17% | 64.40% |
| LIBERO | π0 | 85.17% | 37.80% |
| RoboTwin | LingBot-VA | 83.75% | 59.50% |
| RoboTwin | X-VLA | 54.33% | 49.83% |
The policy ranked first by initial-state success rate does not have the highest recovery success rate, and the drops across policies range from 4.5 to 50.83 percentage points. Another detail: deviations are often caused by a particular policy itself, and that policy’s success rate at recovering from its own deviations is only 15.24%–19.82% on RoboTwin, whereas a different policy can reach 58.06%–70.27%. Using the training split, the authors tried two interventions: monitor-triggered fallback raised π0.5’s recovery success rate on LIBERO from 64.4% to 77.0%, and a correction model reached 75.5%.
Too few real-robot trials: sequential testing and data splits
Beyond Binary Success ([R], by TRI and other authors; paper published 2026-03, collected by the radar in September) targets a practical constraint of real-robot evaluation: each policy can only be run a few dozen times. The method uses safe anytime-valid inference for sequential testing, so evaluators can stop early once the evidence is sufficient while keeping a pre-specified confidence level. It applies not only to binary success rate but also to graded task progress, episode return and trajectory smoothness. The authors validated it on simulation and real-robot data: compared with standard batch methods it cuts the number of evaluations by up to 70%, and compared with sequential methods designed for binary outcomes, by up to 50%. Fine-grained progress metrics distinguish policies faster than binary success rate, which agrees with RoboRecover’s view that a single overall score is not enough to describe policy behavior.
Discipline in data splits affects conclusions just as much. AWS’s π0 fine-tuning tutorial ([O], 2026-09-10) discloses on its own initiative that the roughly 96% improvement reported in an earlier version came from evaluation episodes overlapping with the training set; after strictly enforcing the split on both the training and the evaluation side, the figure dropped to about 89% (MSE reduced by 89.7% on DROID and 88.9% on LIBERO-10). Note that this measures offline action MSE, not closed-loop success rate, and each of the two datasets held out only 5 episodes for evaluation.
After deployment transformations: VLAQuantBench
What gets evaluated should be the version that is actually deployed. VLAQuantBench ([R], 2026-09-21) ran 409 experiment configurations and 94,574 simulation episodes across four models: OpenVLA-OFT, π0, π0.5 and X-VLA. Key results:
- With uncalibrated W4A4 round-to-nearest quantization of π0.5, expanding action-head protection from 126 to 167 layers makes the success rate jump from 7.0% to 70.5%.
- Calibrating with just two episodes eliminates the severe failures on the measured subset; but the same smoothing-and-clipping recipe lowers π0’s success rate and cannot restore end-to-end performance for OpenVLA-OFT.
- For OpenVLA-OFT, protecting a single 28,672-parameter output projection layer is enough to bring the success rate back close to baseline.
- On a real Franka Research 3 with 50 trials per condition, OpenVLA-OFT’s grasping task success rate is 88.0% in BF16, 86.0% in W8A8, down to 54.0% in W4, and only 12.0% in W3.
The authors conclude that these results only show that precision configurations and recipes interact; no general rule about “which layers are sensitive” can be derived. In other words, someone else’s quantization recipe cannot simply be reused, and every model needs its own closed-loop validation.
The simulator itself: fidelity and infrastructure
DreamStream ([R], 2026-09-22, autonomous driving) points out that perceptual metrics such as FID can rank simulators wrongly on whether they preserve the scene features a policy depends on. The authors propose FDπ, a Fréchet distance computed on the scene-context features of public end-to-end driving policies. Measured by FDπ, DreamStream is 1.6x better than the previous strongest closed-loop simulator on nuScenes and 4.7x better on NAVSIM. The Navhard-CL benchmark built on it exposes failure modes that earlier closed-loop benchmarks missed, such as scorer bias and a lack of recovery behavior. The finding comes from driving, but the idea of measuring simulation fidelity by policy features rather than pixel realism applies equally to robot arms.
On the tooling side, two first-party NVIDIA documents turn evaluation into a reusable engineering workflow. Isaac Lab-Arena ([O]) splits scenes, objects, embodiments and tasks into independent modules and supports subtask progress tracking and perturbation sensitivity analysis. A preliminary benchmark in the documentation reaches 2,390 environment-steps/s with 1,024 parallel environments on a single GPU; the version tested is release/0.3.0. The MjWarp migration tutorial ([O], Hugging Face Blog, 2026-09-23) uses an SO-101 block-stacking task as its example and proceeds in five steps: establish a MuJoCo baseline on CPU, verify single-world consistency on GPU, set the capacity of the contact and constraint buffers, scale out to 2,048 parallel worlds, and finally measure throughput. The tutorial deliberately gives no throughput numbers, on the grounds that results depend on the scene, simulation settings and hardware.
Limitations
- Apart from VLAQuantBench’s real-robot table and part of the Beyond Binary Success data, the numbers in this article come mainly from simulation benchmarks such as LIBERO and RoboTwin, where initial-state success rates are already close to saturation.
- All [R] numbers are self-reported by the authors in preprints; RoboRecover’s deviated states come from replaying action prefixes, and whether they match the distribution of real-world disturbances remains to be verified.
- The authors themselves label Isaac Lab-Arena’s throughput numbers as preliminary, and the documentation’s main branch is also marked as not yet fully tested.
- DreamStream’s findings have only been validated on driving datasets.
What to watch next
- Whether new VLA papers report recovery success rates or graded progress metrics alongside initial-state success rates.
- Whether closed-loop validation after quantization and distillation becomes standard when releasing models, rather than just a table in the appendix.
- Whether sequential testing makes it into mainstream evaluation frameworks, so that conclusions like “20 trials per task” come with a confidence level.
