
Several VLAs now exceed 95% success on the LIBERO simulation benchmark, but how much of that score needs generalization beyond the training demonstrations has stayed unclear. In the preprint How Much of Your VLA Is kNN (arXiv 2611.13160, 2026-11-09), Denis Grachev, Damien LaRocque and Omar Moured of Sereact GmbH (Stuttgart) offer a direct reference point: train no policy at all, find the few demonstration frames closest to the current observation, and execute a weighted average of the action chunks that follow them. The authors report that this nearest-neighbor (kNN) policy reaches 91.7% on LIBERO, and that on the perturbed variants of LIBERO-Plus its success rate drops less than that of any other evaluated policy. On that basis the paper recommends reporting retrieval baselines alongside VLA benchmark results. All numbers below are author-reported [R]; this site has not run the authors’ code.
Method: turning demonstrations into a lookup table
Offline, the authors store every second frame of each demonstration: the observation, the action chunk of horizon H that follows it, and the demonstration it comes from. LIBERO’s 1693 demonstrations give about 137k entries. At run time the current observation is compared with every entry by a weighted distance whose terms cover image features, robot state, the language instruction, the episode time step and the recent observation history. The vision, state and time terms are divided by their median within-task distance so that terms on different scales can be added (the language term is normalized by the median distance between different task instructions, and the gripper term is not normalized). The k entries with the smallest distance are kept, with at most one entry per demonstration so that no single demonstration is copied wholesale, and their action chunks are averaged with softmax weights over distance.
Image features come from a frozen ImageNet-pretrained ResNet-18 (a 4 × 4 spatial grid from layer 3), and language from unit-normalized CLIP embeddings. The authors compare three variants. In all of them the executed actions are weighted combinations of demonstrated actions:
- kNN (ResNet): image distance only, retrieval restricted to the demonstrations of the instructed task, k = 8, and the full weighted chunk is executed before retrieving again.
- kNN (multimodal): retrieval from all demonstrations with the full multi-term distance, k = 13, plus mode seeking (down-weighting neighbors outside the dominant action mode) and temporal ensembling (re-planning at every control step and combining the overlapping predictions).
- kNN (LAC key): the same retrieval and execution as the previous variant, with a learned linear projection on the ResNet features, trained on a differentiable surrogate of the LAC metric introduced in the fifth section. It is the only one of the three that involves training.
Hyperparameters are tuned with Optuna. LIBERO has no separate validation split, so the search uses initial states 40–49 of every task (400 episodes), which are also part of the official evaluation protocol. On initial states 0–39, never used in the search, the authors measure 91.6% for the multimodal variant, against 91.7% on all 50 states.
Results on LIBERO and LIBERO-Plus
The LIBERO evaluation covers the Spatial, Object, Goal and Long suites, 40 tasks in total, with 50 episodes per task under the official protocol, or 2000 episodes per policy. LIBERO-Plus applies seven categories of perturbation to the same tasks; the authors fix a subset of 420 variants (60 per category), shared by all policies and held out from training and tuning. The nine VLAs are public LIBERO checkpoints from LeRobot, evaluated as released with their own observation, action and inference conventions. The table gives the average columns of Table 1 in the paper (%). Δ is the LIBERO-Plus average minus the LIBERO average, in percentage points (pt), computed before rounding; the VLAs are ordered by their LIBERO-Plus average, as in the original table.
| Policy | LIBERO avg | LIBERO-Plus avg | Δ (pt) |
|---|---|---|---|
| π0.5 | 97.7 | 85.5 | -12.2 |
| MolmoAct2 | 97.5 | 82.6 | -14.9 |
| GR00T N1.7 | 94.9 | 80.2 | -14.6 |
| VLA-JEPA | 97.2 | 74.0 | -23.1 |
| Evo-1 | 95.4 | 71.4 | -24.0 |
| X-VLA | 94.5 | 64.5 | -30.0 |
| π0 | 71.9 | 42.9 | -29.0 |
| SmolVLA | 82.2 | 27.9 | -54.3 |
| π0-FAST | 90.6 | 25.7 | -64.8 |
| kNN (ResNet), training-free | 88.7 | 75.5 | -13.2 |
| kNN (multimodal), training-free | 91.7 | 82.4 | -9.3 |
| kNN (LAC key), trained retrieval key | 93.0 | 81.7 | -11.3 |
The diagram in this article draws the drop for five of these rows, that is, the LIBERO average minus the LIBERO-Plus average, which equals the absolute value of Δ in the table, in the table’s order: π0.5 12.2 pt, VLA-JEPA 23.1 pt, π0-FAST 64.8 pt, kNN (ResNet) 13.2 pt, kNN (multimodal) 9.3 pt.
π0.5 has the highest success rate on both benchmarks. The kNN policies perform on par with π0-FAST on LIBERO and above SmolVLA and π0. VLA-JEPA, Evo-1 and X-VLA all exceed 94% on LIBERO, yet on LIBERO-Plus they fall below the training-free ResNet variant (75.5%). The multimodal variant outperforms 7 of the 9 VLAs on LIBERO-Plus, and its 9.3 pt drop is the smallest of all policies. The authors read this as showing that strong LIBERO performance does not necessarily carry over to robustness under the LIBERO-Plus perturbations. Appendix C, which breaks LIBERO-Plus down by perturbation category, adds one detail: camera perturbations lower the success of every policy except the LAC key variant, which reaches 88%, against at most 73% for the others.
Two points matter when reading the table. First, the paper gives an interval only for LIBERO: a 95% confidence interval of about ±1.2 pt over 2000 episodes. It gives none for LIBERO-Plus, and the text we read does not say how many times each variant is run. If each variant runs once, a success rate around 80% over 420 trials has a 95% interval half-width of about 3.8 pt (this site’s rough binomial estimate). The drops of kNN (multimodal) and π0.5 differ by 2.9 pt, which should not be read as a firm ordering. Second, the reproduced success rates of π0 and SmolVLA are below those their authors report, which the paper says may reflect differences between the public checkpoints and those used in the original publications. The VLA numbers in the table are results of public checkpoints run as released, not each model’s best result.
What retrieval depends on: the ablations
The authors change one component of the multimodal variant at a time (Table 3 in the appendix, LIBERO, 2000 episodes):
- Keeping only 10% of the demonstrations lowers success from 91.7% to 77.4%; with 25% and 50% it is 82.6% and 88.5%.
- Raw 16 × 16 pixels instead of ResNet-18 features give 57.2%, with 0.0% on LIBERO-Object. The ten tasks of that suite share one scene containing several small objects.
- Ignoring the language prompt costs 27.1 pt, mostly on LIBERO-Goal (8.8%). The ten Goal tasks also share one scene (Appendix D), so they are told apart mainly by the instruction (this site’s inference).
- Removing mode seeking, temporal ensembling, history matching, robot state or the time-step feature changes average success by at most 1 pt, within the ±1.2 pt interval.
- For k between 4 and 32, success stays between 89.4% and 92.7%; with k = 1, replaying only the nearest demonstration, it falls to 78.6%.
The authors conclude that the visual representation, the language prompt and the number of demonstrations account for most of the performance, and the remaining refinements contribute less. A hyperparameter importance analysis (fANOVA) points the same way: k and the at-most-one-neighbor-per-demonstration constraint account for 57% and 25% of the importance.
Using retrieval to measure VLAs
The same lookup can also probe the VLAs themselves. The authors replace the ResNet variant’s image features with the hidden states of the image tokens at one layer of a VLA’s VLM backbone, and compare that backbone as a retrieval key before and after LIBERO fine-tuning (Table 2 in the paper):
- After fine-tuning, retrieval improves for 5 of the 8 fine-tuned VLAs on LIBERO and for 7 of 8 on LIBERO-Plus, by 5 to 37 pt. For these VLAs, the drop from LIBERO to LIBERO-Plus also shrinks, from 20.9–34.7 pt before fine-tuning to 10.4–21.0 pt after. GR00T N1.7 freezes its VLM, so its before and after are identical and it is not one of the 8.
- The effect is not universal. After fine-tuning, retrieval with the π0 backbone falls from 86.9% to 17.9% on LIBERO (from 56.7% to 2.4% on LIBERO-Plus). On a t-SNE map its key collapses into one cluster per task; its nearest neighbors stay within the same task, but the median progress gap between a frame and the nearest frame of another demonstration of the same task grows from 0.028 to 0.052.
- For some models, retrieval on the fine-tuned backbone alone outperforms the full VLA: 2 of 9 on LIBERO and 4 of 9 on LIBERO-Plus, by up to 48 pt. For example, retrieval on the π0-FAST backbone reaches 74.0% on LIBERO-Plus, against 25.7% for the full model. The authors take this to suggest that the action heads of these models do not exploit all the action-relevant information in the backbone.
The authors also compare the action chunks each VLA produces in its own LIBERO rollouts with the kNN prediction (the weighted mean of the chunks of the 8 nearest demonstration frames of the same task). As a reference, this kNN prediction matches held-out ground-truth chunks with R² = 0.78. Between the VLA chunks and the kNN prediction, R² ranges from 0.63 to 0.93, and the chunks of 5 of the 8 VLAs are closer to the kNN prediction than the ground-truth chunks are. For every VLA, the median squared error to the nearest training chunk of the same task is 0.62–0.99 times the held-out ground-truth reference. X-VLA is excluded from this comparison because it predicts absolute poses. In the authors’ words, these results suggest that on LIBERO the actions of the VLAs remain close to the demonstrated actions.
Appendix D adds an observation that matches the ignored-prompt ablation. Within LIBERO-Goal, only representations whose image tokens can attend to the instruction separate its ten tasks: the bidirectional prefix attention of PaliGemma, used in π0, π0.5 and π0-FAST, and the Florence-2 encoder of X-VLA give 96–100% same-task nearest neighbors. VLMs with causal attention, where the image tokens come before the instruction, reach only 76–82%, and ResNet-18 reaches 79%.
LAC: rating a retrieval key without rollouts
Evaluating one retrieval key in closed loop takes thousands of episodes. As an offline estimate the authors propose Local Action Consistency (LAC). For each held-out query frame, it compares how close the future action chunks of its k nearest neighbors in representation space are to its own chunk, divides that by its average action distance to all entries of the same task, and subtracts the ratio from 1. A higher value means that frames close in representation space are also followed by similar actions. The paper reports LAC@8.
The results come in three parts. Excluding the fine-tuned VLM backbones, which were trained on the held-out demonstrations, LAC tracks closed-loop retrieval success with a Pearson r of 0.78, but its Spearman ρ is only 0.61, so it does not reliably rank keys of similar quality. It does flag the degraded fine-tuned π0 backbone, at 0.55 against 0.76–0.88 for all other keys. A linear projection on the ResNet features, trained with a differentiable surrogate of LAC, raises the success rate of a plain kNN lookup from 88.7% to 93.0%; this is the LAC key variant in the table.
What this means for reading VLA results
In their conclusion the authors recommend reporting retrieval baselines alongside VLA performance, to distinguish benchmark memorization from generalization. The two training-free variants reach 88.7% and 91.7% on LIBERO. On this site’s reading, a VLA whose LIBERO score sits only a few points above that reference line can hardly show generalization beyond the demonstrations by that score alone. Physical AI Lab 1.6 covered how imitation-learning errors carry a robot out of the demonstrations’ coverage; this paper adds the other side: the authors argue that on dense benchmarks such as LIBERO, matching test states to demonstrated ones and reusing their actions recovers a large part of successful behavior. For judging whether two success rates really differ using paired tables and random seeds, see Physical AI Lab 1.8.
Two papers covered earlier on this site are related. RESETTLE retrieves same-task demonstration frames to correct a frozen policy when its samples keep disagreeing, using retrieval as an add-on to the policy; this paper uses retrieval as the policy itself and as a measuring tool. Is Success All You Need? also uses the LIBERO-Plus perturbation benchmark and looks only at the execution quality of successful trajectories; the numbers in this paper are task success rates and say nothing about trajectory quality.
Readers who want to check the results have two starting points. Table 4 in the appendix lists the Hugging Face repository and commit of every VLA checkpoint (for π0.5, lerobot/pi05_libero_finetuned_v044 at revision 8e174154), and the authors state that the evaluation and source code are available at github.com/sereact/lerobot_knn. This site has not opened or run that repository.
Limitations
- This is an arXiv preprint. All numbers are author-reported [R]; this site has not reproduced them or run the authors’ released code.
- The conclusions are limited to LIBERO and related benchmarks. The authors note that their relatively dense demonstration coverage may favor retrieval, and that LIBERO-Plus perturbations largely preserve the underlying task and solution trajectories, so the robustness results mainly concern changes in observations and instructions, not situations that require new trajectories.
- The authors state explicitly that retrieval cannot extrapolate beyond the support of its memory, and that the strong results on these benchmarks should not be read as evidence that kNN policies generalize broadly across robotic problems.
- All nine VLAs are public checkpoints run as released, and some score below the results in their original papers (notably π0 and SmolVLA).
- The kNN hyperparameter search used initial states 40–49 from the official evaluation; the authors’ check on states 0–39 (91.6% against 91.7%) shows little effect. LIBERO-Plus has no reported interval, so gaps of a few points between drops should be read with care.
- All results come from simulation and are task success rates, not real-robot or hardware safety validation.
Links
- How Much of Your VLA Is kNN (arXiv 2611.13160)
- Full paper in HTML
- Physical AI Lab 1.6: Imitation Learning — Working Out How an Error Rolls Out of the Demonstrations’ Coverage, Then Putting the ACT, Diffusion Policy and Data Scaling Numbers Back in Their Own Conditions
- Physical AI Lab 1.8: Evaluation — Splitting a “Net Gain of 2” into a Paired Table, Then Checking the Failure Labels and Random Seeds
- Frontier Radar | RESETTLE: step in only when a frozen policy’s two action samples keep disagreeing, then execute one demonstration-guided correction
- Frontier Radar | Is Success All You Need?: two VLAs with similar success rates can succeed in very different ways



