
Fine-tuning a flow-matching VLA with GRPO in online reinforcement learning carries two costs. Every action chunk in a rollout needs a multi-step ODE integration, and because sampling randomness comes only from the initial noise, all rollouts in a group often get the same return, so the group yields no gradient. arXiv 2611.20460 (2026-11-16, Imperial College London) proposes VLA-SSD. It self-distills π0.5’s flow-matching action head into a stochastic policy that produces an action in one step, takes its randomness from a Brownian path fed to the network, and derives a conditional action density that can be approximated cheaply enough for GRPO’s likelihood ratio. The authors report 93.0% LIBERO success after GRPO, 15.9 percentage points above the SFT checkpoint, at 7.6 ms per action-head forward pass per chunk, against 91.5% and 24.5 ms for πRL’s four-step GRPO; the share of groups with identical returns falls from 62.5% to 35.4%. All numbers below are reported by the authors [R] and come from simulation. This site has not run the code or reproduced any result.
Two bottlenecks: multi-step integration and zero-variance groups
GRPO trains no value network. It samples N trajectories from the same task and initial state and uses the group-normalized return as the advantage at every step: A_i = (R_i − R̄) / (std(R) + κ). The paper’s LIBERO and MetaWorld tasks use a binary reward: the return of an episode is 1 on success and 0 otherwise. If every trajectory in a group succeeds or every one fails, all advantages are 0. The paper calls these groups degenerate: they consume rollout compute and contribute no gradient.
Given the initial noise a0 and the state, a flow-matching policy is deterministic, so action differences within a group can only come from different draws of a0. The authors list two existing remedies: an external noise network, or a return to multi-step samplers that inject noise at every integration step, such as πRL’s Flow-SDE. The latter keeps the number of forward passes per chunk at K. Table 1 of the paper gives the action head’s forward time per chunk on one A100: 58.4 ms for the ODE sampler of the π0.5 SFT checkpoint at K = 10, 24.5 ms for πRL’s SDE sampler at K = 4, and 7.6 ms for one-step VLA-SSD.
Method: a one-step map that takes a Brownian path as input
- Base model: π0.5, 3.3B parameters in total, whose 300M-parameter action expert is the flow-matching head. The vision-language backbone (SigLIP and Gemma-2B) stays frozen throughout. Two two-layer MLP pathways are added to the action expert, one taking the time interval t − r and one the Brownian path increment. Their output layers are zero-initialised, so at the start the model reproduces the pretrained velocity field.
- Randomness: each action coordinate carries a standard Brownian motion represented by its first L = 5 Karhunen–Loève modes, with standard normal coefficients ξ. The noise scale is σ = 0.1 and the diffusion profile is β(t) = 2(1 − t). The one-step action is a1 = a0 + G(a0; W | s) + m, where m is the martingale increment from the Brownian path. At σ = 0 it reduces to a deterministic flow map. The construction follows the Itô maps of Pan et al.
- Self-distillation: there is no external teacher. Training starts from the pretrained velocity field and uses a diagonal loss plus Lagrangian self-distillation (LSD). The authors did not use the Eulerian objective (ESD) because it does not identify the Itô map, nor the progressive one (PSD) because it needs two sequential forward passes. Distillation runs 1,000 steps on 4 A100 GPUs, and the checkpoint at step 600 is used.
- Likelihood: the Brownian coefficients are split into the component that determines m and an orthogonal remainder. The remainder and a0 form an auxiliary variable h, which is stored during rollout and reused when rescoring; the Brownian path is never resampled. Assuming the Jacobian’s spectral norm is below 1, the density conditioned on h has a closed form (Proposition 1). In practice one fixed-point iteration approximates the inverse map, log det is truncated to its first-order trace, and a single Hutchinson probe estimates that trace, which costs one extra Jacobian-vector product. The density is computed only on the 5 steps × 7 dimensions = 35 coordinates actually executed. The approaches the authors contrast with, such as πRL and ReinFlow, compute likelihood ratios over K intermediate denoising steps.
The authors checked this approximation. At early, middle and late GRPO checkpoints they replayed 8 LIBERO action chunks each and compared the approximate FP32 log-ratios with reference values from a converged inverse and an explicit Jacobian; the mean absolute error was below 0.0002 at all three.
Main results: one-step GRPO against four-step baselines
All runs start from the SFT checkpoints released with πRL. On LIBERO, RL starts from the few-shot checkpoint (77.1%), as in πRL. GRPO uses groups of N = 8 rollouts and trains in RLinf on 4 A100 GPUs. Table 1 of the paper (success %, deterministic evaluation):
| Policy | Sampler | Passes per chunk | Forward (ms) | LIBERO | MetaWorld |
|---|---|---|---|---|---|
| π0.5 SFT | ODE, K = 10 | 10 | 58.4 | 77.1 | 51.8 |
| πRL, GRPO | SDE, K = 4 | 4 | 24.5 | 91.5 | — |
| πRL, PPO | SDE, K = 4 | 4 | 24.5 | 98.3 | 70.7 |
| Itô student (VLA-SSD) | Itô, K = 1 | 1 | 7.6 | 68.6 | 49.0 |
| Itô + GRPO (VLA-SSD) | Itô, K = 1 | 1 | 7.6 | 93.0 | 72.0 |
Points to keep in mind when reading the table:
- The two πRL rows are numbers the πRL paper published on the same checkpoints and protocol; the authors of this paper did not rerun them.
- Each of the four LIBERO suites is evaluated over 500 episodes, so the mean covers 2,000 episodes; the authors give its standard error as 0.5–0.7 points. 93.0% and 91.5% differ by 1.5 points. The introduction (including its contribution list) and the conclusion describe this result as matching four-step GRPO, while Section 4.2 says it outperforms it. MetaWorld MT50 is evaluated on 50 tasks × 10 fixed reset states, and the paper gives no uncertainty for it.
- πRL’s four-step PPO is higher on LIBERO (98.3%); the authors note that it needs an additional vision-language critic. On MetaWorld, VLA-SSD’s 72.0% is above four-step PPO’s 70.7%; πRL’s GRPO has no MetaWorld number.
- The “3.2× faster per chunk” during training rollouts is the ratio of 24.5 ms to 7.6 ms. Before RL, the one-step student is below the SFT checkpoint: 68.6% versus 77.1% on LIBERO, 49.0% versus 51.8% on MetaWorld.
The relative figures in the abstract come from different teachers and checkpoints. The “20.6% relative improvement” is 15.9 points divided by the SFT checkpoint’s 77.1% (checked by this site); counted from the one-step student’s 68.6%, GRPO adds 24.4 points. The “7.7× faster action head” is the ratio of 58.4 ms to 7.6 ms. The “5.5% relative decrease in LIBERO success” corresponds to Table 4, where the π0 teacher’s 94.15% drops to the Itô student’s 88.95% (checked by this site). The introduction’s “0.3-point drop” is a different pair: distillation from the full-demonstration π0.5 checkpoint, 96.85% to 96.55%. These numbers cannot be merged into one statement.
Zero-variance groups: does sample diversity become a learning signal?
Table 2 of the paper compares five samplers. Multi-step samplers use the SFT checkpoint and one-step samplers use its distilled student. Their success rates are close (83.1%–84.6%), but the share of groups whose returns are all identical differs:
- π0.5 ODE sampler (K = 10): 62.5%;
- deterministic flow map (K = 1): 56.2%;
- deterministic map with isotropic output noise N(0, 0.1²I): 47.9%;
- Flow-SDE (K = 4): 45.8%;
- Itô map (K = 1, σ = 0.1): 35.4%.
The authors also measure action diversity by effective rank. With a0 fixed, the effective rank of the ODE sampler and the deterministic map drops to 0.0; the Itô map has 18.5 with a0 fixed and 18.2 with it free, showing that diversity can come from the Brownian path alone. Isotropic output noise also has an effective rank of 18.5, yet 12.5 more points of its groups have zero variance (this site’s subtraction from the table). From this the authors infer (their word is “suggests”) that the Itô map’s diversity turns into within-group return variation more effectively. Table 2 gives no group count or interval.
The training-time comparison is in Table 3. All runs start from the same distilled checkpoint and are evaluated with exploration noise turned off after 100 GRPO optimizer updates:
- Noise scale: at σ = 0.05, 0.10 and 0.20, LIBERO reaches 69.10%, 73.50% and 65.15%, and MT50 58.80%, 59.40% and 55.20%. The default 0.10 is the highest of the three values tested.
- Noise source: with the same GRPO settings, disabling the Brownian-path input and adding isotropic Gaussian noise to the output gives 71.50% on LIBERO and 59.00% on MT50, so the Itô policy leads by 2.0 and 0.4 points. The paper gives no uncertainty for the 0.4-point gap on MT50.
Exploration noise also has to leave the noisy policy successful often enough for groups to contain both outcomes. On MetaWorld, the Itô student succeeds 36.8% of the time under the training noise σ = 0.1 against 49.0% deterministically, a retention of 75%; after GRPO the figures are 55.4% and 72.0%, a retention of 77%. For comparison the authors cite πRL’s report that Flow-SDE retains only 14.7% of its deterministic success at K = 1 and 86% at K = 4, which is why πRL trains at four steps. In the implementation, groups whose mean return lies outside [0.1, 0.9] are discarded before the update.
For the overall RL post-training workflow and for judging an improvement with paired intervals across training seeds, see Physical AI Lab 3.3. The paper text read here gives no multi-seed GRPO results; for how standard error relates to the number of evaluation episodes, compare the 1/√n calculation in Physical AI Lab 2.8.
Where distillation costs accuracy
Table 4 distills from a π0 teacher (πRL’s full-dataset SFT checkpoint, K = 10) on LIBERO. All students are evaluated at K = 1 over 500 episodes per suite, and the standard error on each mean is about 0.7 points:
- Teacher mean 94.15%, long-horizon suite 85.2%.
- Deterministic flow map: LSD 91.30%, PSD 90.80%, ESD 67.50%.
- Itô map (σ = 0.1): LSD 88.95%, PSD 87.95%, ESD 65.50%; the long-horizon suite under LSD is 77.0%.
Under LSD, the stochastic form is 2.35 points below the deterministic map, which the authors describe as the performance trade-off of the stochastic formulation. The weaker the teacher, the larger the distillation loss: 8.5 points from the few-shot LIBERO checkpoint (77.1% → 68.6%) and 2.8 points on MetaWorld (51.8% → 49.0%). The authors also report that, with sampling depth fixed at one step, self-distillation raises pre-RL success from 40.55% to 68.60%, and that at a matched budget, distilling from the converged SFT checkpoint instead of jointly from the π0.5 base raises a K = 1 student from 12.2% to 42.7%. The same checkpoint evaluated at K = 1 and K = 5 varies by less than ±2 points.
Limitations
- This is an arXiv preprint. All numbers are reported by the authors [R] and have not been independently reproduced. The experiments are confined to LIBERO and MetaWorld MT50 in simulation, with no real-robot experiment. The authors expect the 3.2× rollout speed-up to be particularly valuable for real-world deployment and name real-robot fine-tuning as the next step; this is the authors’ expectation, not a measurement.
- The conditional density is an approximation: the authors state that it relies on a first-order truncation of the log-determinant. The paper’s density diagnostics show that at the end of distillation the Brownian pathway’s output projection is of order 3 × 10⁻⁴ and that the likelihood ratio at the first minibatch equals one to floating-point precision. These values were measured only at the start of training; the pathway norm was not logged during RL. The log-ratio error check used 8 action chunks per checkpoint.
- The πRL GRPO and PPO numbers are taken from the πRL paper. The LIBERO means come with the authors’ standard error; Table 2, Table 3 and the MetaWorld results have no intervals, standard deviations or multi-seed results.
- On LIBERO, the GRPO version remains below four-step PPO with a critic (93.0% versus 98.3%), which the authors state among their limitations.
- The paper text this site read ends partway through Appendix D; Appendix E (the one-step sampling comparison) and Appendix F (the ReinFlow-style PPO baseline) were not available.
- The paper gives a GitHub address for its code. This site did not open it and has not checked the code or its license.
Links
- Stochastic Self-Distillation for Flow-Based Vision-Language-Action Models (arXiv 2611.20460)
- Full paper in HTML
- Physical AI Lab 3.3: RL Post-Training — Putting HIL-SERL’s “2x” and RECAP’s “Doubled Throughput” Back in Their Conditions, Then Recomputing the Paired Interval over Three Seeds
- Physical AI Lab 2.8: Reproducible Evaluation — Working Out Wald and Wilson Intervals by Hand, Sizing a Run from 1/√n, Then Bounding the Failure Rate after Zero Failures
- Physical AI Lab: a physical AI course where every number can be checked



