This article is based on the September 2026 archive of the Physical AI Radar. The radar’s “Training and self-improvement” track collected 9 items in September. Of these, 5 were about LLM training or unrelated to robotics (they matched keywords but have nothing to do with embodied AI), and 1 was a media video roundup. This article looks at the remaining 3 preprints together with 4 related items from other tracks. The conclusion: VLA post-training is turning from “SFT plus a bit of RL” into a staged pipeline, research focus is shifting from offline training to feeding data back after deployment, and this in turn exposes new risks of online updates.
Starting point: the cost of SFT now fits in a budget spreadsheet
AWS’s π0 fine-tuning tutorial ([O], 2026-09-10) gives a set of numbers you can use directly: the 3B-parameter π0, trained for 20,000 steps with FSDP FULL_SHARD on a single ml.p5.48xlarge (8×H100 80GB) with an effective batch size of 32, took 6 hours 15 minutes on DROID and 6 hours 11 minutes on LIBERO, and a single checkpoint is about 45 GB. In other words, adapting an open-source foundation model to your own data is now a single-node, half-day job. The later stages are much harder.
Step two: RL post-training, in simulation and on real robots
Z-1 ([R], 2026-06-30; a “frontier baseline” item hand-curated for the radar, not new work from September) is the simulation-side reference point. It uses π0.5 as the base model, does SFT only on public RoboCasa demonstration data, and then runs GRPO task by task on 24 standard RoboCasa tasks. To make online optimization more efficient and stable, the authors combine shared-prefix rollout construction, tree-structured trajectory branching, completion-aware reward calibration, and selective joint training of the VLM and the action expert. The final average success rate is 80.6%, 13.2 percentage points above the SFT initialization, without any private demonstrations.
VLA post-training for high-DoF dexterous hands ([R], 2026-09-17) represents the real-robot side, with a four-step pipeline. Step one learns a temporal hand-action codec so that an open-source VLA can output absolute dexterous-hand commands. Step two is SFT. Step three is DAgger: when a human takes over, mismatched hand poses cause command jumps that contaminate the correction trajectories, which the authors handle with buffer rollback, pose alignment and smooth command blending. Step four runs residual RL on the real robot, restricting exploration to the coordinated hand motions the codec can represent, to avoid inefficient exploration in raw joint space. On 5 real-robot tasks (spanning bimanual handover, in-hand reorientation and tool use), each tested 20 times, every trial succeeded.
What the two have in common: RL does not explore from scratch in the raw action space. SFT or a codec first narrows the search space, and RL improves within it.
Step three: learning from deployment experience
Once robots are deployed, they naturally accumulate a mix of successful, partially completed and failed trajectories. PACL ([R], 2026-09-24) addresses exactly this setting: learning only from autonomous rollouts, with no extra human corrections and no exploratory interaction. Using these trajectories directly for imitation learning reinforces wrong behavior, while offline RL often misestimates values because rewards are sparse and data coverage is limited. PACL trains a chunk-level critic, adding future latent-state prediction as auxiliary supervision in TD learning; it then discretizes chunk Q-values into quality conditions that guide a diffusion actor to treat trajectories of different quality differently; at inference, the actor generates several candidate action chunks and the critic picks the one with the highest value. The authors report that PACL consistently improves the pretrained policy in both simulation and the real world and outperforms imitation learning and offline RL baselines, but the abstract gives no specific numbers.
Using Non-Expert Data to Robustify Imitation Learning via Offline RL by TRI and others ([R], paper published 2025-10, collected by the radar in September) is earlier work in the same direction. It points out that under the sparse coverage common in the real world, standard offline RL struggles to make use of non-expert data (play data, suboptimal demonstrations, partially completed trajectories); with a few simple algorithmic changes that broaden the support of the policy distribution, this data becomes usable and noticeably widens the range of initial conditions from which the policy can succeed. This paper’s abstract likewise gives no specific numbers.
Another related example is PASSAGE ([R], humanoid traversal, 2026-09-16): with the lower-level tracker frozen, it applies RL post-training only to the upper-level planner to improve closed-loop performance, i.e. RL on just one layer of a hierarchical system.
The risk: online RL can quietly destroy individual tasks
“Continual learning after deployment” sounds natural, but this preprint on task collapse ([R], 2026-09-23) provides counter-evidence. The authors run PPO (with SDE sampling) on LIBERO-10 with the 450M-parameter SmolVLA and compare three kinds of exploration noise under the same small compute budget:
| Exploration noise | Seeds with task collapse |
|---|---|
| Fixed noise | 2 of 3 |
| ReinFlow-style learned noise | Every seed tested up to iteration 200 |
| Uncertainty-gated controller | 0 of 3 |
The hallmark of task collapse is that the overall average score looks normal while the capability on individual tasks is already gone. The gated controller reallocates exploration across task streams based on task-agnostic novelty and competence signals, without needing task labels. The authors are careful in their claims: within this budget no configuration beat the behavior cloning baseline, and lowering fixed noise only slows the decline without stopping it. There is also an engineering detail worth noting: when training in bfloat16 following the reference recipe without an fp32 master copy, 96.02% of the action expert’s parameter elements stayed bit-for-bit unchanged across three consecutive iterations; with an fp32 master copy and the reference learning rate, both compared arms collapsed in a single-seed observation. The authors state explicitly that this is only a phenomenon measured alongside the main results and is not offered as a causal explanation.
Limitations
- Real-robot evidence comes from small samples throughout: the high-DoF dexterous hand work tested each task 20 times, and its results are qualified as “within the reported post-training budget”; the abstracts of the PACL and TRI papers give no numbers.
- Z-1’s 80.6% and the task collapse findings both come from simulation (RoboCasa, LIBERO-10), and the latter also used a small model and a small compute budget.
- All [R] results are self-reported by the authors; the training times in the AWS tutorial apply only to the instance type and configuration given there.
What to watch next
- Whether work on feeding deployment data back (the PACL kind) can deliver reproducible real-robot gains, and how many hours of deployment it takes before the effect shows.
- Whether online RL fine-tuning adopts per-task monitoring as standard practice, rather than looking only at overall success rate.
- Whether the near-zero parameter updates under bf16 training reproduce on larger VLAs and with other RL recipes.
Links
- Physical AI Radar
- Fine-Tuning π0 on SageMaker HyperPod EKS (AWS)
- Z-1 (arXiv 2606.31846)
- Towards High-DoF Dexterous Manipulation through VLA Post-Training (arXiv 2609.19666)
- PACL: Learning from Mixed-Quality Deployment Experience (arXiv 2609.29000)
- Using Non-Expert Data to Robustify Imitation Learning (arXiv 2510.19495)
- PASSAGE (arXiv 2609.18732)
- Uncertainty-Gated Exploration Noise (arXiv 2609.28838)
