VLA models can turn a language instruction into a broadly correct manipulation, but in contact-rich phases such as pressing a button, pushing a block or stacking cups, a small error in contact location or alignment is enough to fail the task. VLaRL, a preprint from Microsoft Research Asia in Tokyo (arXiv 2609.30868, 2026-09-25, evidence level [R]), keeps the VLA frozen and learns only a correction added to its output (a residual policy). All reinforcement learning happens in simulation; on the real robot there is no real-world RL and no online adaptation. What makes the paper worth reading is the interface it chooses for crossing the visual gap between simulation and reality: not images and not object poses, but the vision-language latent the VLA already computes internally, plus a lightweight mapper that pushes simulated latents toward the real distribution.
Why a frozen VLA plus residual RL in simulation
The authors start from the observation that a VLA often produces a sensible nominal action and still fails during physical execution. Fixing that with more imitation learning means collecting demonstrations that capture the corrections and updating the base policy. Residual RL keeps the VLA’s nominal behaviour and learns only the correction, which can be optimised directly from task rewards. Repeated trial and error on a physical robot is costly and raises safety concerns, so the authors train the residual policy entirely in simulation.
The executed action is the nominal action plus a scaled correction:
a_t = a_t^VLA + α · a_t^RL # α = 0.10, shared across tasks
a_t^RL = π_RL(z_t, a_t^VLA, s_t, f_t)The residual policy has four inputs: the VLM latent z_t, the VLA’s nominal action, the proprioceptive state s_t (end-effector position and roll–pitch–yaw orientation, gripper state) and the 3-dimensional Cartesian force f_t measured at the wrist. For both backbones the action is a 7-dimensional normalised Cartesian action (three translations, three rotations, one gripper command); the residual actor outputs a tanh action of the same size and is initialised to produce near-zero corrections.
Choosing the interface: images, poses or the VLM latent
The paper sorts existing sim-to-real interfaces into two kinds. Image-based policies keep rich scene information but are directly exposed to the visual gap; geometric states such as object poses are compact but need a separate perception pipeline and a task-specific state definition. The authors’ own earlier object-centric residual RL used estimated 6-DoF object poses as the interface.
VLaRL instead uses the token matrix produced by the frozen VLA’s own vision-language module (S tokens of dimension D), mean-pools the tokens and hands the resulting D-dimensional vector to the residual policy. For Flower this means language-conditioned tokens from its Florence-2-based vision-language module, 1024-dimensional after pooling; for GR00T N1.7, its language-conditioned image tokens, 2048-dimensional. The authors argue that this representation already abstracts away some pixel detail while keeping the task-relevant visual and instruction context. In the same scene, a different instructed target or object role can call for a different correction, and the nominal action alone does not expose that information.
The authors also point out that the representation is not fully invariant to the visual gap: for similar task states, simulated and real images still yield different latent distributions. That is what the mapper is for.
The four-step workflow and the mapper
- Fine-tune and freeze the VLA, rebuild a digital twin: for each task, 32 real-robot demonstrations are collected by teleoperation with a 3D mouse (Franka Research 3), the VLA is fine-tuned on them and then frozen. The same demonstrations and FoundationPose are used to reconstruct object positions and the environment, giving an approximately corresponding task configuration in MuJoCo.
- Train the mapper: a 4-layer Transformer encoder with 16 attention heads that operates on the simulated token sequence before pooling. It has a learnable residual connection whose scaling factor starts at 0, so it initially acts as the identity. Because simulated and real trajectories correspond only approximately, with no frame-by-frame pairing, the authors align distributions with entropy-regularised optimal transport (Sinkhorn divergence). The cost is the squared Frobenius norm of the difference between mapped simulated tokens and real tokens (divided by S·D), plus λτ times the squared difference in normalised trajectory progress. Hyperparameters: λτ = 0.5, ε = 0.05, 50 Sinkhorn iterations.
- Train the residual in simulation: with the mapper frozen, the residual is trained with TD3 for 60,000 environment steps. The mapped tokens serve two purposes: they go into the frozen VLA action head, reducing the representation shift that a VLA fine-tuned on real data sees on simulated images, and, after pooling, into the residual actor. The critic additionally receives privileged task state available in simulation, while the actor uses only observations available at deployment (an asymmetric actor–critic). Residual targets built from the demonstrations, (a_demo − a_VLA) / α, serve as an auxiliary regularisation term.
- Deploy on the robot: tokens from real images are mean-pooled and passed straight to the residual policy, without the mapper; all models stay frozen.
Results reported by the authors
Each of the four tasks probes a different local-control difficulty: button pressing (precise contact), block pushing (sustained contact), cup stacking (grasping plus spatial alignment) and drawer closing (articulated contact). Initial object positions are randomised by ±3 cm and orientations by ±15°; the drawer opening by ±3 cm. Simulation results use 3 random seeds; each real-robot condition is evaluated over 40 trials. The paper states that VLaRL improves real-robot success in all 8 task–backbone combinations. The combinations for which the text gives numbers are listed below (the rest appear only in bar charts and are not restated here):
| Condition (real robot, 40 trials each) | Frozen base VLA | VLaRL |
|---|---|---|
| Flower, button pressing | 67.5% | 100.0% |
| Flower, block pushing | 22.5% | 50.0% |
| GR00T N1.7, cup stacking | 17.5% | 45.0% |
| Unseen objects, block pushing (star, triangle) | 40.0% (16/40) | 57.5% (23/40) |
| Unseen objects, cup stacking (sky-blue and green cups) | 55.0% (22/40) | 62.5% (25/40) |
The unseen-object experiments involved no additional residual RL or real-world adaptation. A simulation example is Flower block pushing: 32.5% for the base VLA, 42.5% with the mapper, 67.5% with the full VLaRL. The mapper alone raises nominal success in simulation, which is consistent with its purpose of reducing representation shift.
The authors also report cases where simulation and real-robot trends diverge: Flower’s base success on cup stacking is clearly higher on the real robot than in simulation, and the residual adds only a small real-world improvement. Their explanation is that latent alignment does not remove other sim-to-real differences, such as contact dynamics and the distribution of VLA actions.
Ablations: what the mapper and the latent each contribute
Both controlled ablations were run on Flower only:
- Without the mapper (residual trained on unaligned simulated latents): block pushing and cup stacking fail in all 40 real-robot trials, against 50.0% and 70.0% for the full VLaRL. In other words, a residual trained on raw simulated latents does not transfer to real latents.
- Without the VLM latent (nominal action, proprioception and force retained): real-robot success drops on all four tasks, for example from 100.0% to 70.0% on button pressing and from 50.0% to 20.0% on block pushing.
The authors also use integrated gradients to attribute the residual-action norm to its inputs. Across all tasks and both backbones, the VLM latent receives the largest share: 72.1%–89.7% for Flower and 84.0%–90.9% for GR00T N1.7. The share of the physical inputs varies by task: force rises for block pushing, which needs sustained contact, and proprioception matters more for cup stacking, which needs precise position adjustment. Attribution shows how much the network relies on each input; on its own it is not causal proof, and the authors present it together with the ablation results.
How this relates to the course and earlier articles on this site
Frontier Radar deep dive | Training and self-improvement: three stages of VLA post-training and task collapse in online RL traced the path from SFT to RL post-training in simulation and on real robots. VLaRL offers a different division of labour: the VLA is fine-tuned once with supervision and its weights never change afterwards, RL optimises only a small residual network, and none of it runs on the real robot.
The course syllabus plans an L2 episode on domain randomisation and system identification in sim2real. VLaRL targets the visual gap, replacing pixel-level correspondence with alignment of latent distributions; the authors state explicitly that the mapper does not deal with contact dynamics, friction or compliance, so that part of the gap still has to be handled separately. GR00T N1.7, one of the backbones here, was introduced in Physical AI Lab 0.2: Why Now? The Three Curves of Data, Models, and Compute; this paper runs it through the same pipeline as the structurally very different Flower.
Limitations
- This is an arXiv preprint and all figures above are reported by the authors; the material this article is based on contains no independent reproduction and no link to a code repository.
- Training the mapper requires approximately corresponding simulated and real trajectories, which means a task-specific digital-twin reconstruction. This is needed only for training, but the authors acknowledge that it limits how easily the approach scales.
- The mapper addresses the shift that the visual gap causes in the VLM representation. It does not explicitly address contact dynamics, friction, compliance or other physical differences, which the authors suggest may explain the discrepancies between simulation and real-robot results.
- The mapper is specific to each VLA backbone, and the residual policy is currently trained separately for each task category.
- The unseen-object evaluation covers only a small range of object variation, and the authors state explicitly that it does not establish open-world generalisation.
- Both ablations were run on Flower only. Each real-robot condition has 40 trials and the text gives no confidence intervals or significance tests; for the unseen cups, for instance, the difference is 22/40 versus 25/40, three successes.
Links
- arXiv 2609.30868: VLaRL: Augmenting Vision-Language-Action Models with Simulation-Trained Latent-Conditioned Residual RL (authors: Namiko Saito, Kinam Kim, Heecheol Kim, Katsushi Ikeuchi, Yasuyuki Matsushita)
- Full text (HTML)
