A hierarchical vision-language-action (VLA) model has a large vision-language planner write an intermediate plan, which a low-level action expert then turns into continuous actions. The preprint Fast Plans, Faithful Actions from MARS Lab at Nanyang Technological University (arXiv 2609.30833, 2026-09-25, evidence level [R]) argues that this division of labour is only worth having if two conditions hold: the plan is generated fast enough, and the executor actually follows it. On a baseline they built themselves, the authors find that neither holds, and they propose a way to measure both plus two changes to the model. For teams building hierarchical VLAs, the most useful part is the intervention tests: a high end-to-end success rate does not show that the interface between planner and executor is doing anything.
What was studied: a waypoint hierarchy adapted from π0.5
The authors adapt π0.5 into a hierarchy they call Waypoint Token-AR. A PaliGemma backbone (a SigLIP encoder plus the Gemma-2B language model) acts as the planner and predicts waypoints in the robot’s configuration space; each waypoint holds a target configuration, a gripper state and an execution duration (a number of action steps). The 300M-parameter action expert inherited from π0.5 is trained with flow matching, receives the current state and the waypoint as suffix tokens, and can also see the image and language prefix through cross-attention. The paper stresses that this baseline was built by the authors for controlled experiments and is not the original π0.5 architecture. Waypoints are extracted offline from demonstrations with the AWE algorithm, and every segment is kept within the action expert’s prediction horizon of D = 32 steps.
Fast: decode by waypoint, not by token
The baseline generates a plan with token-level autoregressive decoding (Token-AR). A seven-waypoint LIBERO plan needs 57 serial backbone passes (one prefix prefill plus 56 value tokens); on the dual-arm robot, with 16 degrees of freedom and 19 tokens per waypoint, it can take up to 120. The paper says that on the real robot, generating a plan takes longer than executing the action segment it specifies, so the arms spend 71% of the running time waiting.
The authors put this down to a mismatch in granularity: the executor consumes a plan one waypoint at a time, while the planner produces it one token at a time. Block-AR treats each waypoint as a block, with bidirectional attention inside a block and causal attention across blocks. Inputs are shifted by a whole block, and one forward pass yields configuration, gripper and duration through three LM-head calls. A maximum-length plan then takes 8 passes (one prefill plus seven waypoints), regardless of how many slots each waypoint has. The planning costs the authors report are below. Plan latency, request round trip and motion duty cycle come from 35 telemetry episodes on a Rokae dual-arm robot; the end-of-plan metrics come from 2,172 held-out robot planning windows (270 of them true plan endings); the maximum number of serial passes follows from how decoding is structured and is not a measurement.
| Metric | Token-AR | Block-AR |
|---|---|---|
| Max serial passes per plan (LIBERO / robot) | 57 / 120 | 8 / 8 |
| Plan latency | 1094 ms | 125 ms |
| Request round trip (including the action expert) | 1266 ms | 293 ms |
| Motion duty cycle | 28.7% | 65.6% |
| End-of-plan recall / precision | 0.00 / – | 0.61 / 0.85 |
With plan latency cut to 1/8.7, most of the remaining latency comes from the action expert (about 160 ms per segment). On those held-out windows, Token-AR never emits the end code d = 0, and Block-AR’s joint-angle error on the first waypoint falls from 0.156 rad to 0.112 rad, a 28% reduction. The cost shows up on LIBERO: with Block-AR, the four-suite average success rate is 0.85 points below Token-AR, with per-suite differences between +0.6 and −1.4 points.
Faithful: testing whether the executor looks at the plan
The paper defines two complementary measures. Both change only the waypoint while images, language, state and flow noise are held fixed:
- Sensitivity S: the action change caused by altering the waypoint, divided by the action change caused by re-sampling the action expert’s own noise once (computed as RMS, then the median over segments).
- Utility U: how many points task success drops when the spatial endpoint is erased at test time (the target configuration is set to the segment’s start state); U > 0 means erasure hurts.
On 64 fixed LIBERO-Long action segments, Block-AR with suffix-token conditioning changes its actions by only 0.6% of the noise baseline when the endpoint is erased, and erasing every endpoint in closed loop causes no drop in success (U = 0.0). This is not because the action expert ignores all its inputs: swapping the main camera image changes the actions by 281% of the noise baseline, swapping the language instruction by 158%, and adding 4 steps to the duration by 19%. The authors’ explanation is that the action expert also sees the key/value representations of images and language, and empirical risk minimization does not force it to decide its actions through the waypoint.
The fix: normalized goal modulation (NGM) and the “displacement-integrator shortcut”
The obvious fix is to give the goal a deep path into every layer of the action expert. Without constraints (naive goal injection), the authors find it overshoots the other way. Note that in the paper this variant is built on Token-AR, not Block-AR, so it is not a controlled ablation of the Block-AR + suffix system above. The reason for the overshoot: in the demonstrations, the relative goal q − s roughly equals the integral of the very actions the expert is learning, so feeding it to every layer leaks the label into the condition: the model can fit the training loss by integrating the displacement without looking at the images. The authors call this the displacement-integrator shortcut. Endpoint sensitivity jumps to 348%, the response to image changes drops to 27% of the suffix baseline, LIBERO-Long success falls to 15%, and every failure is a timeout. High sensitivity is therefore not proof that the plan is used well; it has to be read together with success under the intact plan and the response to other inputs.
NGM keeps the deep path and works as follows (the first item is how the deep path is built; the other two are constraints):
- Input and injection: the displacement q − s is normalized per dimension with a robust scale, concatenated with a learnable gripper embedding, and passed through an MLP with a zero-initialized output projection; the resulting residual is added to the AdaRMS condition of every layer. At initialization the model behaves exactly as before.
- Phase gating: the path is open only in the coarse stage of denoising (fully open for t ≥ 0.5, closed for t ≤ 0.2), leaving the final refinement to what the action expert sees.
- Anti-shortcut training: during training the endpoint gets Gaussian noise (σc = 0.7 in the normalized domain), and with probability 0.15 it is replaced by a learnable null embedding. Both perturbations apply to the suffix tokens and the deep path at once; otherwise an unperturbed suffix would carry the label straight in. At inference, noise and dropout are off, the gate stays on, and classifier-free guidance is not used.
| System (LIBERO-Long) | S endpoint | S duration | Image response retained | Success, intact plan | U (points) |
|---|---|---|---|---|---|
| Block-AR + suffix conditioning | 0.6% | 19% | 100% | 91.0% | 0.0 |
| Token-AR + naive injection | 348% | 106% | 27% | 15% | +14 |
| Block-AR + NGM | 42.6% | 26.4% | 82% | 96.2% | +7.4 |
With NGM, erasing the endpoint lowers success by 7.4 points: the executor now depends on the plan while keeping most of its response to vision. The four-suite LIBERO average rises from 95.85% to 98.45%, with gains on every suite. For reference, the paper quotes a π0.5 average of 96.85% from the OpenPI LIBERO evaluation README; that row is a cited value, not a result the authors re-ran under the same training setup.
Real-robot results, and how this relates to earlier articles on this site
The real-robot study has three bimanual tasks: placing a pepper and a banana into two boxes among distractors, swapping a red and a green pepper between two plates, and colour-matched shelf placement. Each method ran 20 trials per task, judged by an operator: Token-AR 48/60, Block-AR 46/60, Block-AR + NGM 51/60. The authors themselves describe the success rates as “comparable”, and this sample size does not support conclusions about a difference of a few successes. The gain that can be confirmed on the robot is in time: at comparable success, the fraction of time the arms are moving more than doubles.
This site’s Frontier Radar deep dive | Edge and real-time: where the latency budget of VLAs and world action models goes walked through the latency budgets of several sources in milliseconds; this paper adds the planner-decoding step of a hierarchical system and measures it with the motion duty cycle, a metric you can observe directly on a robot. Frontier Radar deep dive | Simulation and evaluation: what VLA evaluation is missing beyond initial-state success rate covered evaluation dimensions beyond success rate; the intervention tests here add another: only by changing the plan while holding the other inputs fixed can you check whether the plan is used at all. For the basic idea behind VLAs, see the course article Physical AI Lab 0.2: Why Now? The Three Curves of Data, Models, and Compute.
Limitations
- This is an arXiv preprint. All figures above are reported by the authors; the material this article is based on contains no independent reproduction and no link to a code repository.
- Block-AR and NGM were designed for and validated on a single hierarchy derived from π0.5: Block-AR assumes fixed-format waypoint token groups, and NGM relies on the AdaRMS conditioning of the π0.5 action expert. The authors state explicitly that transfer to other VLA backbones is untested.
- Each model variant was trained only once, so run-to-run variability cannot be estimated; on LIBERO, one point in a suite corresponds to 5 of its 500 episodes.
- The real-robot study covers only 3 tasks with 20 trials per task and method; the authors also note that this limits the precision of the success-rate estimates and conclusions about small differences.
- In simulation, a new plan is requested only after the whole plan has been executed; on the robot, as soon as the first segment is done. The duty-cycle figures depend on this setup.
- “Duration within the prediction horizon” only guarantees that the requested action segment fits into the action expert’s output length (at most 1.07 s at 30 Hz on the robot, 1.6 s at 20 Hz in LIBERO); it does not mean the motion is collision-free or dynamically feasible.
Links
- arXiv 2609.30833: Fast Plans, Faithful Actions: Closing the Planning–Execution Gap in Hierarchical Vision-Language-Action Models (authors: Chuanliang Xie, Boyu Ma, Gen Li, Yizhou Liu, Houwang Chen, Xinyu Zhou, Jianfei Yang)
- Full text (HTML)
