Most VLAs send the current images and the language instruction back through a multi-billion-parameter backbone every time they predict an action chunk, even though the task itself barely changes within an episode. Fast-VLA, a preprint from a Huawei team (arXiv 2610.21389, 2026-10-15), separates the two. At the start of an episode a frozen 4B VLM runs once, encodes the instruction and the initial multi-view images into a 34-layer key–value (KV) task memory, and then leaves the control loop. At every action chunk after that, a lightweight online actor combines the current images and joint state to query this fixed memory and outputs a whole chunk in one forward pass. The authors report success rates of 92.46% / 92.40% on RoboTwin 2.0 Clean / Randomized and 95.95% / 79.11% on LIBERO and LIBERO-Plus; on the same hardware, RoboTwin amortized inference time is 10.9–17.1× lower than three baselines that keep calling their backbone, and online resident memory after the VLM is offloaded is 1.22 GiB. Below: what each speed-up is measured against, whether the actor really depends on that memory, and what “thinking only once” gives up. All numbers are author-reported [R]; this site has not reproduced them, and every experiment in the paper is on simulation benchmarks.
The method: the slow part runs once per episode
At the start of an episode the frozen VLM reads the instruction L and the initial multi-camera views I_0 and produces the task memory M_0. The authors use the 4B VLM of DM0.5 (Dexmal Team) and keep the key and value tensors of all 34 Transformer layers. After this prefill the VLM is offloaded, and the memory is not updated for the rest of the episode. The appendix states that in the RoboTwin configuration this reusable memory accounts for 174.6 MiB of the 1.22 GiB online resident memory.
The online actor consists of visual encoders, query fusion, an action decoder and an action head. In the RoboTwin configuration:
- Live inputs: current head-camera and two wrist-camera images, a short head-camera history (offsets −17, −8 and 0 frames) and the robot state; each camera has its own trainable DINOv2 ViT-B/14 encoder.
- Query fusion: one learned query for each future step τ (0–49) and each arm, formed as a time embedding plus an arm embedding. Each arm’s 7-dimensional proprioceptive state and the CLS token of its wrist view are fused by an MLP into feedback, which is added to the queries through a gated residual and reinjected before every later decoder layer.
- Action decoding: four decoder layers, each applying self-attention (coordinating time steps and the two arms), Cross-Visual attention (reading current and historical image tokens), Cross-Memory attention (retrieving task context from the fixed memory) and a feed-forward network, in that order. Inside Cross-Memory attention, a KV memory adapter learns softmax weights over the 34 memory layers for each decoder layer, initialized to favour the depth-aligned layer.
- Output: a linear action head outputs a 50 × 14 bimanual action chunk (50 steps of absolute joint commands) in one forward pass, without iterative denoising. The single-arm LIBERO configuration outputs 10 × 7 and uses no image history.
Training updates only the actor (including its visual encoders), with an ℓ1 loss on normalized actions. The VLM is frozen, and its KV caches are precomputed offline and reused throughout training. The action decoder and head are trained from scratch, without pretraining on external action data. Trainable parameters are 272.0M for RoboTwin and 184.7M for LIBERO, with roughly 150 and roughly 5 accelerator-hours of training compute.
At inference each episode costs one VLM prefill and then one actor call per action chunk. The authors report amortized latency as the average per environment step, (T_prefill + ⌈H / K⌉ · T_actor) / H, where H is the number of executed steps and K the chunk length, using the observed episode lengths.
What the 10.9–17.1× is measured against
The table below is an excerpt from the paper’s Table 2 (RoboTwin 2.0); it omits five rows, X-VLA, LingBot-VLA, Qwen-VLA, Motus and Flash-WAM, all with mean success below 90%. “Pred./exec.” is the number of steps predicted per call and the number executed before replanning; ms/chunk is the time of one action-generation call; ms/step is amortized over the execution schedule and includes any once-per-episode cost.
| Method | Pred./exec. | ms/chunk | ms/step | GiB | Mean success (%) |
|---|---|---|---|---|---|
| π0 | 50/50 | 72.38 | 1.580 | 6.26 | 62.16 (compilation) |
| π0.5 | 50/50 | 80.51 | 1.758 | 6.47 | 79.75 (compilation) |
| LingBot-VLA 2.0 | 50/25 | 931.29 | 38.892 | 11.98 | 93.16 |
| DM0.5 | 50/50 | 736.10 | 16.069 | 11.68 | 93.45 |
| LingBot-VA | 32/32 | 9553.12 | 317.765 | 29.59 | 92.20 (compilation) |
| Fast-WAM | 32/24 | 564.80 | 24.624 | 23.13 | 91.83 |
| AHA-WAM (ms/chunk excludes a 111.40 ms video prefill every two chunks; ms/step includes it) | 16/16 | 553.27 | 39.291 | 28.21 | 92.80 |
| Faster-WAM (Noah; authors’ own implementation for cost profiling) | 32/24 | 119.94 | 5.229 | 21.34 | 89.17 |
| Faster-WAM (HUST) | 32/28 | 416.12 | 15.704 | 22.27 | 92.52 |
| Fast-VLA | 50/50 | 44.87 | 1.437 | 1.22 | 92.43 |
Fast-VLA’s first chunk takes 153.74 ms (108.87 ms of VLM prefill plus 44.87 ms of actor), and each later chunk 44.87 ms; with a mean RoboTwin episode length of 237.3 steps, this amortizes to 1.437 ms/step. The paper’s own statement is that, among the baselines in the table with at least 90% mean success and measured inference costs, Fast-VLA provides at least 10.9× faster amortized inference. There are six such baselines: LingBot-VLA 2.0, DM0.5, LingBot-VA, Fast-WAM, AHA-WAM and Faster-WAM (HUST). The 10.9–17.1× in the abstract and conclusion uses only three of them (the abstract calls them “three continually conditioned baselines”): 10.9× faster than Faster-WAM (HUST) at 0.09 percentage points lower success; 11.2× faster than DM0.5 at 1.02 points lower, both replanning every 50 steps; and 17.1× faster than Fast-WAM at 0.60 points higher. The other three give larger ratios: from the table’s ms/step, LingBot-VLA 2.0 (38.892), AHA-WAM (39.291) and LingBot-VA (317.765) are about 27×, 27× and 221× Fast-VLA’s value (this site’s arithmetic), so 17.1× is not the upper end among these six baselines. The paper also reports a 3.6× speed-up over Faster-WAM (Noah) with 3.26 points higher success.
The scope of that range matters. π0 and π0.5 in the same table also execute 50 steps per call, at 1.580 and 1.758 ms/step, only about 1.1× and 1.2× slower than Fast-VLA (this site’s arithmetic), but with much lower mean success (62.16% and 79.75%). So “at least 10.9×” applies only to the group of baselines with at least 90% mean success, not to all VLAs.
Measurement conditions are in appendix Table 11: BF16 for the VLM and BF16 autocast with gradients disabled for the actor; batch size 1; 3 warm-up calls followed by 10 timed calls, reporting the mean, with device synchronization before and after each timed call; timing starts from raw uint8 images and includes resizing, normalization and host-to-device transfer; PyTorch 2.7.1+cu128 without torch.compile or TensorRT; HTTP communication excluded. The text this site read says only “identical hardware” and does not name the device.
Two further points need to be kept separate. First, memory: the 1.22 GiB in Fast-VLA’s row is online resident memory after the VLM is offloaded; the peak during prefill, 12.11 GiB, is listed separately in a note, and the text read does not say whether the baselines’ column is peak or resident memory. Second, where success and cost come from: baseline success rates are taken from published reports, benchmark compilations (marked “compilation” in the table) or the authors’ evaluations of released checkpoints, while latency and memory were measured separately by the authors under a common protocol, so the success rate and the cost in one row do not necessarily come from the same runs.
On LIBERO, Fast-VLA reaches 95.95% (standard) and 79.11% (LIBERO-Plus) at 3.975 amortized ms/step and 0.86 GiB resident memory. Compared with Faster-WAM (Noah) and Faster-WAM (HUST), its LIBERO-Plus success is 4.11 and 5.54 points higher with 3.1× and 21.2× faster amortized inference, but its standard LIBERO success is lower (theirs: 98.50% and 99.00%); all three replan every 10 steps. On LIBERO-Plus it trails π0.5 and DM0.5 by 6.59 and 6.79 points. The authors note that both baselines use pretrained action-generating weights, whereas Fast-VLA’s action decoder is trained from scratch; this is the authors’ explanation, and the paper does not test this cause separately.
Does the actor really use this memory?
A fixed memory only counts if the actor actually reads it. The authors replace the memory content at test time while keeping the actor weights fixed, without retraining (Table 4, RoboTwin 2.0):
| Memory condition | Clean (%) | Randomized (%) |
|---|---|---|
| Original memory | 92.46 | 92.40 |
| Swapped language segment | 35.18 | 29.66 |
| Same task, different episode | 6.26 | 8.36 |
| Swapped initial-view segment | 2.64 | 2.62 |
| Different task | 0.62 | 0.52 |
Swapping the initial-view segment hurts more than swapping the language segment, and memory from another episode of the same task leaves success in single digits. The authors conclude that the actor relies on memory matched to the current episode. This site’s reading: the memory carries not only “which task” but also information about this episode’s initial scene. A swap is a test-time intervention on a frozen actor; it shows that the actor’s output depends on the memory content, not which specific information it reads from it.
How the memory is built also matters (Table 5, LIBERO, actor retrained for each variant): with no task memory (constant zero KV), 69.90% / 53.69% (standard / LIBERO-Plus); text-token KV only from image-conditioned memory, 93.20% / 73.84%; last layer only, 94.70% / 73.70%; full 34-layer instruction-plus-image KV, 95.95% / 79.11%.
The online inputs are just as important (LIBERO, each variant retrained): without wrist vision, 65.60% / 33.49%; head camera only, 70.80% / 30.80%; wrist camera only, 87.10% / 68.72%; without proprioceptive state, 94.30% / 72.41%; without the head view, 92.70% / 74.54%. On standard LIBERO, removing wrist vision (65.60%) costs about as much as removing the task memory (69.90%). The authors call wrist vision a key source of closed-loop feedback. These data are in the paper’s Table 6, which the text cites as “Table 7”.
What thinking only once gives up
The authors compare memory-refresh schedules using the last-layer memory variant (Table 8, LIBERO, matched training for each schedule; amortized cost includes a 34.25 ms actor call per chunk and one VLM prefill per memory construction):
| Refresh schedule | VLM calls per episode | Amortized ms/step | Standard (%) | LIBERO-Plus (%) |
|---|---|---|---|---|
| Every action chunk | 17.37 | 12.391 | 96.50 | 72.47 |
| Every 2 chunks | 8.93 | 8.052 | 95.95 | 70.63 |
| Every 4 chunks | 4.74 | 5.898 | 95.95 | 68.24 |
| Once per episode | 1.00 | 3.975 | 94.70 | 73.70 |
Refreshing at every chunk raises standard LIBERO by 1.80 percentage points and lowers LIBERO-Plus by 1.23 points, with 17.37× more VLM calls and 3.12× the amortized time. The authors take this as support for once-per-episode construction as an efficient trade-off between success and VLM computation. This comparison was run only on LIBERO and only with the last-layer memory variant; the paper does not explain why refreshing lowers LIBERO-Plus success.
The memory encoder can also be swapped (Table 9, LIBERO, actor architecture unchanged; memory is the bf16 KV cache stored per episode): PaliGemma v1, 9.8 MB, 95.35% / 73.15%; π0.5, 9.8 MB, 94.90% / 71.48%; Qwen2.5-VL-7B, 31.8 MB, 95.35% / 75.55%; the default DM0.5, 79.7 MB, 95.95% / 79.11%. On standard LIBERO the four differ by less than 1.1 points; the differences are mainly on LIBERO-Plus.
On task length (Table 7, RoboTwin 2.0 grouped by official step budgets), Fast-VLA reaches 94.00%, 90.27% and 90.25% on the short, medium and long groups, which contain 29, 15 and 6 tasks. put_bottles_dustbin has a 1,700-step budget, permitting up to 34 actor calls on the same memory, and reaches 87.00%.
There is also a question of time granularity. The RoboTwin configuration predicts 50 steps and executes all of them before replanning, so the actor reads a new observation only every 50 environment steps; on LIBERO it is every 10 steps. The text read gives no control rate for an environment step, so this cannot be converted to seconds. The limitation the authors state: memory is built at episode start and kept fixed, and changes in task goals or new instructions during an episode remain unexplored; they suggest combining event-triggered memory updates with high-level replanning.
How this relates to existing articles on this site
Physical AI Lab 0.3: See, Think, Act covered the split between a system that “thinks it through slowly” and one that “acts quickly”, and the π0 paper’s robot re-running inference after every 25 executed steps, about every 0.5 s. Fast-VLA pushes the slow system to once per episode, while the actor still replans per chunk: every 50 steps on RoboTwin, every 10 on LIBERO. The rule from 0.3 applies here too: a rate or interval belongs only to the system and configuration that reported it.
Frontier Radar | Fast Plans, Faithful Actions reported a preprint finding that, in a π0.5-derived hierarchical VLA, erasing the plan’s endpoint did not change success, meaning the executor was not really using the upper-level plan. Fast-VLA’s memory swaps address the same kind of question: does the downstream module actually read the context the upper level provides? The two papers use different systems and benchmarks, so their numbers cannot be compared directly, but this kind of intervention test is worth checking in any design where the upper level computes once or rarely.
Frontier Radar deep dive | Edge and real-time traced where the latency budget goes. When reading Fast-VLA’s ms/step, note that it is an amortized value that depends on the predicted/executed steps and episode length; for a single reaction, look at ms/chunk and the 153.74 ms first chunk. The fact that baseline success and cost come from different sources can be checked item by item with the questions from Physical AI Lab 1.8: Evaluation.
Limitations
- This is an arXiv preprint; all numbers are author-reported [R]. The authors are from Huawei Technologies Co., Ltd and the Huawei Central Research Institute. This site has not reproduced any result.
- All experiments in the text read are on simulation benchmarks (RoboTwin 2.0, LIBERO, LIBERO-Plus); there are no real-robot experiments, and simulation results are not a hardware safety validation.
- Fast-VLA builds its memory with DM0.5’s 4B VLM, and DM0.5 itself is one of the baselines.
- Baseline success rates come from published reports, benchmark compilations or the authors’ evaluations of released checkpoints, while latency and memory were measured separately by the authors; the cost for Faster-WAM (Noah) comes from the authors’ own implementation. The text read does not name the measurement device and gives no confidence intervals.
- Amortized latency depends on episode length and executed steps; 1.22 GiB is resident memory after offloading the VLM, and the prefill peak is 12.11 GiB.
- Memory comes from the initial observation and stays fixed within an episode; the ablations on memory construction, online inputs and refresh frequency were run only on LIBERO, and the refresh comparison uses the last-layer memory variant.
- The paper’s text cites the online-input ablation as “Table 7”, but the data are in Table 6; this article cites the table data.
Links
- Fast-VLA: You Only Think Once for Robot Manipulation (arXiv 2610.21389)
- Full paper in HTML
- Physical AI Lab 0.3: See, Think, Act — Putting Each Rate Back in Its Own System
- Physical AI Lab 1.8: Evaluation — Splitting a “Net Gain of 2” into a Paired Table, Then Checking the Failure Labels and Random Seeds
- Frontier Radar | Fast Plans, Faithful Actions: when a hierarchical VLA plans too slowly and its executor ignores the plan
- Frontier Radar deep dive | Edge and real-time: where the latency budget of VLAs and world action models goes
- Physical AI Lab: a physical AI course where every number can be checked (course outline)
