Mobile manipulators and humanoids have to run VLAs on onboard embedded boards, where prefill of the language backbone is compute-bound and its latency grows with the hundreds of visual tokens fed in at every control step. Existing training-free accelerators recompute only some of the visual tokens at each step and either prune all the rest or reuse all of their previous KV caches. In the preprint TRIAGE (arXiv 2610.34519, 2026-10-25), authors from Seoul National University and Carnegie Mellon University argue that whether a token that is not recomputed should be reused or pruned depends on how far its cache has “drifted”, and should be decided token by token. The authors report that, on LIBERO and SimplerEnv in simulation, average success stays within 3 percentage points of full recomputation while prefill latency on a Jetson AGX Orin falls by up to 48%. This article looks at three things: how the diagnostic that decides between reuse and pruning works; how TRIAGE splits tokens into three sets at each step; and what prefill, end-to-end and p99 latency each tell you in the main results table. All figures below are reported by the authors [R]; this site has not reproduced them.
The problem: what to do with visual tokens that are not recomputed
Background figures from the paper: OpenVLA takes 256 visual tokens per step, and OpenVLA-OFT takes 512 from two cameras; visual tokens make up roughly 90% of the input sequence. For OpenVLA-OFT, which decodes all actions in one forward pass, prefill accounts for about two thirds of each control step (Table I). The authors also explain why they do not offload inference to a remote server: end-to-end latency then depends on the network, wireless links to mobile robots fluctuate and drop out, and a single late or lost response stalls the control loop.
Both existing approaches have a weakness. Pruning estimates each token’s importance at a shallow layer and permanently discards the less important ones; because this is irreversible, aggressive pruning removes task-relevant tokens. Reuse finds tokens that have barely changed since the previous frame and keeps their KV caches; but those caches were computed in the context of the previous frame, become stale as the scene changes, and repeated reuse accumulates error. The authors’ view is that applying one operation to every token outside the recomputed set either wastes compute or costs accuracy. For an important token that has not changed, a pruning method can only recompute it or discard it; for an unimportant token, a reuse method keeps it and keeps serving its cache even after it has gone stale. To the authors’ knowledge, no earlier VLA acceleration method chooses between the two per token.
The diagnostic: drift decides between reuse and pruning
The signal the authors use is drift: the relative change in a token’s attention value at a shallow layer (layer 2 in the paper) since it was last recomputed, averaged over attention heads. Because these shallow layers run on every token at every step, drift costs no extra computation.
The diagnostic is a controlled comparison. The authors run full recomputation on LIBERO-Spatial (20 episodes per backbone) and use its actions as the reference. Each “probe” takes 16 tokens and handles them once by reuse and once by pruning in layers 3 to 31, with every other token recomputed, then compares how far each resulting action is from the reference; the difference is Δ, and a negative Δ means reuse stays closer to full recomputation. Cache ages are 1, 2, 5, 10, 20 and 40 queries (up to 20 for OFT); the probed tokens are drawn from nine combinations of three drift levels and three importance levels. There are 14,442 probes on OpenVLA and 9,925 on OFT. The findings:
- Drift below 0.3: on both backbones the pooled Δ is negative with its 95% confidence interval entirely below zero, and no individual cache age rises above zero; reuse is no worse than pruning.
- Drift above 0.5: the trend reverses, and reusing the stale KV gives 1.6× (OpenVLA) and 4.4× (OFT) the action error of pruning it, more so for older caches.
- Between 0.3 and 0.5: reuse is still harmless on OpenVLA but already slightly worse than pruning on OFT. The threshold therefore depends on the backbone, and the authors suggest calibrating it for a new backbone by rerunning this diagnostic, which needs only full-recomputation rollouts and no training.
- Independent of importance: the authors state that the result holds regardless of token importance. Cache age affects how much reuse helps: on OpenVLA, reuse is still better for caches 20 or 40 queries old, while on OFT the advantage disappears from age 10, although reuse never becomes harmful.
For simplicity both backbones use δ = 0.3, the largest tested value below which reuse was never harmful on either; in the appendix, setting δ to 0.2 or 0.4 leaves end-to-end success statistically unchanged.
How TRIAGE splits tokens into three sets at each step
This separates the roles of two signals: cache validity decides whether a token that is not recomputed is reused or pruned; importance decides where the limited recomputation budget goes, namely to important tokens whose caches are no longer valid, for which neither reuse nor pruning is adequate.
- Importance: the attention the action-decoding tokens give the token at the shallow layer, plus the deep-layer saliency forecast memory adopted from SAFE-Pruner. The memory is recorded only at the first query, then carried to the current frame by patch correspondence using cosine similarity of projector outputs, and never refreshed. SAFE-Pruner re-records it on keyframes using all tokens, which the authors note raises tail latency, so TRIAGE runs no full-token query at any step. The important set is the top B + κ tokens (κ = 70), deliberately larger than the budget so that important tokens whose caches are still valid can be reused instead of recomputed.
- Validity: a cache is valid if and only if the token has one, its drift is below 0.3, and it was recomputed no more than T = 10 queries ago. The drift reference is anchored at the last recomputation rather than the previous query, so slow changes accumulate and are eventually detected. The age limit is a conservative safeguard; the authors report that removing it does not significantly change success.
- Filling budget B, in priority order: (1) maintenance: the round(μB) oldest cached tokens outside the important set, μ = 0.06, which keeps background reuse alive; (2) mandatory recomputation: important tokens with invalid caches, and tokens that have just entered the important set, whose caches were computed while they were judged unimportant; (3) optional refresh: the remaining slots go to important tokens with valid caches that are changing fastest.
- All other tokens: decided by validity alone, independent of importance; reused if valid, pruned otherwise.
Prune-only and reuse-only methods are the two extremes of this partition, with an empty reuse set and an empty prune set respectively. Because exactly B tokens are recomputed at every step, B determines prefill latency, and in deployment B is chosen as the largest value that meets the control loop’s latency target. In the experiments B is set to match SAFE-Pruner’s FLOPs: 88 tokens on OpenVLA (out of 256) and 163 on OFT (91 on OFT-Goal, out of 512). Hyperparameters were tuned on development episodes, mainly from LIBERO-Spatial, and frozen.
The budget sweep: why neither extreme is enough
Figure 4 lowers B from 88 to 18 (34% to 7% of the visual tokens) on LIBERO-Object with OpenVLA, with 500 episodes per budget and Wilson 95% intervals. The drop-all and keep-all variants share TRIAGE’s importance ranking and recompute the same B tokens, but prune or reuse all of the rest.
- Drop-all declines steadily as the budget shrinks, reaching 22.4% at B = 18; SAFE-Pruner, which also prunes every token it does not keep, reaches 19.4%.
- Keep-all is competitive at large budgets but collapses below B = 46, reaching 0.0% at B = 18; VLA-Cache with its recomputed set capped at B falls to 0.4%. Which extreme is ahead depends on the budget: keep-all leads at B = 46 and 64, drop-all at B = 28.
- TRIAGE matches full recomputation at B = 64, stays within 3 percentage points down to B = 28 (about 11%), and still reaches 75.8% at B = 18, where every other method is at or below 51.4%. Against the strongest baseline, VLA-Pruner (paired bootstrap over episodes), it is statistically indistinguishable at B ≥ 46, leads by 13.2 percentage points at B = 28 and by 24.4 at B = 18.
The abstract’s “recomputing only 7% of visual tokens, it beats the strongest baseline by 24.4 pp” refers to this single point: one suite, one backbone, B = 18.
The main results table: first ask which latency
The table below is taken from the paper’s Table I (four LIBERO suites, 500 episodes per suite; latency on a Jetson AGX Orin in bf16, in ms). Latency was measured over 250 queries per suite, replaying the same recorded observation stream, not timed while a robot ran in closed loop.
| Backbone / method | Average success | FLOPs | Prefill | End-to-end | p99 |
|---|---|---|---|---|---|
| OpenVLA full recomputation | 76.5 | 1.00× | 229.5 | 1261.3 | 1307.1 |
| OpenVLA VLA-Cache | 75.2 | 0.78× | 211.3 | 1365.7 | 1415.3 |
| OpenVLA VLA-Pruner | 73.3 | 0.39× | 204.5 | 1368.7 | 1428.5 |
| OpenVLA SAFE-Pruner (authors’ port) | 70.1 | 0.45× | 158.4 | 1166.6 | 1255.9 |
| OpenVLA SpecPrune-VLA (authors’ reimplementation) | 70.2 | 0.40× | 171.4 | 1177.0 | 1248.1 |
| OpenVLA TRIAGE | 73.7 | 0.45× | 167.7 | 1173.4 | 1228.2 |
| OFT full recomputation | 97.1 | 1.00× | 433.9 | 653.0 | 674.8 |
| OFT VLA-Cache | 96.2 | 0.77× | 421.5 | 702.3 | 774.2 |
| OFT VLA-Pruner | 95.8 | 0.53× | 328.3 | 555.2 | 765.8 |
| OFT SpecPrune-VLA (authors’ reimplementation) | 96.8 | 0.46× | 240.6 | 460.7 | 672.4 |
| OFT SAFE-Pruner | 95.5 | 0.44× | 238.3 | 466.5 | 768.0 |
| OFT TRIAGE | 95.8 | 0.44× | 224.0 | 443.1 | 461.1 |
- “Up to 48%” is OFT prefill. 224.0 versus 433.9 ms, or 0.52×. On OpenVLA, prefill is 0.73× and end-to-end only 0.93×, which the authors explain by 82% of the full-recomputation step lying outside prefill. End-to-end on OFT is 0.68×.
- Fewer FLOPs is not the same as faster on a Jetson. On OpenVLA, VLA-Pruner needs only 0.39× the FLOPs, yet its end-to-end step is 17% longer than TRIAGE’s, VLA-Cache’s 16% longer, and both are slower than full recomputation.
- TRIAGE is not the fastest in every column. On OpenVLA, SAFE-Pruner has lower prefill (158.4 ms) and end-to-end latency (1166.6 ms) than TRIAGE, but 3.6 percentage points lower average success. The authors’ claim is that TRIAGE lies on the Pareto frontier of success and prefill latency, with no baseline both more accurate and faster.
- Tail latency. On the Jetson, TRIAGE has the lowest p99 in every setting. On OFT its p99 is only 4% above its mean, while SAFE-Pruner’s p99 reaches 1.6× its mean, which the authors attribute to its keyframes running full-token inference. This column is where the fixed budget pays off most clearly (this site’s reading).
- “Within 3 pp” is an average. Per suite in Table I (this site’s arithmetic), TRIAGE on OpenVLA scores 49.2% on Long versus 56.0% for full recomputation, a gap of 6.8 percentage points; on OFT, Long is 88.6% versus 93.6%, a gap of 5.0.
On SimplerEnv (Google Robot, base OpenVLA checkpoint, no retuning), TRIAGE averages 54.5% against 54.8% for full recomputation, with SpecPrune-VLA and VLA-Pruner 4.2 and 6.1 percentage points lower; Jetson prefill is 0.74× that of full recomputation. Move Near uses 240 episodes per setting and Pick Coke Can 297.
How this relates to other articles on this site
Frontier Radar deep dive | Edge and real-time tallied where the latency budget of VLAs and world action models goes; TRIAGE targets only the prefill part of it, and for a model like OpenVLA, where prefill is only about 18% of a full-recomputation step on the Jetson, the end-to-end gain is limited. Frontier Radar | Fast-VLA also uses KV, but in the opposite way: its VLM thinks once per episode and stores the result as a task memory, whereas TRIAGE still calls the backbone at every control step and only reduces how many visual tokens are recomputed. The authors list asynchronous execution, quantization and layer skipping as orthogonal techniques that can be combined with TRIAGE; for the timing of an asynchronous action queue, see Physical AI Lab 2.3. When reading latency figures like these, the rule from Physical AI Lab 0.3 applies: a millisecond figure belongs only to the model, hardware and measurement method that produced it; “within 3 pp” and Wilson intervals can be unpacked with the methods of Physical AI Lab 1.8: Evaluation.
Limitations
- This is an arXiv preprint; all figures are reported by the authors [R], and this site has not reproduced them. The text read does not say whether code is available.
- All success rates come from LIBERO and SimplerEnv simulation; the paper has no real-robot experiments. Latency was measured by replaying a recorded observation stream, not in onboard closed-loop operation. Simulation results are also not a hardware safety validation.
- Both the diagnostic and most of the hyperparameter tuning used LIBERO-Spatial, so that suite is not a held-out evaluation. The sensitivity analysis instead uses LIBERO-Goal, which was not used for tuning (OpenVLA, B = 28, default 70.6%): no single change moves success by more than 4 percentage points and every 95% interval contains zero; removing the age limit gives 68.6% (paired difference −2.0 percentage points, 95% CI [−6.2, +2.2]). The intervals are about 8 percentage points wide, so this experiment cannot resolve smaller effects (this site’s inference).
- SAFE-Pruner on OpenVLA is the authors’ own port (no official code), and SpecPrune-VLA is a reimplementation of its static stage only (no public code). The main results tables give no confidence intervals in the text; among the success-rate results, only the budget sweep (Wilson intervals) and the sensitivity analysis report intervals.
- The authors themselves note that δ, T, κ, λ and μ are set empirically; that the drift threshold was observed to differ by backbone and, in their view, may shift further with task and scene speed, so a method that adapts it during deployment is needed; and that drift detects change only after it happens.
Links
- TRIAGE: Efficient Vision-Language-Action Inference Needs Both Reuse and Pruning (arXiv 2610.34519)
- Full paper in HTML
- Frontier Radar deep dive | Edge and real-time: where the latency budget of VLAs and world action models goes
- Frontier Radar | Fast-VLA: the 4B VLM thinks once per episode, and the actor queries a fixed KV task memory at every action chunk
- Physical AI Lab 0.3: See, Think, Act — Putting Each Rate Back in Its Own System
- Physical AI Lab 1.8: Evaluation — Splitting a “Net Gain of 2” into a Paired Table, Then Checking the Failure Labels and Random Seeds
- Physical AI Lab 2.3: Fine-Tuning SmolVLA — Planning Demonstrations for an Instruction, Checking What Each Number Measures, Then Timing the Asynchronous Action Queue
- Physical AI Lab: a physical AI course where every number can be checked (course outline)
