Frontier Radar deep dive | Edge and real-time: where the latency budget of VLAs and world action models goes

9 sources: Rolling-WAM cuts replanning from 978 to 215 ms, early exits plus fewer denoising steps cut ~80% of latency; quantization saves memory, not time.

This article is based on the September 2026 archive of the Physical AI Radar. The radar’s “Edge and real-time” track states one constraint bluntly: a model that exceeds the real-time budget does not get into the control loop. What changed in September is that latency no longer shows up only in paper appendices; more and more VLA and world action model (WAM) papers report milliseconds, control frequency and target hardware as headline results. This article puts those numbers side by side to see where the latency budget goes, which savings work, and where the evidence is still insufficient.

A latency table that cannot be compared directly

The table below lists the self-reported latency or frequency of each work used in this article. Hardware, resolution and model size all differ, so the table only shows orders of magnitude; the entries cannot be ranked against each other directly.

WorkEvidenceHardwareReported latency / frequency
Cosmos 3 Edge (4B)[O]Jetson AGX Thor T5000About 1.53 s to generate a 32-step action chunk (15 Hz)
Rolling-WAM[R]A100215 ms per replanning cycle (978 ms for Joint-WAM)
Decoupled Early Exits (π0.5, LIBERO)[R]A100 40GB0.537 s → 0.095 s
OpenVLA-OFT (BF16, measured in VLAQuantBench)[R]RTX 4090 (batch size 1)139.9 ms for full inference
Agile-WAM[R]Not stated in the abstract11.9 ms
PASSAGE (humanoid traversal)[R]Jetson AGX OrinPlanning at 6.25 Hz, control at 50 Hz

How the budget is spent: action chunks and replanning cycles

When a single inference is much slower than the control frequency, the usual approach is to generate a chunk of actions at once and prepare the next chunk while executing the current one. NVIDIA’s Cosmos 3 Edge post-training tutorial ([O], 2026-08-19, a “frontier baseline” item in the radar) gives a complete on-device example: a 4B-parameter omni model (containing a 2B Nemotron reasoner), about 9 GB in BF16, that fits entirely in the Jetson AGX Thor’s onboard memory. It generates 32 action steps at a time, which at 15 Hz covers about 2.13 seconds, while generating a chunk takes about 1.53 seconds, so the next chunk is ready before the current one finishes executing and no data-center GPU is needed in the control loop. The price is capability: in closed-loop simulation evaluation on 120 language-conditioned manipulation tasks in RoboLab (built on Isaac Lab-Arena), the Edge version reaches a 22.9% success rate, versus 36.8% for the more compute-hungry Nano version. Also, the tutorial’s sample code executes only the first 16 steps of each chunk before replanning, which at 15 Hz is about 1.07 seconds, shorter than the 1.53-second generation time; the tutorial does not explain how the two fit together.

The problem is worse for WAMs, because every replanning step has to jointly denoise video and actions. Rolling-WAM ([R], 2026-09-24) maintains a sliding window in which the video-action chunks sit at different noise levels: at each step only the chunk about to be executed is fully denoised, and chunks further out are only partially denoised, to be continued when new observations arrive. On an A100, this cuts each replanning cycle from 978 ms for Joint-WAM to 215 ms (about 4.5x), about 2.5x faster than the 548 ms of Fast-WAM. Success rates did not drop as a result: 98.1% average on LIBERO (98.5% for Joint-WAM), and 85.0% average on a real Unitree G1 (78.3% for Joint-WAM, 61.7% for π0.5).

PASSAGE ([R], arXiv 2609.18732) is an example of a hierarchical approach: a flow matching planner generates reference trajectories at 6.25 Hz and a whole-body tracker executes at 50 Hz, both on a Jetson AGX Orin, with real-time chunking keeping consecutive chunks coherent.

Three ways to cut compute

Fewer layers and fewer denoising steps. Decoupled Early Exits ([R], 2026-09-24) treats VLM backbone depth V, action expert depth A and the number of denoising steps D as three jointly tunable knobs. Across two models (SmolVLA, π0.5) and two benchmarks (LIBERO, Meta-World), it reduces latency by 79.2% and FLOPs by 31.8% on average, raises average success rate by 5.6% (the abstract does not say whether relative or in percentage points), and adds only 2.1% (SmolVLA) or 4.1% (π0.5) parameters per exit point. The configuration tables in the paper show that the best configurations all cut denoising steps from 10 to 2 or 3, and the authors’ own analysis says D improves both FLOPs and latency, so a sizeable part of the gain may come from running fewer denoising steps rather than from early exiting itself. Another finding from the authors is that the optimal configuration varies by task; there is no universal best setting.

Cross-execution caching. rMuscle ([R], 2026-09-16) exploits the similarity that comes from repeated execution at factory workstations: a Context Cache reuses visual token outputs, and an Action Cache reuses neuron activation patterns to reduce weight accesses. It achieves a 1.29–1.42x speedup on an RTX 4090 and a Jetson Thor with unchanged real-robot success rates. The gain is modest, but it needs no model changes and can be stacked with other methods.

Small specialized models. Agile-WAM ([R], 2026-09-17) does not use a large pretrained generative backbone. Instead it encodes vision and touch into a shared latent space and does flow matching directly; visual latents are predicted at larger time intervals, while tactile latents predict the next frame to keep up with sudden changes at contact. On 5 contact-rich real-robot tasks, overall success rate improves by 29.4% in relative terms, with an inference latency of 11.9 ms.

Quantization saves memory, but not necessarily time

The measured-kernel results in VLAQuantBench ([R], 2026-09-21) deserve deployment teams’ attention. For the LLM part of OpenVLA-OFT: BF16 uses 15.0 GiB and takes 139.9 ms for full inference; AWQ brings memory down to 6.1 GiB but latency rises to 181.1 ms; NF4 is 6.3 GiB and 152.6 ms; LLM.int8() is 9.0 GiB and 379.5 ms. In the authors’ words, every quantization path tested reduced peak memory, but all were slower than BF16. On the real robot, W4 dropped OpenVLA-OFT’s success rate on a grasping task from 88.0% to 54.0%, while W8A8 held at 86.0%. So on memory-constrained boards such as Orin and Thor, quantization first solves “will it fit”, not “will it be faster”; getting faster requires efficient kernels for the corresponding precision.

Lightweight modules beside the control loop

Not every on-device module is the policy itself. Online fault detection for mobile robots ([R], 2026-09-24) uses the time-series foundation model TSPulse to generate pseudo-labels offline, distills them into a MiniRocket student model, and adds recursive least squares for online adaptation. A single inference takes 4.30 ms on a CPU, and under real mechanical degradation VUS-PR rises from 0.26 to 0.75. Albireo ([R], 2026-08-29) maintains a Kalman filter per object and calls the detector only when prediction uncertainty exceeds a threshold. On Jetson AGX Thor and Orin, it keeps AP@50 within ±1.2 percentage points of per-frame inference while cutting total energy use by 12.1%–17.6%; for comparison, a fixed skip-every-other-frame scheme loses 8.6 percentage points of AP@50. Both show that on the edge, “when not to compute” matters as much as “how fast to compute”.

Limitations

  • The numbers in the table come from different hardware (A100, RTX 4090, Jetson Thor, Jetson Orin, CPU) and different resolutions and cannot be compared across rows; only Cosmos 3 Edge, PASSAGE, rMuscle and Albireo report results on Jetson.
  • Decoupled Early Exits has no real-robot experiments; Rolling-WAM’s 215 ms was measured on an A100, and its on-device performance is unknown; the Agile-WAM abstract does not state the hardware used to measure latency.
  • Cosmos 3 Edge is a vendor tutorial, the 22.9% is a closed-loop simulation success rate, and how the sample code’s executed steps line up with the generation time is not explained.
  • All [R] results are self-reported by the authors in preprints.

What to watch next

  • Whether WAMs can bring the replanning cycle below a few hundred milliseconds on Jetson-class hardware while keeping real-robot success rates.
  • Whether mature on-device kernels appear for the precisions VLAs commonly use (W4, W8A8), so that quantization saves both memory and time.
  • Whether papers converge on reporting the same set of numbers, “hardware + resolution + latency per cycle + control frequency”, so results can be compared across works.

Links

Related Posts