A preprint from a Johns Hopkins University team, EdgeVLN (arXiv 2609.35570, 2026-09-28, evidence level [R]), tries to run an 8.03B-parameter vision-language navigation (VLN) model entirely on an NVIDIA Jetson Orin NX 16 GB and measures latency, energy and resident memory on the board. Several 4-bit formats reach almost the same navigation success rate, but the paper reports a 36.8× difference in step energy by execution path (format implementation and runtime). The authors attribute on-board memory and speed mainly to inference runtime rather than bit width, and add LATTE, an 8.2 MB stop-decision head, within the budget freed by quantization.
The problem: 16 GB of weights on a 16 GB unified memory
The model studied is StreamVLN (arXiv 2507.05240): a SigLIP-SO400M vision tower, a two-layer MLP projector and a 28-layer Qwen2 language model, 8.03B parameters in total, with 16.06 GB of BF16 weights. The Orin NX’s 16 GB of LPDDR5 is shared by the GPU, CPU, operating system, camera pipeline, KV cache and compute buffers; the paper gives the unified pool as 16.14 GB. The paper lists several reasons why estimating from bit width alone fails:
- On this board PyTorch has no 4-bit tensor-core path, so 4-bit weights mainly save storage and memory traffic; the PyTorch framework itself takes about 3 GB.
- Nominal weight size does not predict peak resident memory; runtime overhead, the KV cache and compute buffers decide whether the model fits.
- The authors quantize only the language model’s feed-forward (FFN) blocks, 5.70B of the 8.03B parameters; the vision tower, projector, attention projections, embeddings and LM head stay at 16 bits.
- Navigation is closed-loop: one wrong action changes later observations and errors compound along the trajectory, so a quantized model has to be evaluated in closed loop.
Same 4 bits, different execution paths
The authors first sweep precisions with fake quantization (weights rounded to the target grid but still stored in 16 bits) to see the effect of the format alone, then compare actual deployment paths. Success rate (SR) and oracle success rate (OSR) come from all 1,839 closed-loop R2R VLN-CE val-unseen episodes in the Habitat simulator; all configurations are scored on the same GPU. Latency, power, energy and memory come from a separate replay measured on the Orin NX. Table I(a) of the paper (energy and latency per navigation step):
| Format | Weights (GB) | SR (%) | OSR (%) | Resident memory (GB) | Power (W) | Energy (J) | Step latency (s) |
|---|---|---|---|---|---|---|---|
| BF16 | 16.06 | 57.69 | 65.04 | 14.99 | 10.85 | 151.68 | 13.98 |
| INT8 (loads, out of memory after 8 steps) | 10.36 | 57.04 | 64.82 | 15.98 | – | – | – |
| INT4 | 7.68 | 57.48 | 64.60 | 15.85 | 19.79 | 419.70 | 21.21 |
| NF4 | 7.68 | 57.15 | 64.76 | 14.70 | 18.18 | 19.31 | 1.06 |
| IQ4_NL (deployed) | 7.94 | 57.31 | 64.11 | 11.35 | 16.97 | 11.41 | 0.67 |
| INT2 | 6.44 | 46.00 | 55.79 | 16.03 | 20.08 | 437.86 | 21.81 |
Every 4-bit format stays within 0.54 SR points of BF16; 2 bits drops to 46.00%. INT4 and NF4 have smaller weights (7.68 GB) than IQ4_NL (7.94 GB) and also complete the on-board replay under PyTorch, yet peak at 15.85 GB and 14.70 GB resident, against 11.35 GB for IQ4_NL under the authors’ driver. On energy, IQ4_NL uses 11.41 J per step, NF4 1.7× that and INT4 36.8×; the paper attributes INT4’s 21.21 s per step to its dequantization path. Within PyTorch and with the same weight size, INT4 uses 21.7× the energy of NF4 per step (419.70 J vs 19.31 J) and 20× the latency (21.21 s vs 1.06 s). BF16 cannot stay fully resident: only 20 of the 28 transformer blocks plus the output layer stay resident, the rest is read from storage at each token, giving 20.8× the latency and 13.3× the energy of IQ4_NL for 0.38 SR points more. It has the lowest average power (10.85 W) because the GPU waits on storage.
The 0.672 s per step at the deployed point is amortized: one generation emits 4 actions, only 24 of the 89 requests trigger a generation, and the rest return an already buffered action in 0.024 s. Per four-action decision it is 2.49 s and 42.3 J.
What the llama.cpp driver does
llama.cpp has native low-bit kernels but encodes one image per turn and has no interface for StreamVLN’s rolling context. The authors’ VLN driver fills that gap:
- It exports the vision tower and projector to the runtime’s multimodal format and re-implements the 27×27→14×14 pooling, giving 196 tokens per frame.
- It clears the context every 32 steps and re-injects stride-sampled earlier frames as raw embedding batches.
- It implements the voxel-based spatial token pruning that the StreamVLN paper describes but did not release in its code: each token cell takes the median of its valid depth pixels, is back-projected to a voxel, and only the most recent token per voxel is kept within each temporal bucket; pruning runs only when history is re-injected. The paper reports that across the 3,252 chunk boundaries of the full run, pruning removed 41.7% of the re-injected memory tokens, bringing the amortized step from 0.744 s to 0.672 s at +0.05 SR points (1 episode of 1,839).
The final inference path consists of GGUF weights, the driver and the runtime, with no deep-learning framework.
LATTE: deciding when to stop from hidden states the backbone already computes
The gap between SR and OSR barely changes with quantization: 7.35 points at BF16, 6.80 points at IQ4_NL. OSR counts whether the trajectory entered the goal radius at any point, so this gap means that in about 7% of episodes (6.80–7.35%) the trajectory entered the goal radius but did not stop inside it. The authors conclude that wrong stopping is a property of the base model, not an artifact of quantization.
LATTE reads the backbone’s layer-24 hidden state and mean-pools the 196 image tokens of each frame into a 3584-dimensional feature. It buffers up to 256 decision steps, averages them over time, projects them to 256 dimensions, and passes them through two causally masked pre-norm transformer encoder layers (4 heads, FFN width 512) to output a “stop rank” between 0 and 1. The training target is clip(1 − d/5.0 m, 0, 1), zeroed after the closest-approach step and whenever the goal fails a camera-height visibility test; the head is trained separately on features from 3,184 successful rollouts in the R2R VLN-CE training split, with the backbone frozen. The trigger has hysteresis: it issues STOP only once the running maximum of the score has exceeded a threshold and the current score has fallen a set margin below it, with a minimum step count. The head stays in FP32 and runs on the CPU.
The comparison of seven candidate heads shows that what the head reads determines the outcome. MLP heads over pooled vision features lower BF16 SR from 57.69% to 51.70%; concatenating the instruction lowers it to 45.79%, and FiLM gives 55.14%. Only MLP heads over decoder hidden states improve it (58.24% at layer 20, 58.13% at layer 24, 15.7 MB each). LATTE reaches 58.35% on BF16 and 58.02% on IQ4_NL at 8.2 MB. On IQ4_NL it fixes 26 episodes and breaks 13, raising SR by 0.71 points while lowering OSR by 0.65 points: it changes stop decisions, not paths. The authors report an extra cost of 0.013 s and 0.13 J per step (about +1.9% and +1.1%). Separately, the authors state that on-board differences between candidate heads on the same backbone are below run-to-run spread and should not be compared; that does not establish that LATTE’s added overhead is below that spread.
How this relates to earlier pages on this site
In the September material collected in “Frontier Radar deep dive | Edge and real-time: where the latency budget of VLAs and world action models goes”, quantization in practice saved memory but was slower. In EdgeVLN execution paths need to be compared individually; slowdown is not a property of quantization in general: within PyTorch, INT4 takes 21.21 s per step and NF4 1.06 s, and the paper attributes INT4’s time to its dequantization path. INT2 is also slow at 21.81 s and 437.86 J per step, but its runtime is not specified. The BF16 comparison point is itself not a clean baseline: 8 of its 28 transformer blocks are read from storage at every token, and its 13.98 s includes that storage traffic. So neither NF4 being about 13× faster than BF16 nor INT4 being slower generalizes to “quantization is faster” or “quantization is slower”. The same applies to peak resident memory: INT4 peaks at 15.85 GB, above the partially resident BF16 (14.99 GB), NF4 at 14.70 GB, and IQ4_NL under the authors’ driver at 11.35 GB. Whether quantization makes a model faster has to be answered together with the format implementation and the runtime. “Frontier Radar deep dive | Simulation and evaluation: what VLA evaluation is missing beyond initial-state success rate” discussed closed-loop checks after quantization; this paper follows the same idea over 1,839 closed-loop episodes and uses the SR–OSR gap to isolate stopping errors. The planned L2 episode on quantization, latency and control frequency in edge deployment (see the course introduction) and the discussion of onboard compute in Episode 0.2 can both use these numbers as an example: beyond peak compute, look at resident memory, energy per step and the execution path.
Limitations
- This is an arXiv preprint and all numbers are reported by the authors; the material this article relies on contains no independent replication and no link to a code repository.
- SR and OSR come from simulation. The on-board measurements replay a recorded 89-frame real-world sequence to a stationary robot; they are not closed-loop real-world navigation, and the paper reports no real-robot navigation success rate. Power is measured on the module’s supply rail, excludes the motors, and does not subtract idle draw.
- LATTE’s trigger threshold and drop margin were chosen by a 16-cell grid search on the same val-unseen episodes that are reported. The authors state that, without a held-out split, the gain should be read as an upper bound. “Exceeding the BF16 baseline” means 58.02% versus 57.69%, a difference of 0.33 points.
- The authors note that on-board differences between heads on the same backbone fall below the run-to-run spread and should not be compared; the BF16 rows are single runs, the other configurations the mean of three.
- In one place the paper says voxel pruning is inactive in the RGB-only VLN-CE setting (the evaluation agent gives the encoder a placeholder pose and a zero-valued depth map); elsewhere it reports that pruning removed 41.7% of re-injected tokens across the full run. The paper does not explain how the two fit together.
- The paper states that “only IQ4_NL under our VLN driver fits the board”, yet NF4 and INT4 also complete the on-board replay, peaking at 14.70 GB and 15.85 GB resident against a 16.14 GB unified pool. The paper does not define what “fits” means, for example how much headroom a real camera pipeline needs or whether sustained operation is required, so this article does not treat which formats fit as settled.
- The paper states that INT4 and NF4 ran under PyTorch and IQ4_NL under the authors’ llama.cpp driver; it does not clearly name the runtime for INT8 or BF16. Differences between formats in the table therefore mix format and runtime effects.
Links
- arXiv 2609.35570: EdgeVLN: Runtime-Aware Deployment Ready Quantized Vision Language Navigation Model (authors: Rithvik Jonna, Man Namgung, Aakash Gurram, Tinoosh Mohsenin)
- Full paper in HTML
