Frontier Radar | STVLA: geometry-model spatial tokens stored in pairs with appearance tokens in a bounded memory; authors report 30.0–58.3 percentage points above π0.5 on three real-robot tasks

STVLA stores frozen VGGT-Ω spatial tokens with appearance tokens in a 16-entry memory. Authors report +30.0–58.3 pp over π0.5 on 3 real tasks.

Horizontal bar chart on a light background. The title reads "Authors report: real-robot block stacking, STVLA 100.0% vs π0.5 41.7%", and the subtitle says the values are real-robot block stacking success (%), 60 trials per method over 3 heights, reported by the authors. Four horizontal bars drawn to scale from zero: π0.5 at 41.7%, Spatial Forcing at 58.3%, MemoryVLA at 88.3% and STVLA at 100.0%, with the STVLA bar in a darker colour. The source line at the bottom cites arXiv 2611.16589 Table V and notes that the two other real-robot tasks are not drawn.
Drawn from arXiv 2611.16589 Table V (reported by the authors): real-robot block stacking success, 60 trials per method over 3 target heights. The darker STVLA bar is the result discussed in the article and the highest of the four bars shown; the other three methods are lower. The paper gives no intervals; by this site's rough Wilson intervals, STVLA does not overlap π0.5 and just overlaps MemoryVLA. Object organization and phone packaging are not drawn. Not measured by this site.

Many VLAs choose each action primarily from the current observation. When two observations look alike but the task is at different stages, the policy may repeat a step it has already finished; when the action depends on geometry, such as the height of a surface or whether a spot is still free, 2D features from a single frame and a single view do not express it. STVLA, arXiv 2611.16589 (2026-11-12), takes spatial tokens from a frozen visual geometry model, VGGT-Ω, and stores them in pairs with the backbone’s appearance tokens in one bounded memory, so the policy can recall past scene structure when it generates actions. The authors report success rates 30.0–58.3 percentage points above π0.5 on three tasks on an AGIBOT G2 dual-arm robot. All numbers below are reported by the authors [R]; this site has not run the code or reproduced the results.

What the memory stores: appearance tokens plus geometry tokens

STVLA adds two things to π0.5’s PaliGemma vision-language model and flow-matching action expert: spatial tokenization and a bounded visual–spatial memory. The paper describes the implementation as follows:

  • Spatial tokens: the frozen 1B-parameter VGGT-Ω reads K = 4 frames (512 px) from the fixed head camera, uses the earlier frames as context and keeps only the current frame’s features, 1024 patch features of width 2048 per frame. A trainable compressor with one learned-query cross-attention layer pools them into N_s = 256 spatial tokens, passing through a variational bottleneck on the way.
  • Memory bank: at each step the current SigLIP visual tokens and spatial tokens are concatenated into one entry and stored. Capacity is L_m = 16; when it is exceeded the oldest entry is dropped, and the bank is cleared at the start of every episode. The current tokens query the whole bank through L = 2 cross-attention layers, and each query can see both the visual and the spatial tokens of past entries; an element-wise gate fuses what is retrieved with the current tokens.
  • Action conditioning: the spatial tokens occupy their own prefix segment, attend only within that segment and are hidden from the original prefix tokens; action tokens see the whole prefix. The authors say this keeps the backbone’s original image-language attention pattern.
  • Training: a flow-matching action loss plus a feature reconstruction loss (weight 1) and a KL loss (weight 10⁻⁴) on the compressor; at inference the posterior mean is used instead of a sample. Training uses 8 NVIDIA H100 GPUs with a batch size of 16 per GPU.

The authors start from this observation: a memory that stores only appearance features fails to retrieve once the gripper occludes the workspace or the viewpoint changes, because the current frame no longer resembles the stored frame with the same physical configuration; a spatial representation computed on the current frame alone does not know which parts of the scene the policy has already changed. Other memory designs covered on this site store different things: REKI caches the last few frames in register tokens inside the action expert, and Fast-VLA has the VLM produce a KV task memory once per episode. Why task-level progress memory reduces repeated steps is worked out with chain success rates in Physical AI Lab 3.4. What STVLA adds is explicit geometry tokens inside each memory entry.

Which entries to recall: a relevance and non-redundancy score

All 16 entries take part in attention; STVLA only adds a shared bias to each entry. For entry j, take the token-wise means h_j of the entry and x̄_t of the current tokens and compute two quantities:

  • similarity r_j = (1 + cos(h_j, x̄_t)) / 2;
  • difference u_j = ‖h_j − x̄_t‖ / (‖h_j‖ + ‖x̄_t‖).

Their product s_j is standardized across entries, multiplied by β / τ_r and added to the attention logits of every token in that entry. The score has no learned parameters; the idea comes from the max-relevance, min-redundancy criteria used in feature selection. The paper states that the product is a feature-space proxy, not a direct estimate of task-specific information gain. The bias only changes weights and does not exclude any past entry.

The score is sensitive to β. On real-robot block stacking (τ_r = 1, 60 trials per setting over 3 heights), β = 0, which turns the bias off, gives 80.0%; β = 0.05, 0.3 and 0.8 all give 71.7%, below the unbiased setting; β = 0.5 gives 100.0% and β = 1.0 gives 81.7%. The authors also state that the benefit depends on the scale and does not grow with it. For VGGT-Ω input frames K from 1 to 5, success is 88.3%, 93.3%, 85.0%, 100.0% and 86.7%, again not monotonic; the authors explain that more frames give richer features but raise inference latency, and the paper gives no latency figures.

Four simulation benchmarks: only π0.5 is compared under the same protocol

The authors report that STVLA has the highest average success among the listed methods on LIBERO, LIBERO-Mem, MIKASA-Robo and SimplerEnv Bridge. When reading the tables, first check which rows the authors ran themselves: in all four tables only π0.5 and STVLA are marked as trained and evaluated by the authors, and the authors state that in simulation both use the same data and the same number of optimization steps. Most other rows, MemoryVLA among them, are taken from the MemoryVLA paper or other work, whose evaluation settings may differ. The table below lists only the two rows the authors ran (success rate, %):

BenchmarkTrials per taskπ0.5STVLADifference (pp)
LIBERO (five suites, evaluation seed 7)5096.697.8+1.2
LIBERO-Mem (10 tasks)2038.041.0+3.0
MIKASA-Robo (5 tasks)10032.445.2+12.8
SimplerEnv Bridge (4 tasks)2466.774.0+7.3

The difference column is this site’s subtraction of the table values. Per task, the gains are uneven. On LIBERO-Mem, the repeated-placement tasks B×5 and B×7 each rise by 30 percentage points (70 vs 40, 40 vs 10), but Cs→Bskt falls from 55 to 25, and Bt→P and Swap2 are each 5 percentage points lower. On SimplerEnv, STVLA is below π0.5 on the spoon task (79.2 vs 83.3) and below MemoryVLA on the carrot task (62.5 vs 75.0). MIKASA-Robo is the only benchmark where STVLA beats π0.5 on all five tasks. The paper gives no intervals or multi-seed results for the simulation success rates.

Three real-robot tasks and the ablations, read by trial count

The real robot is an AGIBOT G2 dual-arm robot with one fixed head RGB camera and two wrist RGB cameras. Demonstrations and STVLA training steps per task: block stacking, 817 demonstrations and 30k steps; object organization, 237 and 20k; phone packaging, 203 and 40k. Block stacking is tested at 3 target heights with 20 trials each, 60 trials in total; the other two tasks get 20 trials per method. The two comparison methods, as the paper’s related-work section describes them: Spatial Forcing aligns intermediate VLA features with representations from a pretrained 3D foundation model and needs no explicit 3D input at inference; MemoryVLA keeps a bounded perceptual–cognitive memory bank. Success rates (%):

MethodBlock stacking (60 trials)Object organization (20 trials)Phone packaging (20 trials)Average
π0.541.755.010.035.6
Spatial Forcing58.340.010.036.1
MemoryVLA88.335.015.046.1
STVLA100.085.045.076.7

The authors’ failure breakdown: in block stacking, π0.5 and Spatial Forcing mostly fail through execution hesitation or dropping the object during grasping and placing; in object organization, π0.5 mainly fails to grasp the block and presses the left-arm button again at the final step; in phone packaging, neither MemoryVLA nor STVLA shows hesitation failures, and STVLA’s remaining failures are in grasping the charger or the final placement of the phone case. From this the authors suggest that memory may ease transitions between subtasks, while precise grasping and placing remain hard.

The paper gives no intervals for the real-robot success rates. A rough Wilson 95% interval, as in Physical AI Lab 2.8 (this site’s estimate, treating trials as independent and ignoring differences between the three heights in block stacking):

  • Block stacking: STVLA 60/60, about 94.0%–100%; π0.5 25/60, about 30.1%–54.3%; no overlap. MemoryVLA 53/60, about 77.8%–94.2%, just overlaps STVLA’s interval.
  • Object organization: STVLA 17/20, about 64.0%–94.8%; π0.5 11/20, about 34.2%–74.2%; overlapping.
  • Phone packaging: STVLA 9/20, about 25.8%–65.8%; π0.5 2/20, about 2.8%–30.1%; overlapping.

Of the three gains of 30.0–58.3 percentage points over π0.5, only block stacking has clearly separated intervals at the current sample sizes. The other two point the same way, but with 20 trials per method the single-group intervals overlap; overlap only means this rough check cannot separate them, not that the difference is insignificant, and the paper reports no statistical test.

The ablations are also run on block stacking. Full STVLA reaches 100.0%, without memory 95.0% and without the spatial branch 96.7%, which at 60 trials is a difference of 3 and 2 trials; the rough Wilson intervals (about 86.3%–98.3% and 88.6%–99.1%) both overlap the full model’s. The authors add a difference in failure modes: without memory there is more hesitation, and without the spatial branch there are empty grasps and placement errors. The defaults β = 0.5 and K = 4 are each the highest point of the non-monotonic sweeps in the second section, and they belong to the same task and the same default configuration as the 100.0% in the main table; the text we read does not say whether the two values were chosen beforehand on a separate evaluation.

Limitations

  • This is an arXiv preprint and every number is reported by the authors [R], with no independent reproduction. The paper gives no intervals, standard deviations or multi-seed results for any success rate; the Wilson intervals above are this site’s rough estimates from the counts.
  • The authors are affiliated with AgiBot, and the real robot, the AGIBOT G2, carries the same brand.
  • For simulation the authors state that π0.5 and STVLA use the same data and steps. For the real robot, the text we read gives demonstrations and training steps only for STVLA and does not say whether the baselines used the same data and training budget.
  • In the simulation tables most baselines such as MemoryVLA come from other papers, so “highest average on each benchmark” includes cross-paper comparisons whose evaluation settings may differ; the LIBERO table note also says that four-suite and five-suite averages are not directly comparable.
  • Limitations stated by the authors: features from the pretrained 3D model do not always yield high-quality spatial tokens, and the memory bank covers only a short time window (capacity 16), which limits its benefit on much longer-horizon tasks.
  • Inference runs the 1B-parameter VGGT-Ω at every step. The paper notes that more frames raise inference latency but reports no latency or control rate.
  • The paper gives a project website address; this site did not open it and did not find out whether code or models are public.

Links