Most VLAs predict the next stretch of actions from the current frame alone. In the preprint REKI (arXiv 2610.22178, 2026-10-15), authors from Kyutai, Inria and École nationale des ponts et chaussées add a memory that costs very little: for every frame, 32 register tokens in the flow-matching action expert summarise that frame’s VLM output, and their keys and values stay in the expert’s cache, where the next two frames can read them. The authors report that this 2.4B model, pretrained on public data only, leads the compared public-data VLAs on LIBERO-Plus and RoboTwin Clean2Rand; the memory adds under 1% to inference time, and episodes in which an arm swings back and forth between two poses become about four times rarer. This article focuses on that oscillation: how it is defined, how it is measured and how the authors explain it, then looks at the conditions behind the benchmark, latency and real-robot numbers. All numbers below are author-reported [R]; this site has not reproduced them.
The design: memory lives in the action expert, and the VLM encodes each frame once
REKI has two parts, a VLM and an action expert. The VLM is Qwen3.5-2B; it packs the prompt and all camera views into one stream, and its final-layer hidden states serve as the encoded context. Besides the language instruction, the prompt states the embodiment, the control mode, the control frequency and a binned episode length. The action expert has 10 blocks of width 768 and reads the encoded context through cross-attention; the robot state is pooled and modulates the expert through adaptive normalisation. At inference, ten Euler steps from noise produce one action chunk.
The memory works as follows: for every frame, n_r = 32 learnable register tokens are placed in the expert’s stream ahead of the noisy action tokens. The paper states three attention rules:
- the registers cross-attend that frame’s encoded context and self-attend the registers of earlier frames;
- the action tokens read the registers of the current and past frames as well as themselves; the registers never read the action tokens;
- the registers’ keys and values are kept in the expert’s cache, a rolling window over the K − 1 preceding frames with K = 3; when a new frame enters, the registers of the oldest frame are dropped.
Because the registers do not read the action tokens and skip adaptive normalisation, they depend only on the observations, not on the flow time or the robot state. The paper draws two cost consequences. At inference, the current frame’s registers are computed in the first Euler step and reused in the other nine, and the VLM still processes only the current view for each frame, so the encoder costs the same as in a memoryless policy. In training, a sample is K frames from one episode taken at a 1 s stride, packed into a single batched pass with no recurrence to unroll; every frame has its own action-chunk loss, so a K-frame window yields K training signals. The paper also notes that each window is processed in isolation during training, whereas at inference the cached registers were themselves computed from the registers of earlier frames, so information can persist beyond the window.
Figure 2 of the paper sets this approach beside three existing families: feeding past frames into the VLM together with the current one, whose cost grows with the window (at least in the vision encoder); a recurrent state written and read step by step in the backbone or the head, which must be unrolled in training; and an added memory module. The authors state that at inference the memory adds under 1% to the prediction time of the model without memory (K = 1).
All pretraining data are public: teleoperated robot data (RoboSet, Galaxea, AgiBot World 2026, RoboCOIN, RoboMIND, RDT-1B, DROID, BridgeData V2, twelve datasets of Open X-Embodiment and others), motion-planned simulation data, about 350k episodes and 170M frames of egocentric human manipulation video, and vision–language data (10% of the training samples). Each dataset keeps its native control convention rather than being remapped to a shared action space. The model is pretrained for 200k steps on 32 H100 GPUs with a global batch of 2,880, then finetuned on each benchmark separately.
Oscillation: with the same weights, removing memory raises the share from 4.1% to 14.8%
Section 6.2 of the paper defines oscillation precisely: for each arm, compare the direction it moves during one chunk with the direction during the previous one, ignoring small movements; if an arm moves in roughly the opposite direction at three or more consecutive chunks, that is, it goes back and forth at least twice, the episode counts as oscillating. The count is taken on RoboTwin Clean, 50 tasks with 20 matched seeds each. The authors set this criterion on 16 of the tasks and confirmed the effect of memory on the other 34. Fewer than 5% of oscillating episodes succeed.
| Condition (training window → evaluation window) | Share of oscillating episodes |
|---|---|
| Trained with K = 3, evaluated with K = 3 (3 → 3) | 4.1% |
| Same weights, one past frame at evaluation (3 → 2) | 4.6% |
| Same weights, past frames removed at evaluation (3 → 1) | 14.8% |
| Trained and evaluated without memory (1 → 1) | 18.6% (half of the failed episodes oscillate) |
“About four times rarer” depends on which memoryless condition is the reference: 14.8 / 4.1 ≈ 3.6 and 18.6 / 4.1 ≈ 4.5 (this site’s arithmetic). The difference is largest in two-arm tasks. There, the most frequent cycle is an arm swap, one arm reaching while the other retracts: 65 of 1000 episodes without memory against 1 with memory; it typically starts a few seconds after a release. The authors also state that memory does not rescue all of these episodes: on the seeds where the memoryless policy oscillates, the policy with memory succeeds on 45%.
The authors’ explanation is perceptual aliasing. A memoryless policy maps the current observation to the next chunk; if poses A and B each look like demonstration states that continue towards the other, the chunk predicted at A leads to B and the chunk predicted at B leads back to A. A short history resolves the ambiguity, because a window that has just come from B does not match demonstrations that move towards B. This is the authors’ attribution, drawing on classic reinforcement-learning work on perceptual aliasing; the text read here reports changes in oscillation share, not an experiment that tests the mechanism directly. The paper also mentions that temporal ensembling and bidirectional decoding can reduce switching between chunks, but they need extra policy queries, which REKI’s memory does not.
The same ablation also compares success rates (Fig. 5 of the paper; LIBERO-Plus is evaluated after finetuning on LIBERO):
| Condition | LIBERO-Plus | RoboTwin Clean | RoboCasa365 |
|---|---|---|---|
| 3 → 3 | 90.4 | 73.4 | 38.4 |
| 3 → 2 | 90.1 | 73.6 | 39.3 |
| 3 → 1 | 88.9 | 67.2 | 34.9 |
| 1 → 1 | 88.9 | 67.9 | 33.0 |
Most of the gain comes with the first past frame. Because every frame in a window receives a loss, the model finetuned with memory also learns to act without it: evaluated without memory (3 → 1), it performs close to the model finetuned at K = 1. The authors conclude that the gain comes mainly from reading past frames at inference, not from training on windows.
Benchmarks: the lead is among “VLAs pretrained on public data”
The paper’s headline claim is a new Pareto frontier between performance and model size: highest score on LIBERO-Plus, and outperformed only by models about twice its size on RoboTwin Clean2Rand and RoboCasa365. When reading these numbers, keep the scope of the comparison in view:
- Out of distribution: 90.4 on LIBERO-Plus, 5.6 points above the next public-data VLA, InternVLA-A1.5 (84.8); on Clean2Rand, 10.1 points above ABot-M0 (55.7), i.e. 65.8 (this site’s arithmetic). For Clean2Rand, models are finetuned on clean scenes only and evaluated under six conditions (clean scenes, each of four randomisations alone, all four together), averaged. On the RoboCasa365 composite tasks unseen in training, REKI scores 14.1, on par with Qwen-RobotManip (14.9) and behind Xiaomi-Robotics-1 (32.1).
- In distribution: 38.4 on RoboCasa365, ahead of RLDX-1 (36.0) and Qwen-RobotManip (35.9); 58.6 on RoboCasa-GR1, on par with RLDX-1 (58.7) and ABot-M0 (58.3); 97.5 on LIBERO, where most models lie between 97.6 and 99.4; 85.9 / 84.6 on RoboTwin, below the similarly sized InternVLA-A1.5 (93.3 / 93.0). The authors note that the latter is supervised by an additional video generation model during training and uses a more complex, multi-stage pretraining.
- What is compared: baseline numbers are taken from published results. The larger models pretrained on in-house data (Qwen-RobotManip, RLDX-1, Xiaomi-Robotics-1) and ABot-M0.5, built on a video generation model, are explicitly described by the authors as a gauge of the current upper bound, not a like-for-like comparison.
The effect of pretraining: trained from scratch, REKI reaches 52.0 on Clean2Rand and 86.1 on LIBERO-Plus; pretraining adds 13.8 and 4.3 points out of distribution, and in distribution 2.9 and 0.8 on RoboTwin and 1.0 on LIBERO. Training settings also matter a great deal for robustness: on LIBERO-Plus, disabling camera dropout or the image augmentations lowers success under camera-viewpoint changes by more than 20 points, and disabling state dropout lowers success under changes of the initial robot state by 8.1 points.
Latency and the real robot: the conditions behind 21.8 ms and 7–37.5 points
Latency is measured for one call, from the raw observation to the unnormalised actions of one chunk, including preprocessing and host–device transfers; inputs are two 256 × 256 images, the robot state and a fixed prompt, at batch size one, with each model’s released checkpoint and inference code; each value is the median over 320 chunks and three repetitions. REKI’s VLM prefill and expert are each compiled into a single CUDA graph, and at K = 3 it predicts a chunk (ten actions) in 21.8 ms, the lowest among the compared models: X-VLA 25.5 ms, π0.5 with openpi 33.8 ms, ABot-M0 46.1 ms, InternVLA-A1.5 51.3 ms. The compiled paths for X-VLA, ABot-M0, InternVLA-A1.5 and ABot-M0.5 were built by the authors, who state that they did not change these models’ computation. Without compilation, REKI, π0.5 and ABot-M0 take 118–139 ms, InternVLA-A1.5 352 ms and X-VLA (0.9B) 51 ms.
In GPU time, REKI takes 16.5 ms per chunk, on par with X-VLA (16.1 ms). In the measurement of Fig. 5, the VLM prefill takes 9.5 ms and the Euler steps 8.1 ms, and reading the cached registers adds 0.1 ms. In a separate measurement (Appendix Table 10), REKI’s Euler steps take 8.2 ms against 34.9 ms for InternVLA-A1.5, which the authors attribute to the latter’s larger action expert (0.5B in 24 layers vs 0.2B in 10). The 21.8 ms is a value for one H100 and these inputs, not a control frequency; the paper notes that it bounds how quickly a policy can react to a new observation and how often it can replan.
The real-robot experiments use a 6-DoF UR5 arm with a Weiss WSG50 gripper and UMI-style parallel jaws, with a 15 Hz action loop and one fixed Orbbec Femto Mega RGB camera; observations are an RGB image and a 7D absolute joint state. Five tasks cover pick, place, stack and sweep; for each task, 50 training episodes are collected with a Meta Quest controller and 10 test episodes are evaluated, with randomised object placements and distractors in each. REKI and π0.5 are both finetuned from their pretrained checkpoints for 3,000 steps with a global batch of 360 and a learning rate of 5 × 10⁻⁵, training only the last four vision encoder blocks. Results: REKI outperforms π0.5 by 7 to 37.5 percentage points across the five tasks; without memory, the two shorter tasks are unchanged (65 vs 65, 57.5 vs 57.5), while the three longer tasks drop by 12 to 17.5 points.
Two points need to be kept apart here. First, the action chunk lengths differ: 50 steps (3.3 s) for π0.5, fixed by its pretraining, and 30 steps (2 s) for REKI, so the gap between REKI and π0.5 cannot be attributed to the model alone. Second, each task has only 10 test episodes, scores include partial credit, and the text read here gives no confidence intervals. The authors state that finetuning REKI takes 8 H100 GPU-hours on the real-robot tasks and 40 to 400 on the benchmarks.
How this relates to existing pages on this site
Physical AI Lab 1.6: Imitation Learning explains that behaviour cloning looks only at the current observation and that errors can carry a robot out of the demonstrations’ coverage; it also covers ACT’s action chunking and temporal ensembling. REKI’s oscillation analysis adds a different failure: an error need not roll ever further away, it can also bounce between two states that look alike. According to the paper, methods such as temporal ensembling need extra policy queries, while REKI uses a few frames of memory.
Frontier Radar | Fast-VLA also uses a KV cache as memory, but in the opposite direction: Fast-VLA calls the VLM only once per episode and stores the result as a KV task memory that stays fixed during the episode; REKI runs the VLM at every frame, and its memory is a rolling set of registers in the action expert covering the last few frames. The two papers use different benchmarks and hardware, so their numbers cannot be compared directly.
Physical AI Lab 2.3: Fine-Tuning SmolVLA works with another open VLA; the REKI paper lists SmolVLA among earlier models that released their full pipeline on public data. The rule from 2.3, that a number describes only the model, platform and settings that produced it, applies equally to REKI’s 21.8 ms and its real-robot scores. REKI’s pretraining mixes teleoperated robot data, egocentric human video and vision–language data; for how these sources differ in units and action spaces, see Physical AI Lab 1.5: Where Does Data Come From? The oscillation criterion was set on 16 tasks and confirmed on another 34, with 20 matched seeds per task; that procedure can be read with the pairing and held-out questions from Physical AI Lab 1.8: Evaluation.
Limitations
- This is an arXiv preprint and all numbers are author-reported [R]. This site has not reproduced any result.
- The paper states that the full code from data curation to deployment and all pretrained and finetuned weights are released, at
github.com/kyutai-labs/reki; this site has not checked that repository’s contents or licence. - Baseline scores are taken from published results; latency was measured by the authors under one protocol, and some baselines’ compiled paths were built by the authors. Comparisons with models trained on in-house data are a reference only.
- The oscillation criterion is the authors’ own (reversal at three or more consecutive chunks) and is counted only on RoboTwin Clean; perceptual aliasing is the authors’ attribution, not a separately verified mechanism.
- The memory window is only 3 frames at a 1 s stride. The authors consider a memory of a few seconds sufficient for most tasks in standard benchmarks, but whether the registers can carry the much longer context of tasks that depend on events far in the past remains to be tested.
- The real-robot part covers one UR5 platform, five tasks and 10 episodes per task, with partial credit, and the chunk length differs from π0.5’s. Neither the simulation results nor this real-robot evaluation is a hardware safety validation.
- This article is based on the paper’s HTML main text (up to the early part of the references) and does not draw on the full tables in the appendix.
Links
- REKI: Registering Knowledge in VLAs (arXiv 2610.22178)
- Full paper in HTML
- Physical AI Lab 1.6: Imitation Learning — Working Out How an Error Rolls Out of the Demonstrations’ Coverage, Then Putting the ACT, Diffusion Policy and Data Scaling Numbers Back in Their Own Conditions
- Physical AI Lab 2.3: Fine-Tuning SmolVLA — Planning Demonstrations for an Instruction, Checking What Each Number Measures, Then Timing the Asynchronous Action Queue
- Physical AI Lab 1.5: Where Does Data Come From? Working Out Demonstration Time from SO-101 Metadata, Then Separating the Units and Action Spaces of Three Sources
- Physical AI Lab 1.8: Evaluation — Splitting a “Net Gain of 2” into a Paired Table, Then Checking the Failure Labels and Random Seeds
- Frontier Radar | Fast-VLA: the 4B VLM thinks once per episode, and the actor queries a fixed KV task memory at every action chunk
- Physical AI Lab: a physical AI course where every number can be checked (course outline)
