Frontier Radar | LaST0.5: an interaction latent layer between the world model and the actions, supervised by future frames with a drawn trajectory and target box, and predicted progress that decides how much of a chunk to execute

LaST0.5 adds box- and trajectory-supervised interaction latents to a WAM and cuts chunks by predicted progress. Authors: LIBERO 99.5%, real robot 63.8%.

Horizontal bar chart on a light background, with the title “RLBench mean success over ten tasks (authors)” at the top, a subtitle stating that values are means over three independent runs with 20 rollouts per task in each, and a source line at the bottom. Five bars drawn to scale from zero show, from top to bottom, π0.5 65.0%, Cosmos Policy 77.0%, LaST0 82.0%, LaST0.5 without AWAT 85.0% and LaST0.5 with AWAT 88.0%.
Table 1 of arXiv 2611.14048 (reported by the authors): mean success rate on ten RLBench tasks with a simulated Franka Panda in CoppeliaSim, over three independent runs with 20 rollouts per task in each. Standard deviations across the three runs in the table range from 2.0 to 4.0 percentage points, and the 3.0-point gap between LaST0.5 with and without AWAT is of the same order, so no bar is highlighted. OpenVLA-OFT, HybridVLA and Fast-WAM from Table 1 are not shown. Bars follow the order of the paper’s table, not a ranking; drawn from Table 1 of the paper, not a measurement by this site.

The preprint LaST0.5 (arXiv 2611.14048, 2026-11-10; authors from the State Key Laboratory of Multimedia Information Processing at Peking University’s School of Computer Science, The Chinese University of Hong Kong and Simplexity Robotics) is a world action model (WAM). It adds a layer of interaction-centric latents between a video world model and action generation, and supervises that layer with future frames on which the end-effector trajectory and a target-object box have been drawn. At inference the world model also assigns a task-progress value to each predicted segment of the future; once progress stops rising, the robot stops executing the rest of the chunk and observes again. The authors report a 99.5% average success rate on LIBERO, 88.0% on ten RLBench tasks and a 63.8% average on four real bimanual tasks (π0.5: 52.5%). This article sets out the two designs and their ablations, and estimates how large each gap is from the evaluation counts the paper gives. All numbers below are reported by the authors [R]; this site has not reproduced them or run the model.

The gap the authors want to fill: global scene representations and local interaction

The authors divide existing WAMs into two groups. Imagine-then-Execute models first predict future frames and then infer actions with inverse dynamics; Joint Modeling predicts future frames and actions together. The paper argues that both assume representations optimized for predicting how the world evolves are naturally suited to generating actions. In the authors’ view, a global prediction objective can underweight fine-grained robot–object interaction, so they add an abstraction dedicated to interaction between the world representation and the actions. This is the paper’s motivating argument; the evidence offered for it is the ablations and attention maps that follow.

Forward Dynamics Models, covered earlier on this site, take a different route: future prediction reads action features, actions never read the future, and inference skips future generation. LaST0.5 keeps future prediction at inference and lets actions read it through the latents. How WAM pretraining data can be organized is covered in yesterday’s SWAM article; what a world model is conditioned on and what it outputs is covered in the course lesson Physical AI Lab 3.2.

Three experts in one one-way attention stream

LaST0.5 places a world expert, a latent expert and an action expert in a Mixture-of-Transformers (MoT) architecture: the three share one global attention stream and keep separate weights. The paper factorizes the joint distribution as predicting future video first, then interaction latents from the video, and finally actions from both.

  • World expert: instantiated with the pretrained diffusion transformer Cosmos-Predict2.5-2B. It takes 5 historical primary-view frames at 256×256, compressed into latent context by a causal video tokenizer, with the instruction injected through cross-attention; its output is 12 denoised future frames. The appendix states that the 5 history frames become 2 latent frames and the 12 future frames become 3, each latent frame holding 256 tokens. Training keeps the original Cosmos generation objective.
  • Latent expert: initialized from Janus, a unified multimodal model. A small set of learnable query tokens reads the future representation from the world expert and produces M interaction latent tokens in one parallel forward pass; by default M = 3.
  • Action expert: initialized from the same language backbone as the latent expert but with separate parameters, it refines the whole action chunk together with flow matching. Each timestep has two noise tokens and matching MLP heads, one predicting wrist motion and one predicting actuator commands; dimensions take the maximum over embodiments, with missing ones zero-padded. Each arm’s action is a relative 6-DoF end-effector pose plus an embodiment-specific actuator command, such as a binary gripper state or a multi-finger hand configuration.

The 12-step prediction horizon is split into 3 segments, each covering 4 future frames, 1 latent token and 4 action steps. A blockwise-causal mask lets the latents and actions of segment i attend directly only to the world and latent tokens of segments 1 to i. The world expert’s keys and values are detached so that gradients from the latent and action experts do not flow back into it, which the authors say protects the pretrained physical prior. The appendix adds that the world expert keeps bidirectional attention internally, so later world predictions can still influence earlier representations indirectly; the mask restricts direct cross-expert conditioning and does not guarantee strict temporal isolation.

The latent training target: drawing a trajectory and a box on future frames

The construction of the supervision target is the most concrete part of the paper. The last frame of each segment gets two visual cues. One is a 2D curve: the demonstrated 3D end-effector trajectory projected onto the image plane with the camera parameters. The other is a box: Grounding DINO proposes candidate object boxes from the task instruction, the interaction target is chosen by spatially matching them to the projected tool center point (TCP) at gripper opening and closing events, and the chosen object is assigned to the preceding motion segment. This annotation is done offline while preparing training data and is not needed at inference.

The annotated frames are encoded into patch tokens by a frozen SigLIP, a trainable MLP aggregates them into a single 1 × d_v vector, and projection and normalization give 3 latent targets. The latent expert’s outputs are aligned with the targets through a cosine-similarity loss. The total loss is a weighted sum of the world, latent and action losses, with coefficients used only to balance their numerical scales.

The ablations were run on LIBERO-Spatial, with LaST0.5 without AWAT as the base model at 99.6%:

VariantLIBERO-Spatial success rate
Base (LaST0.5 without AWAT)99.6%
Without the world expert96.0%
Without the latent expert97.8%
Latent targets from future frames without cues98.6%
Without the trajectory curve99.0%
Without the target box99.0%

The direction matches the authors’ argument, but every gap lies in LIBERO-Spatial’s near-perfect range, and the smallest is only 0.6 percentage points. The text this site read does not give the number of evaluation episodes or random seeds for these ablations, so it cannot be judged whether gaps of 0.6 to 1.0 points exceed evaluation noise.

AWAT: predicted progress decides how far into an action chunk to execute

The authors argue that the future predicted by the world expert is not equally reliable over its whole horizon; their example is a LIBERO case in which the world model generates a failed grasp and the bowl stays on the table. Adaptive World-to-Action Transfer (AWAT) works as follows:

  1. After joint training, the base model is frozen and a lightweight MLP value head is attached to the denoised future-frame features of the world expert. Each segment of 4 frames is compressed by the video tokenizer into one latent frame; the value head pools the corresponding DiT hidden tokens and regresses one progress value.
  2. The progress target is t_i / T, where t_i is the median environment timestep of the segment’s four frames and T is the length of the full episode. Progress in demonstrations rises almost monotonically, so the authors reverse some training clips in time or repeat selected frames, with labels changed to decreasing or constant, so that the value head sees stagnation and regression.
  3. At execution, the robot keeps executing a segment’s actions as long as its progress value is higher than the previous segment’s; as soon as the value drops, it stops, takes a new camera observation and runs inference again.
  4. If the first segment’s progress value is already lower than the progress estimate from the previous cycle, the whole prediction is treated as unreliable: the model samples several world futures in parallel, picks the one with the highest value and passes its KV cache to the latent and action experts.

Appendix A.3 states that at inference the world expert runs only two denoising steps and caches its KV, while the action expert runs a default of 10 iterative denoising steps over 24 continuous action tokens. The granularity of the progress value was also ablated (LIBERO-Spatial, AWAT on): within a fixed 12-step horizon, one progress value every 1, 2, 4 or 12 steps gave 98.9%, 99.3%, 99.7% (the default of 4 steps) and 98.2%. The authors explain that per-step estimates fluctuate at high frequency, while a single estimate for the whole horizon lacks the temporal resolution needed for truncation.

This site has covered two related ideas. PAV predicts the consequence of each action prefix in the VLA’s own representation space and scores them to decide how much of a chunk to run; VeriVLA uses a JEPA latent world model to pick one of 10 candidate action chunks by predicted progress gain. LaST0.5 takes its progress values directly from its own world expert’s features, aligns them one-to-one with action segments, and resamples only when the first segment’s progress is below the previous cycle’s. How chunk length trades off against control rate and latency is covered in Physical AI Lab 2.6.

Results, and how large each gap is

EvaluationLaST0.5Without AWATSelected comparisonsProtocol
LIBERO, average of four suites99.5%99.2%InternVLA 98.9%, Cosmos Policy 98.5%Best success rate obtained during training
RLBench, average of ten tasks88.0% ± 3.085.0% ± 2.0LaST0 82.0% ± 3.0, Cosmos Policy 77.0% ± 2.0, π0.5 65.0% ± 4.0Mean ± standard deviation over 3 independent runs, 20 rollouts per task in each run
Real robot, average of four tasks63.8%Not reportedπ0.5 52.5%, Cosmos Policy 43.8%, LaST0 43.8%20 rollouts per task, counted by full-task completion

LIBERO: the four suites are Spatial 99.7%, Object 100.0%, Goal 99.1% and Long 99.3%, 0.6 percentage points above the best baseline in the table, InternVLA. The paper states that LIBERO results are the best success rate obtained during training, which amounts to choosing a checkpoint on the evaluation result and makes the numbers optimistic; the text this site read does not give the number of LIBERO evaluation episodes.

RLBench: ten tasks on a simulated Franka Panda in CoppeliaSim, a single front-view RGB camera, 100 demonstrations per task and 300 training epochs; baselines start from their official pretrained checkpoints and are fine-tuned with the hyperparameters their papers recommend. AWAT adds 3.0 percentage points, while the standard deviation across the three runs is between 2.0 and 3.0, so the gap is of the same order as run-to-run variation; the paper gives no interval or test. LaST0.5 is not the best on every task: on Toilet Seat Down it scores 95.0% against 100.0% for HybridVLA, Cosmos Policy and LaST0; on Sweep to Dustpan 80.0% against HybridVLA’s 90.0%; on Wine at Rack 80.0% against LaST0’s 85.0%.

Real robot: the platform is a dual-arm Tianji Marvin with one head view and two wrist views. The four tasks are bag pulling, object organization, screw fastening and parcel-box packing; for each, 150 demonstrations were collected with PICO VR teleoperation. All policies were trained on the same demonstrations with the same multi-view inputs, and each task was run 20 times under varying initial object poses and arrangements. In the insertion stage of screw fastening LaST0.5 scores 50.0% and π0.5 30.0%; in the final sealing stage of packing, 30.0% against 20.0%.

The following is this site’s estimate from the counts the paper gives, not a number from the paper. Four tasks at 20 runs each make 80 runs, so 63.8% and 52.5% correspond to 51/80 and 42/80 full-task successes. Treating the two groups as independent samples, the 95% Wald half-width of the difference is about 15 percentage points, larger than the 11.3-point lead. A single task has only 20 runs, giving a single-group half-width of about 22 points near 50%; the insertion stage’s 50.0% against 30.0% is 10/20 against 6/20. The paper does not say whether the models shared the same set of initial arrangements, so a paired comparison is not possible. How to work out the intervals by hand is shown in Physical AI Lab 2.8.

Out-of-distribution tests cover only two tasks, bag pulling and object organization, with changes to object position, object instance and background. Averaged over the two, LaST0.5 scores 40.0% under position shifts, against 15.0% for π0.5 and 17.5% for Cosmos Policy; 55.0% under object shifts, level with π0.5; and 72.5% under background shifts, against 67.5% for π0.5. In bag pulling with new objects, π0.5’s 60.0% is above LaST0.5’s 45.0%. The text this site read does not state how many trials each cell of this test used.

Limitations

  • This is an arXiv preprint, all numbers are reported by the authors [R], and this site has not reproduced them. The main text and the early appendix that this site read do not mention a release of code or model weights.
  • LIBERO uses the best success rate during training; the architecture and latent-target ablations were run on the near-perfect LIBERO-Spatial, one number per variant, with no episode or seed counts. RLBench is the only result reported with a run-to-run standard deviation.
  • The pretraining corpus exceeds 14k hours and includes teleoperated demonstrations, data collected with handheld grippers such as UMI, and human egocentric data. Its detailed composition is in Appendix B.1, which the material this site read did not include. The interaction cues need a 3D end-effector trajectory and camera parameters; the text read does not say how these are obtained for human egocentric data.
  • AWAT truncates action chunks early and samples several futures in parallel when needed, so the number of inference calls varies; the text read reports no inference latency, resampling frequency or compute overhead. Training used 8 NVIDIA H20 GPUs.
  • The real-robot evaluation has one bimanual platform and four tasks at 20 runs each; the zero-shot generalization the authors describe comes from two tasks. Success rates in simulation or on a lab robot are not hardware safety validation.

Links