
The preprint LaST0.5 (arXiv 2611.14048, 2026-11-10; authors from the State Key Laboratory of Multimedia Information Processing at Peking University’s School of Computer Science, The Chinese University of Hong Kong and Simplexity Robotics) is a world action model (WAM). It adds a layer of interaction-centric latents between a video world model and action generation, and supervises that layer with future frames on which the end-effector trajectory and a target-object box have been drawn. At inference the world model also assigns a task-progress value to each predicted segment of the future; once progress stops rising, the robot stops executing the rest of the chunk and observes again. The authors report a 99.5% average success rate on LIBERO, 88.0% on ten RLBench tasks and a 63.8% average on four real bimanual tasks (π0.5: 52.5%). This article sets out the two designs and their ablations, and estimates how large each gap is from the evaluation counts the paper gives. All numbers below are reported by the authors [R]; this site has not reproduced them or run the model.
The gap the authors want to fill: global scene representations and local interaction
The authors divide existing WAMs into two groups. Imagine-then-Execute models first predict future frames and then infer actions with inverse dynamics; Joint Modeling predicts future frames and actions together. The paper argues that both assume representations optimized for predicting how the world evolves are naturally suited to generating actions. In the authors’ view, a global prediction objective can underweight fine-grained robot–object interaction, so they add an abstraction dedicated to interaction between the world representation and the actions. This is the paper’s motivating argument; the evidence offered for it is the ablations and attention maps that follow.
Forward Dynamics Models, covered earlier on this site, take a different route: future prediction reads action features, actions never read the future, and inference skips future generation. LaST0.5 keeps future prediction at inference and lets actions read it through the latents. How WAM pretraining data can be organized is covered in yesterday’s SWAM article; what a world model is conditioned on and what it outputs is covered in the course lesson Physical AI Lab 3.2.
Three experts in one one-way attention stream
LaST0.5 places a world expert, a latent expert and an action expert in a Mixture-of-Transformers (MoT) architecture: the three share one global attention stream and keep separate weights. The paper factorizes the joint distribution as predicting future video first, then interaction latents from the video, and finally actions from both.
- World expert: instantiated with the pretrained diffusion transformer Cosmos-Predict2.5-2B. It takes 5 historical primary-view frames at 256×256, compressed into latent context by a causal video tokenizer, with the instruction injected through cross-attention; its output is 12 denoised future frames. The appendix states that the 5 history frames become 2 latent frames and the 12 future frames become 3, each latent frame holding 256 tokens. Training keeps the original Cosmos generation objective.
- Latent expert: initialized from Janus, a unified multimodal model. A small set of learnable query tokens reads the future representation from the world expert and produces M interaction latent tokens in one parallel forward pass; by default M = 3.
- Action expert: initialized from the same language backbone as the latent expert but with separate parameters, it refines the whole action chunk together with flow matching. Each timestep has two noise tokens and matching MLP heads, one predicting wrist motion and one predicting actuator commands; dimensions take the maximum over embodiments, with missing ones zero-padded. Each arm’s action is a relative 6-DoF end-effector pose plus an embodiment-specific actuator command, such as a binary gripper state or a multi-finger hand configuration.
The 12-step prediction horizon is split into 3 segments, each covering 4 future frames, 1 latent token and 4 action steps. A blockwise-causal mask lets the latents and actions of segment i attend directly only to the world and latent tokens of segments 1 to i. The world expert’s keys and values are detached so that gradients from the latent and action experts do not flow back into it, which the authors say protects the pretrained physical prior. The appendix adds that the world expert keeps bidirectional attention internally, so later world predictions can still influence earlier representations indirectly; the mask restricts direct cross-expert conditioning and does not guarantee strict temporal isolation.
The latent training target: drawing a trajectory and a box on future frames
The construction of the supervision target is the most concrete part of the paper. The last frame of each segment gets two visual cues. One is a 2D curve: the demonstrated 3D end-effector trajectory projected onto the image plane with the camera parameters. The other is a box: Grounding DINO proposes candidate object boxes from the task instruction, the interaction target is chosen by spatially matching them to the projected tool center point (TCP) at gripper opening and closing events, and the chosen object is assigned to the preceding motion segment. This annotation is done offline while preparing training data and is not needed at inference.
The annotated frames are encoded into patch tokens by a frozen SigLIP, a trainable MLP aggregates them into a single 1 × d_v vector, and projection and normalization give 3 latent targets. The latent expert’s outputs are aligned with the targets through a cosine-similarity loss. The total loss is a weighted sum of the world, latent and action losses, with coefficients used only to balance their numerical scales.
The ablations were run on LIBERO-Spatial, with LaST0.5 without AWAT as the base model at 99.6%:
| Variant | LIBERO-Spatial success rate |
|---|---|
| Base (LaST0.5 without AWAT) | 99.6% |
| Without the world expert | 96.0% |
| Without the latent expert | 97.8% |
| Latent targets from future frames without cues | 98.6% |
| Without the trajectory curve | 99.0% |
| Without the target box | 99.0% |
The direction matches the authors’ argument, but every gap lies in LIBERO-Spatial’s near-perfect range, and the smallest is only 0.6 percentage points. The text this site read does not give the number of evaluation episodes or random seeds for these ablations, so it cannot be judged whether gaps of 0.6 to 1.0 points exceed evaluation noise.
AWAT: predicted progress decides how far into an action chunk to execute
The authors argue that the future predicted by the world expert is not equally reliable over its whole horizon; their example is a LIBERO case in which the world model generates a failed grasp and the bowl stays on the table. Adaptive World-to-Action Transfer (AWAT) works as follows:
- After joint training, the base model is frozen and a lightweight MLP value head is attached to the denoised future-frame features of the world expert. Each segment of 4 frames is compressed by the video tokenizer into one latent frame; the value head pools the corresponding DiT hidden tokens and regresses one progress value.
- The progress target is t_i / T, where t_i is the median environment timestep of the segment’s four frames and T is the length of the full episode. Progress in demonstrations rises almost monotonically, so the authors reverse some training clips in time or repeat selected frames, with labels changed to decreasing or constant, so that the value head sees stagnation and regression.
- At execution, the robot keeps executing a segment’s actions as long as its progress value is higher than the previous segment’s; as soon as the value drops, it stops, takes a new camera observation and runs inference again.
- If the first segment’s progress value is already lower than the progress estimate from the previous cycle, the whole prediction is treated as unreliable: the model samples several world futures in parallel, picks the one with the highest value and passes its KV cache to the latent and action experts.
Appendix A.3 states that at inference the world expert runs only two denoising steps and caches its KV, while the action expert runs a default of 10 iterative denoising steps over 24 continuous action tokens. The granularity of the progress value was also ablated (LIBERO-Spatial, AWAT on): within a fixed 12-step horizon, one progress value every 1, 2, 4 or 12 steps gave 98.9%, 99.3%, 99.7% (the default of 4 steps) and 98.2%. The authors explain that per-step estimates fluctuate at high frequency, while a single estimate for the whole horizon lacks the temporal resolution needed for truncation.
This site has covered two related ideas. PAV predicts the consequence of each action prefix in the VLA’s own representation space and scores them to decide how much of a chunk to run; VeriVLA uses a JEPA latent world model to pick one of 10 candidate action chunks by predicted progress gain. LaST0.5 takes its progress values directly from its own world expert’s features, aligns them one-to-one with action segments, and resamples only when the first segment’s progress is below the previous cycle’s. How chunk length trades off against control rate and latency is covered in Physical AI Lab 2.6.
Results, and how large each gap is
| Evaluation | LaST0.5 | Without AWAT | Selected comparisons | Protocol |
|---|---|---|---|---|
| LIBERO, average of four suites | 99.5% | 99.2% | InternVLA 98.9%, Cosmos Policy 98.5% | Best success rate obtained during training |
| RLBench, average of ten tasks | 88.0% ± 3.0 | 85.0% ± 2.0 | LaST0 82.0% ± 3.0, Cosmos Policy 77.0% ± 2.0, π0.5 65.0% ± 4.0 | Mean ± standard deviation over 3 independent runs, 20 rollouts per task in each run |
| Real robot, average of four tasks | 63.8% | Not reported | π0.5 52.5%, Cosmos Policy 43.8%, LaST0 43.8% | 20 rollouts per task, counted by full-task completion |
LIBERO: the four suites are Spatial 99.7%, Object 100.0%, Goal 99.1% and Long 99.3%, 0.6 percentage points above the best baseline in the table, InternVLA. The paper states that LIBERO results are the best success rate obtained during training, which amounts to choosing a checkpoint on the evaluation result and makes the numbers optimistic; the text this site read does not give the number of LIBERO evaluation episodes.
RLBench: ten tasks on a simulated Franka Panda in CoppeliaSim, a single front-view RGB camera, 100 demonstrations per task and 300 training epochs; baselines start from their official pretrained checkpoints and are fine-tuned with the hyperparameters their papers recommend. AWAT adds 3.0 percentage points, while the standard deviation across the three runs is between 2.0 and 3.0, so the gap is of the same order as run-to-run variation; the paper gives no interval or test. LaST0.5 is not the best on every task: on Toilet Seat Down it scores 95.0% against 100.0% for HybridVLA, Cosmos Policy and LaST0; on Sweep to Dustpan 80.0% against HybridVLA’s 90.0%; on Wine at Rack 80.0% against LaST0’s 85.0%.
Real robot: the platform is a dual-arm Tianji Marvin with one head view and two wrist views. The four tasks are bag pulling, object organization, screw fastening and parcel-box packing; for each, 150 demonstrations were collected with PICO VR teleoperation. All policies were trained on the same demonstrations with the same multi-view inputs, and each task was run 20 times under varying initial object poses and arrangements. In the insertion stage of screw fastening LaST0.5 scores 50.0% and π0.5 30.0%; in the final sealing stage of packing, 30.0% against 20.0%.
The following is this site’s estimate from the counts the paper gives, not a number from the paper. Four tasks at 20 runs each make 80 runs, so 63.8% and 52.5% correspond to 51/80 and 42/80 full-task successes. Treating the two groups as independent samples, the 95% Wald half-width of the difference is about 15 percentage points, larger than the 11.3-point lead. A single task has only 20 runs, giving a single-group half-width of about 22 points near 50%; the insertion stage’s 50.0% against 30.0% is 10/20 against 6/20. The paper does not say whether the models shared the same set of initial arrangements, so a paired comparison is not possible. How to work out the intervals by hand is shown in Physical AI Lab 2.8.
Out-of-distribution tests cover only two tasks, bag pulling and object organization, with changes to object position, object instance and background. Averaged over the two, LaST0.5 scores 40.0% under position shifts, against 15.0% for π0.5 and 17.5% for Cosmos Policy; 55.0% under object shifts, level with π0.5; and 72.5% under background shifts, against 67.5% for π0.5. In bag pulling with new objects, π0.5’s 60.0% is above LaST0.5’s 45.0%. The text this site read does not state how many trials each cell of this test used.
Limitations
- This is an arXiv preprint, all numbers are reported by the authors [R], and this site has not reproduced them. The main text and the early appendix that this site read do not mention a release of code or model weights.
- LIBERO uses the best success rate during training; the architecture and latent-target ablations were run on the near-perfect LIBERO-Spatial, one number per variant, with no episode or seed counts. RLBench is the only result reported with a run-to-run standard deviation.
- The pretraining corpus exceeds 14k hours and includes teleoperated demonstrations, data collected with handheld grippers such as UMI, and human egocentric data. Its detailed composition is in Appendix B.1, which the material this site read did not include. The interaction cues need a 3D end-effector trajectory and camera parameters; the text read does not say how these are obtained for human egocentric data.
- AWAT truncates action chunks early and samples several futures in parallel when needed, so the number of inference calls varies; the text read reports no inference latency, resampling frequency or compute overhead. Training used 8 NVIDIA H20 GPUs.
- The real-robot evaluation has one bimanual platform and four tasks at 20 runs each; the zero-shot generalization the authors describe comes from two tasks. Success rates in simulation or on a lab robot are not hardware safety validation.
Links
- LaST0.5: A Generalist World Action Model with Interaction-Centric Latent Reasoning (arXiv 2611.14048)
- Full paper in HTML
- Physical AI Lab 3.2: World Models — Making Data with Cosmos, Writing Down What Each Kind of Generation Is Conditioned On, Putting 78.3% Back in Its Human Evaluation, Then Setting Acceptance Rules for Generated Data
- Physical AI Lab 2.6: To the Edge — Working Out the Control-Rate Ceiling from a Latency Log, Telling Whether Quantization Saves Size or Time, Then Sizing an Action Chunk
- Physical AI Lab 2.8: Reproducible Evaluation — Working Out Wald and Wilson Intervals by Hand, Sizing a Run from 1/√n, Then Bounding the Failure Rate after Zero Failures
- Frontier Radar | PAV: predict each action prefix’s consequence in the VLA’s own representation space, then decide how much of a chunk to execute
- Frontier Radar | VeriVLA: two samples decide when extra computation is worth it, then a latent world model picks one of 10 candidate action chunks by predicted progress
- Frontier Radar | Forward Dynamics Models: future prediction reads action features, actions never read the future, and inference skips future generation
- Frontier Radar | SWAM: feeding 47,000 hours of video and demonstrations to a world action model, with ablations of caption detail, UMI action labels and data mixture
- Physical AI Lab: a physical AI course where every number can be checked (course outline)



