
The preprint Phoenix (arXiv 2611.24938, 2026-11-18; authors from the Institute of Automation, Chinese Academy of Sciences, Anyverse Dynamics, Tsinghua University, National University of Singapore and Fudan University) asks what kind of visual representation a world-action model (WAM) should predict. A WAM predicts future observations and robot actions together. The authors argue that latents trained for video reconstruction keep appearance but do not explicitly emphasize the spatial relationships and object-state changes that manipulation needs. Phoenix first distills geometry and spatiotemporal semantics into an encoder, then compresses its features into generative scene latents, which it predicts together with Wan VAE appearance latents and action chunks. This article goes through the paper’s architecture, representation ablation, main results on six benchmarks and data-efficiency experiments, and marks which gaps fall within evaluation sampling. All numbers below are reported by the authors [R]; this site has not reproduced them.
Three representation steps: distilled encoder, CS-VAE and joint prediction
Phoenix Encoder is a ViT-L/16 student initialized from the DINOv3 backbone in VGGT-Ω that learns from two frozen teachers. The geometry branch aligns intermediate per-frame and cross-view features with VGGT-Ω, plus a confidence-weighted log-depth loss that matches teacher-generated depth, so no ground-truth depth labels are needed. In the spatiotemporal semantic branch, V-JEPA 2.1-L sees the full clip while the student receives spatial masks applied consistently across frames, and a predictor maps its outputs into the teacher’s feature space. Updates from the two teachers alternate, and only the EMA weights of the backbone are kept for policy training and deployment. At 256 × 256 input, each frame’s feature map is 16 × 16 × 1024.
CS-VAE (causal-semantic VAE) compresses these per-frame features along channels and time while keeping the spatial patch grid: nine feature frames become three latent time steps with 48 channels each. The first latent, z0, depends only on frame 0; each later latent aggregates a causal window ending at its own time step. The objective is MSE and cosine feature reconstruction plus a KL term that pulls the posterior toward a standard Gaussian; Phoenix Encoder is frozen while CS-VAE trains.
Joint prediction: the 48-channel scene latents are concatenated along channels with the 48-channel Wan VAE appearance latents, giving a 96-channel visual representation. A video diffusion Transformer, initialized from Wan2.2-TI2V-5B, and a randomly initialized action diffusion Transformer are coupled at every layer through joint attention with an asymmetric mask: current-frame visual tokens attend only to one another; future visual tokens attend to all visual tokens but not to actions; action tokens attend to all visual and action tokens. The action stream also cross-attends to T5 instruction embeddings, robot state and soft prompt tokens indexed by arm. Training uses flow matching with the loss 0.5 × appearance term + 0.5 × feature term + action term. Each action chunk covers 32 consecutive control steps, and inference uses 10 denoising steps.
Physical AI Lab 3.2 writes a world model as x̂[t+1] = W(x[0:t], c[t]), and Cosmos predicts future video. Phoenix is a different design: besides appearance latents, its prediction target includes scene features distilled from visual foundation models, generated in the same denoising process as the actions. Under the attention mask above, prediction of future frames does not read action tokens, while action generation can read visual tokens.
39,306 hours of data and three training stages
The authors filter 54,413 hours of data down to 39,306 hours. By duration, robot demonstrations make up 31.09%, egocentric video 63.58% and UMI data 5.33%. Training has three stages: (a) distilling Phoenix Encoder on all filtered data; (b) training CS-VAE on the same data; (c) WAM pretraining on robot demonstrations only. For cross-embodiment training, robot states and actions are mapped into a shared 80-dimensional interface, and each arm ID selects eight learnable soft prompt tokens. WAM pretraining freezes Phoenix Encoder, CS-VAE and the video VAE and updates only the video and action experts, the state projection and the soft prompts, with a global batch size of 4,480 and one epoch at each data scale. The model is then post-trained separately on each benchmark with target-task demonstrations, for 25k steps by default.
SWAM, covered earlier on this site, also pretrains a WAM on tens of thousands of hours of video and demonstrations; what it varies is captions, action labels and data mixture. Phoenix’s ablation holds the data fixed and varies the visual representation itself, so the two papers answer different questions.
Representation ablation: changing the encoder, changing the compression
Table 5 of the paper compares visual representations on LIBERO-Plus. Every setting keeps the appearance VAE branch, and none uses WAM pretraining; each is post-trained directly. The first five rows fix compression to PCA48 and change only the encoder; the last row fixes Phoenix Encoder and replaces PCA48 with CS-VAE.
| Encoder added beside the appearance branch | Compression | Robot initialization perturbation | Total success |
|---|---|---|---|
| None (appearance VAE only) | — | 69.0% | 76.9% |
| VGGT-Ω | PCA48 | 60.4% | 78.3% |
| V-JEPA 2.1-L | PCA48 | 72.3% | 83.2% |
| DINOv3 | PCA48 | 66.2% | 74.4% |
| Phoenix Encoder | PCA48 | 70.2% | 84.4% |
| Phoenix Encoder | CS-VAE | 90.2% | 88.2% |
With the same compression, Phoenix Encoder reaches 84.4%, above the strongest alternative encoder, V-JEPA 2.1-L, at 83.2%, and above DINOv3, its initialization source, at 74.4%. The table also shows that adding DINOv3 features alone (74.4%) scores below the appearance VAE alone (76.9%). With Phoenix Encoder fixed, replacing PCA48 with CS-VAE raises total success from 84.4% to 88.2% and improves six of the seven perturbation categories; the largest gain is under robot initialization perturbation, from 70.2% to 90.2%. The full configuration is 11.3 percentage points above the VAE-only setting.
The authors also probe frozen features (Table 4, a ScanNet++ subset, with identical probe heads and training schedules for every encoder). Phoenix Encoder has an average rank of 1.80; it ranks first on direct future-feature prediction from camera motion, with a Global R@1 of 34.42%, and second on the other four readouts. DINOv2-L’s average rank is 2.60.
Main results on six benchmarks
The full model goes through WAM pretraining and is then post-trained separately on each benchmark. The table shows part of Tables 1–3 and appendix Tables S1 and S2; evaluation sizes come from the benchmark descriptions in the paper’s appendix.
| Benchmark | Setting and evaluation size | Phoenix | Selected comparisons from the paper’s tables |
|---|---|---|---|
| LIBERO-Plus | 7 perturbation categories, 10,030 evaluation instances | 94.8% | InternW0-Δ 92.8%, π0.5 85.7% |
| RoboTwin 2.0 C2R | 50 bimanual tasks, trained on clean demonstrations, evaluated in randomized scenes, 5,000 episodes in total | 72.4% | InternW0-Δ 71.9%, π0.5 46.0% |
| Bench2Dex | 26 task–embodiment settings, 12 arm–hand combinations, 5,200 episodes per policy | 45.6% | GR00T N1.5 28.3%, π0.5 27.2% |
| RoboDojo-RealWorld | Real ARX X5, Piper and Piper X, 18 tasks, 180 trials per policy | 30.0% | OpenWAM-α 24.4%, π0.5 12.8% |
| LIBERO-Pro | 4 perturbation settings, 40 tasks, 50 episodes per task | 66.5% | Continuous Reasoning 64.0% |
| RoboCasa-GR1 | Fourier GR1 with two arms and dexterous hands, 24 tasks, 1,200 episodes in total | 82.7% | CF-WAM 82.5% |
Phoenix leads in five of the seven LIBERO-Plus perturbation categories. It is not highest on camera viewpoint (89.9%, against 90.6% for InternW0-Δ) or background (98.6%, against 99.9% for Qwen-RobotManip-Context). On RoboDojo-RealWorld its success rates on the three configurations are 33.3%, 31.7% and 25.0%; on Piper, OpenWAM-α's 36.7% is higher than Phoenix. The task-level RoboTwin Table S3 also lists GigaBrain-0.7 with a 50-task mean of 67.9%; Phoenix is highest or joint-highest on 27 of those tasks.
Pretraining scale, few-shot adaptation and video prediction
Pretraining scale (Figure 4, RoboTwin 2.0 C2R): the comparison is a VAE-only Fast-WAM baseline pretrained under the authors’ data-scaling protocol. As the pretraining data grows from 25% to 100%, Phoenix’s success rate rises by 27.0 percentage points and the baseline’s by 11.4. With 50% of the data Phoenix reaches 67.7%; the baseline with all of it reaches 47.7%. Table 1 gives Fast-WAM 1.9% on RoboTwin 2.0 C2R; the paper says the Figure 4 baseline was pretrained under the authors’ protocol, while the text read here does not state the training setting behind the Table 1 row. Discussion of data scale should use the 47.7% baseline.
Few-shot post-training (Figure 5): starting from the same checkpoint pretrained on the full dataset, the model is post-trained with 25%, 50%, 75% and 100% of the demonstrations; at 100% each task has 50 demonstrations. On LIBERO-Plus, 25% of the demonstrations gives 91.1%, 3.7 percentage points below the full-data 94.8%, and 50% gives 94.0%. On RoboTwin 2.0 C2R, 25% gives 58.6% and 50% gives 66.1%, both above the Fast-WAM baseline above that was post-trained on all demonstrations (47.7%).
Video prediction (Table 6): 200 windows are evaluated per benchmark, and each model predicts eight future frames, spanning 32 action steps, from its own rollout observations. On RoboTwin 2.0 C2R, PSNR goes from 16.51 for Fast-WAM to 21.19 for Phoenix without WAM pretraining and 26.30 for the full Phoenix; LPIPS goes from 0.39 to 0.30 and then 0.16. The authors read this as higher manipulation success being accompanied by more faithful predictions of interaction dynamics. The two sets of results appear side by side; the paper does not separate cause and effect between them experimentally.
Which gaps exceed evaluation sampling
The following are this site’s estimates from the evaluation sizes the paper gives, not numbers from the paper. Treating episodes as independent trials and using 95% Wald intervals: RoboTwin 2.0 C2R has 5,000 episodes per policy, and near a 72% success rate the 95% half-width of a difference between two independent evaluations is about 1.8 percentage points; Phoenix’s 0.5-point lead over InternW0-Δ falls inside it. RoboCasa-GR1 has 1,200 episodes, giving a half-width of about 3.0 points, and 82.7% against 82.5% also falls inside. RoboDojo-RealWorld has 180 trials per policy; 30.0% corresponds to about 54 successes, the half-width for the difference from OpenWAM-α is about 9 points, and the 5.6-point lead does not exceed it. With LIBERO-Plus’s 10,030 instances the half-width is about 0.7 points, and the 2.0-point lead over InternW0-Δ exceeds it.
If the Table 5 ablation also uses all 10,030 instances (the text read here does not say so separately), the half-width of a difference is about 1.0 point: the 3.8-point gain from replacing PCA48 with CS-VAE clearly exceeds it, and the 1.2-point lead of Phoenix Encoder over V-JEPA 2.1-L is only slightly above that scale. These estimates cover only the sampling of evaluation episodes, not variation across training seeds, and the text read here reports no multi-seed results or confidence intervals. The hand calculation of the intervals is in Physical AI Lab 2.8.
Limitations
- This is an arXiv preprint; all numbers are reported by the authors [R], and this site has not reproduced them. The paper says code, pretraining data and models are available on the project page; this site has not opened or checked those releases.
- “State of the art on six benchmarks” is the authors’ wording. A separate model is post-trained for each benchmark; one model is not evaluated on all six at once. The text read here does not say whether the comparison numbers were rerun by the authors or taken from the original papers.
- The Table 5 representation ablation uses no WAM pretraining in any setting, so it shows differences under post-training only; the paper does not separately ablate how much the scene latents contribute within the full model.
- The real-robot part is three RoboDojo-RealWorld configurations with 10 trials per task. Success rates in simulation and on a lab robot are not hardware safety validation.
- The full paper text available to this site is cut off at the encoder architecture table in Appendix D; the training details after that point were not checked.
Links
- Phoenix: Learning Scene Structure Representations for Generalizable World-Action Models (arXiv 2611.24938)
- Full paper in HTML
- Physical AI Lab 3.2: World Models — Making Data with Cosmos, Writing Down What Each Kind of Generation Is Conditioned On, Putting 78.3% Back in Its Human Evaluation, Then Setting Acceptance Rules for Generated Data
- Physical AI Lab 2.8: Reproducible Evaluation — Working Out Wald and Wilson Intervals by Hand, Sizing a Run from 1/√n, Then Bounding the Failure Rate after Zero Failures
- Frontier Radar | SWAM: feeding 47,000 hours of video and demonstrations to a world action model, with ablations of caption detail, UMI action labels and data mixture
- Physical AI Lab: a physical AI course where every number can be checked (course outline)



