
The preprint SWAM (arXiv 2611.12870, 2026-11-09, by authors from Shanghai Jiao Tong University, Shanghai Innovation Institute, Noematrix Ltd., Westlake University and Tongji University) studies how to organize data and supervision when pretraining a world action model (WAM). A WAM starts from a pretrained video model and, conditioned on language, predicts future video and robot actions together. The authors ask whether the language–video correspondence learned in video pretraining survives once action learning is added. To find out, they assembled about 47,000 hours of general videos, egocentric videos, UMI demonstrations and robot demonstrations, then varied caption detail, action-label coverage, data mixture and model architecture one at a time, comparing success rates in randomized RoboTwin 2.0 scenes. This article goes through those ablations in turn and marks which gaps are small relative to the authors’ evaluation size. All numbers below are author-reported [R]; this site has not reproduced them.
One model that learns from action-free video and from demonstrations with actions
The model is a two-branch MoT architecture. The video branch is initialized from Wan2.2-TI2V-5B and the action branch is randomly initialized; each keeps its own residual stream, and all parameters are updated in training. Information flows only from video to action: video tokens attend causally to current and past video tokens, while action tokens attend to video tokens from past and current chunks and to the state and action tokens within the current chunk. The authors point out that this one-way link lets video-only samples keep training the video branch without the video objective directly updating the action branch.
Language enters only the video branch, through cross-attention. The action branch has no text cross-attention and gets language information only through video features, which is meant to reuse the language–visual correspondence already built in video pretraining. Each chunk holds 2 video latent frames and 48 action steps. UMI and robot demonstrations share a 16-dimensional bimanual end-effector representation: per hand, a 3-dimensional position, a 4-dimensional quaternion in wxyz order and a 1-dimensional gripper value, with each source normalized by its own q1 and q99 statistics.
The training objective is one flow matching loss for video and one for actions, added with equal weights. Every sample contributes to the video loss; only samples with valid action annotations contribute to the action loss, and missing or invalid action targets are masked. Video without action labels can therefore take part in the same pretraining run. HWAM, covered earlier on this site, deals with the gap between a humanoid’s reference action and its actual state; SWAM is about data and supervision, and by the paper’s modeling choice its video branch is not conditioned on proprioceptive state.
A 47,016.7-hour data pool: four kinds of source, each with different supervision
Table 1 of the paper lists the pretraining data: 4,407,996 episodes and 47,016.7 hours in total.
| Source | Hours | Composition | Supervision provided |
|---|---|---|---|
| General videos | 10,909.6 | Panda-70M 10,087.4 h, OpenVid 822.1 h | language–video |
| Egocentric videos | 10,986.3 | EgoSuite 8,600.6 h, Ego4D 1,339.7 h, EgoVerse 1,046.0 h; 371,212 trajectories | language–video |
| UMI demonstrations (in-house) | 15,284.9 | 411,754 trajectories | language–video–action |
| Robot demonstrations | 9,836.0 | subsets of ABC, AgiBot, InternData-A1, Galaxea, RoboCOIN and RoboMind; 768,356 trajectories | language–video–action |
The UMI demonstrations were collected with RoboPocket, a smartphone combined with a handheld adaptive gripper; operators manipulate objects by moving, opening and closing the gripper. Head and both wrist views are recorded, the phone’s visual–inertial tracking supplies the action trajectories, and external encoders record gripper opening and closing. For robot demonstrations, the authors keep only subsets from bimanual robots with head and both wrist views that pass their quality checks, excluding data with missing views, inconsistent view semantics, insufficient resolution, overly simple tasks or single-arm embodiments. All data is packaged in LeRobot 2.1 format. Filtering differs by source: for general videos (OpenVid, Panda-70M), the original annotations and a VLM content review remove animations, static scenes, television programs and other unsuitable content; for egocentric videos, a VLM assesses motion blur, brightness, occlusion and whether a clip contains meaningful action; the UMI and robot demonstrations use similar filtering rules. Qwen 3.8 Max and Seed 2.1 Pro then segment subtasks and write the captions.
Physical AI Lab 1.5 covered the difficulty with human video: a human hand is not a robot hand, and the video carries no precise joint angles. SWAM handles this by having people demonstrate with a gripper whose structure resembles a robot end effector, then writing the actions into the same 16-dimensional end-effector representation the robot demonstrations use. The action-label ablation in the next section measures how much those human action labels help robot control.
Caption detail and action labels: three ablations
All ablations share one evaluation protocol. After pretraining, the model is post-trained on the 50 RoboTwin 2.0 tasks with Clean demonstrations only. Easy evaluates under the same clean conditions (Clean-to-Clean, C2C); Hard evaluates with randomized background textures, clutter, lighting, table height and camera placement (Clean-to-Random, C2R). Ablations run 25 episodes per task and setting. The figures below are Hard success rates.
- Language captions (Table 4, pretraining on a mix of egocentric video and UMI for 20k steps): subtask-level captions at Brief, Medium and Detailed give 47.20%, 49.60% and 53.52%; adding scene and lighting descriptions gives 55.04%. At the Detailed level, episode-level captions give 51.60% and subtask-level captions 53.52%. Randomly shuffling which subtask description goes with which video segment within a trajectory lowers 53.52% to 52.00%.
- Captions for robot data (Table 5): nine randomly chosen robot-data subsets hold 145,430 trajectories but only 592 task-level instructions. The original instructions give 50.80%; 100 paraphrases per instruction, one sampled at random during training, give 54.76% (+3.96 percentage points); a description of the specific manipulation process and environment for each trajectory gives 56.16%. For reference, post-training directly from Wan with no multisource pretraining gives 34.56%.
- UMI action-label coverage (Table 6): all UMI videos are kept and only the share of trajectories with action labels changes. Coverage of 0%, 25%, 50%, 75% and 100% gives 46.72%, 47.76%, 49.92%, 50.16% and 51.28%, a gain of 4.56 percentage points from 0% to 100%. Unlabeled videos still take part in video learning, so this comparison separates “more video” from “more action labels”.
Mixture, architecture and pretraining scale
With the pretraining budget fixed at 20k steps, the authors compare five sampling ratios of egocentric video (Ego) to UMI (Table 8): 1:0 gives 42.56%, 2:1 gives 52.64%, 1:1 gives 53.84%, 1:2 gives 53.28% and 0:1 gives 51.28%. All three mixtures score above either source alone; equal sampling is highest, 2.56 percentage points above UMI only. When the training on one source is increased (Table 7), holding UMI fixed while adding Ego (10k, 20k and 30k total steps) gives Hard rates of 49.68%, 53.84% and 55.60%; holding Ego fixed while adding UMI gives 42.08%, 53.84% and 51.20%, so the last step goes down, and the authors’ wording is “generally improves”. Adding 25k steps of transitional training on general videos before the mixed training raises Hard from 53.84% to 56.48%.
The architecture comparison (Table 9) uses 30k steps of multisource pretraining. Single-stream, where video and actions share a residual stream, gives 52.56%; standard Dual-stream gives 52.48%; the Dual-stream variant in which the action branch gets language only through video features gives 57.36%, 4.88 percentage points above standard Dual-stream. On Easy this variant is the lowest: Single-stream 86.48%, standard Dual-stream 84.96% and the variant 84.80%. In another experiment, with 20k steps and equal Ego–UMI sampling, adding image features from CLIP, DINOv2, SigLIP2 or Qwen3-VL 2B through cross-attention lowers Hard from 49.52% with native inputs to 39.76%–42.96%. The authors’ explanation is that this new fusion pathway has not had language–video pretraining, so the model may lean more on appearance cues in the current training distribution; the paper says “may”.
Pretraining scale is measured in training video hours, with 50k steps corresponding to about 45k video hours. On RoboTwin, the best Hard rate after multisource pretraining is 59.60%, against 34.56% with no pretraining. On real robots, the Hard SR and PSR averaged over three tasks rise from 33.33% and 46.90% to 48.89% and 61.43% as training video grows from 22.5k to 90k hours; the 0-hour setting completes no task. Pretraining for 75k steps on Ego and UMI, then continuing for 25k steps on robot data, gives a RoboTwin Hard rate of 61.52% (Table 10).
Main results: 50 RoboTwin tasks and three real-robot tasks
Each of the 50 RoboTwin 2.0 tasks has 50 Clean demonstrations, 2,500 in total. Starting from the pretrained checkpoint, one model is post-trained jointly on all tasks for 20k steps, and main results run 100 episodes per task and setting. Part of Table 11:
| Model | RoboTwin Easy | RoboTwin Hard | Real Easy SR / PSR | Real Hard SR / PSR |
|---|---|---|---|---|
| SWAM (Ours) | 88.10% | 61.56% | 56.67% / 68.57% | 40.00% / 50.36% |
| OpenWAM | 89.30% | 48.70% | 37.78% / 46.35% | 23.33% / 31.67% |
| π0.5 | 70.66% | 46.00% | 50.00% / 71.87% | 20.00% / 37.78% |
| LingBot-VA | 77.04% | 36.16% | 41.11% / 53.61% | 10.00% / 24.01% |
| Fast-WAM | 77.76% | 1.92% | 35.56% / 49.33% | 6.67% / 14.52% |
On RoboTwin, SWAM’s Hard rate is 12.86 percentage points above OpenWAM’s and its Easy rate is 1.2 points below; its average over the two settings is 74.83%. The real-robot tests use a bimanual Flexiv Rizon 4 for cup stacking, towel folding and gift-in-box. For each task, a separate model is adapted from upstream pretrained weights for 50k steps, and all models use the same task data. Easy changes only the positions of familiar objects; Hard changes the objects, the background or both. Each task and setting uses 10 layouts fixed in advance and shared across models, with 3 trials per layout, 30 in total; actions run synchronously at 30 Hz. Under Hard, SWAM’s SR is 16.67 percentage points above OpenWAM’s and its PSR 12.58 points above π0.5’s; under Easy, π0.5’s PSR (71.87%) is higher than SWAM’s (68.57%). The paper also shows two executions with new objects and scenes, fruit placement and a rainbow cup stack, without success rates.
The sample sizes to work out before reading these gaps
The following is this site’s estimate from the evaluation counts in the paper, not a number from the paper. Each ablation setting is 50 tasks × 25 episodes = 1,250 episodes. Treating episodes as independent trials, at a success rate near 50% the standard error is about 1.4 percentage points and the 95% Wald half-width about 2.8 points. For the difference between two independently evaluated settings, the 95% half-width is about √2 × 2.8 ≈ 3.9 points. On that scale, the gap between the 1:1 and 1:2 ratios (0.56 points), the drop from shuffled captions (1.52), the gap between episode-level and subtask-level captions (1.92) and the gap between equal sampling and UMI only (2.56) all fall within sampling noise. The 4.88 from the architecture change exceeds 3.9, but only by a modest margin. The drops from adding external image features (6.56–9.76) and the 25.04 between pretraining and none are clearly larger. This estimate covers only the sampling of evaluation episodes, not variation from training seeds, and the text this site read reports no multi-seed results or confidence intervals.
The main results use 5,000 episodes per setting; the standard error of the 61.56% Hard rate is about 0.7 percentage points, small next to a 12.86-point lead. On real robots each model and setting has 3 tasks × 30 trials = 90 trials. Because every task has the same number of trials, 40.00% and 23.33% correspond to 36/90 and 21/90 successes, and the 95% Wald half-width for a single arm near 40% is about 10 percentage points. All models share the same layouts, so a paired comparison by layout would be tighter, but the paper gives no per-layout results. Physical AI Lab 2.8 works through these intervals by hand.
Limitations
- This is an arXiv preprint and every number is author-reported [R]; this site has not reproduced them. The paper says it releases the curated data, training code and model weights. This site has not opened or checked those releases, and the text it read does not say item by item whether the release includes the 15,284.9 hours of in-house UMI data.
- The caption-detail (Table 4), action-label coverage (Table 6), ratio (Table 8) and architecture (Table 9) ablations each compare settings under one fixed budget (such as 20k or 30k steps) and one configuration, and their conclusions hold only for those settings; the text does not give the pretraining step count for Table 5. Captions were generated by Qwen 3.8 Max and Seed 2.1 Pro; the paper does not test other captioning models.
- The real-robot scaling curve gives a Hard SR of 48.89% at 90k hours, while Table 11 lists SWAM’s real-robot Hard SR as 40.00%. On RoboTwin, the last row of Table 10 (88.08%, 61.52%) also differs slightly from Table 11 (88.10%, 61.56%). The text this site read does not say which checkpoint the main table uses. The scaling study uses Ego and UMI data, while the main table is described as pretrained on the full multisource dataset.
- Table references in the text do not always match the table captions; for example, the caption experiment is described as Table 6, but its caption is numbered Table 4. This article cites the caption numbers.
- The real-robot results come from one platform and three tasks, with a separate post-trained model per task. Fruit placement and the rainbow cup stack are examples, not statistics. Neither the simulation nor the lab real-robot success rates are a hardware safety validation.
- Limitations stated by the authors: the method offers limited support for complex reasoning and explicit multi-stage planning, and its limited historical context may hinder long-horizon tasks; they plan to explore memory mechanisms and hierarchical planning.
Links
- SWAM: Scaling World Action Models with Cross-Modal Alignment (arXiv 2611.12870)
- Full paper in HTML
- Physical AI Lab 1.5: Where Does Data Come From? Working Out Demonstration Time from SO-101 Metadata, Then Separating the Units and Action Spaces of Three Sources
- Physical AI Lab 2.8: Reproducible Evaluation — Working Out Wald and Wilson Intervals by Hand, Sizing a Run from 1/√n, Then Bounding the Failure Rate after Zero Failures
- Frontier Radar | HWAM: a humanoid’s reference action is not the motion it performs, so generate the post-execution body state alongside it
- Physical AI Lab: a physical AI course where every number can be checked (course outline)



