Physical AI Lab 3.2: World Models — Making Data with Cosmos, Writing Down What Each Kind of Generation Is Conditioned On, Putting 78.3% Back in Its Human Evaluation, Then Setting Acceptance Rules for Generated Data

World Models — Making Data with Cosmos, Writing Down What Each Kind of Generation Is Conditioned On, Putting 78.3% Back in Its Human Evaluation, Then Setting Acceptance Rules for Generated Data. English video, reading, self-checks and sources in one lesson, updated 2026-11-08.

Lesson video

Narration is AI-generated. Video chapters, transcript and sources

Transcript

Transcript

UNIT-01 asks

A robot can't keep breaking parts in the real world, over and over. So where does it practise? One answer is to practise in a virtual world. This episode is about NVIDIA's Cosmos. How it builds a world, and uses it to generate training data.

What is a world model?

First, what is a world model? NVIDIA's paper puts it plainly. Physical AI has to be trained in a digital world first. It needs two digital twins. One of the robot itself, and one of the world it lives in. That second one is the world model. A model of the world does one thing. Give it past frames, plus an action or an instruction. It predicts what the next moment looks like. One thing to be clear about. A world model is not a rendering engine. It doesn't run on rules people wrote. It learns physics from video. There's one more key part, the video tokenizer. It squeezes video into a string of compact tokens. That's what lets the model digest it. Models like this are fed on oceans of video. NVIDIA's paper says it used twenty million hours of raw video. Cut into about a hundred million clips. Why so much? Because the variety of the data decides how many kinds of physics this virtual world has seen. The more it has seen, the less reality can trip it up. Its ambition is to be a general simulator of the physical world. Any robot can run through it first. Run through it first, then go to the real robot.

Predict: predicting the future

The first tool is Cosmos Predict. As the name says, it predicts the frames that come next. It also has pre-training and post-training. First, oceans of video for general knowledge. Then a little target data, for fine-tuning. How does it connect to a robot? Feed the action to the model as a condition. It predicts the frames that follow from that action. There are two approaches. NVIDIA's paper describes the diffusion line first. It comes in two sizes, seven billion and fourteen billion. The other line is autoregressive. Like a language model, it guesses one piece after another. Each line has its strengths. Models like this can go on clip after clip. Continuing autoregressively. Short clips roll into a long video. How much compute did training take? According to NVIDIA's paper, ten thousand H. one hundred chips, for three full months. What good is it for robots? NVIDIA reports a human evaluation of robot instruction video prediction. The fine-tuned model got seventy-eight point three percent overall preference, against thirteen percent for the baseline. The paper also lists some possible uses. Planning, for one. Let the world model rehearse a few actions, then pick the best one. It could be the examiner, and the environment, so a policy is tested and trained inside it first. But the paper says it has not yet shown results for these uses. If that works, we could practise and test the policy in an imagined world first!

Transfer: transfer and sim2real

Why do we need Transfer too? Simulation is cheap, but it looks fake. Reality looks right, but it's slow and expensive. The second tool is Cosmos Transfer. NVIDIA's paper says you give it segmentation maps, depth and edges. It generates the matching realistic frames. Its control is adaptive. Different regions can get different strengths. Wherever you want it strict, it stays strict. What's the use? Frames from simulation often look fake. Transfer can repaint them to look real. The paper calls this world-to-world transfer. The same scene, in a new style or new weather, generated again. The paper also uses it to enrich self-driving data. Back to Predict, which the paper fine-tunes to follow camera commands. Like in a game, forward, back, turn left, turn right. So you can walk into the generated world. The paper also fine-tunes Predict for driving-related tasks. What it generates is driving footage. How fast? The Transfer paper says with a full rack of top-end GPUs, it can reach real-time generation at best. NVIDIA also gave it guardrails. Harmful inputs and outputs both get blocked. That's required homework before release. Join the two tools and you get a data pipeline. Simulation gives the structure. Transfer gives the look. Predict fills in the future. Generated data is cheap and varied. As many lighting setups and placements as you want, you generate. A real robot could never go that fast. Back to our example. Putting the cup into the basket, rehearsed first in a generated world. Change the placement, and try again. The paper also proposes using a model of the world as the starting point for a policy. First learn how things in the world move, then learn how to act. Here too, the paper has no results yet. What's still unsolved: however real the frames look, that isn't real physics. The paper itself says the world-model problem is still far from solved. It makes silly mistakes, too. Objects appear from nowhere, and vanish into nothing. Or they defy gravity, and pass right through the table. How do you tell if a model of the world is any good? That alone is hard. Human ratings are subjective, and metrics may not match either. So Cosmos gives us two things. Predict predicts the future, Transfer changes the style. Together, they make training data. Remember one line. It's a data amplifier, not a source of ground truth. It can save real-robot time, but not the checking. The longer-term use is a closed loop. A policy tries and fails in the world, and uses the feedback to improve itself. That's exactly next episode. Let's recap, three points. One, a model of the world predicts the future. Two, Predict generates, Transfer transfers. Three, looking real isn't the same as the right physics. Practise again and again in an imagined world, then come back to reality. That's why we build worlds! So how does a policy get stronger by trial and error? See you next time!

Evidence and sources

20 million hours · of raw video used to pre-train Cosmos, cut into about 100 million clips (NVIDIA) [R] · arXiv 2501.03575 · Cosmos WFM Platform (NVIDIA)

7 billion / 14 billion · parameters, the two sizes of the Cosmos-Predict1 diffusion world model (plus autoregressive 4 billion and 12 billion, NVIDIA) [R] · arXiv 2501.03575 · Cosmos WFM Platform (NVIDIA)

10,000 H100s · in the GPU cluster that trained all the paper's world foundation models, over three months (NVIDIA) [R] · arXiv 2501.03575 · Cosmos WFM Platform (NVIDIA)

78.3% · overall human-rater preference for the fine-tuned Cosmos-Predict model in robot instruction video prediction (baseline 13.0%, NVIDIA) [R] · arXiv 2501.03575 · Cosmos WFM Platform (NVIDIA)

Segmentation · depth · edges · are the spatial control inputs Cosmos-Transfer uses to generate worlds (NVIDIA) [R] · arXiv 2503.14492 · Cosmos-Transfer1 (NVIDIA)

Real time (top-end rack) · world generation on one NVIDIA GB200 NVL72 rack (NVIDIA, peak configuration) [R] · arXiv 2503.14492 · Cosmos-Transfer1 (NVIDIA)

Practice and self-check

The quiz is public and explains each answer as you go; answers to the guide's exercises are for signed-in members.

Horizontal bar chart on a light background, titled "Overall preference, robot instruction videos", with a source line at the bottom. Two row labels on the left; on the right, two solid bars extend from the same vertical zero line, their lengths in proportion to the values: the first row, "Cosmos Diffusion 7B Video2World (instruction-tuned)", has the longer bar, labelled 78.3%; the second row, "VideoLDM-Instruction (baseline)", has the shorter bar, labelled 13.0%.
One human evaluation reported in the platform paper: robot instruction-based video prediction, with a baseline fine-tuned on the Cosmos-1X dataset, and 10 raters comparing anonymous videos in pairs over 23 test episodes; this is the preference on the "overall" dimension. It rates whether a generated video is reasonable enough for the robot to plan from, not task success and not physical correctness; the two numbers add up to less than 100%, and the passages read do not explain the difference. Author-reported numbers. Source: arXiv 2501.03575 v1.

The video for episode 3.2 covers how NVIDIA’s Cosmos uses world models to make training data: a world model takes past frames plus an action or instruction and predicts the next moment; Cosmos Predict continues a video forward, and Cosmos Transfer generates realistic frames from control inputs such as segmentation, depth and edges; however real the frames look, that is not the same as the right physics. The number cards in the video come from two NVIDIA arXiv papers: the Cosmos World Foundation Model Platform (2501.03575) and Cosmos-Transfer1 (2503.14492). This page is the reading and exercises for the episode. It does not repeat the transcript; instead it does three things the video only touches on. It writes the paper’s definition as one formula and spells out, for each kind of generation, what it is conditioned on and what it outputs. It puts the paper’s two robot evaluations, the 78.3% human preference and the four frame metrics on the Bridge dataset, back with their own models, data and scoring, to see what they measure and what they do not. And it sets acceptance rules for a fictional batch of generated cup-placing data. One rule runs through the whole page: a use the paper lists is not a result the paper reports, and an evaluation number belongs only to the model, dataset and scoring that produced it.

The video, transcript and sources for this lesson are at the top of this page; the season plan is in the course outline. The previous lesson, Physical AI Lab 3.1: Inside VLAs — Encoding One Cup Motion into 256-Bin Tokens, Then Labelling the Scope of Three Models’ Parameter Counts and Rates, took the policy model apart. This lesson turns the other way: it leaves the policy alone and builds the world and the data the policy practises on. Why simulation is cheap, and which experimental conditions simulation numbers need, was covered in Physical AI Lab 0.4: Ten Thousand Tries in a Virtual World — Putting the Conditions Back on Three Simulation Numbers. Narrowing the gap between simulation and the real robot with domain randomization and system identification was covered in Physical AI Lab 2.5: Sim2real with Domain Randomization and System Identification — Putting Two Papers’ Numbers Back in Their Setups, Then Working Out Friction by Hand from One Push of a Cup; Cosmos Transfer deals with another side of it: simulated frames look fake. The acceptance rules in section 4 use paired comparisons and intervals, worked out in Physical AI Lab 1.8: Evaluation — Splitting a “Net Gain of 2” into a Paired Table, Then Checking the Failure Labels and Random Seeds and Physical AI Lab 2.8: Reproducible Evaluation — Working Out Wald and Wilson Intervals by Hand, Sizing a Run from 1/√n, Then Bounding the Failure Rate after Zero Failures; this lesson does not derive them again. This lesson reads the Cosmos 1.0 platform paper and Cosmos-Transfer1. The site’s radar post Frontier Radar | NVIDIA Cosmos 3 on AWS: reasoning, video and actions in one forward pass, and the validation conditions attached to instance sizing is about Cosmos 3, a different version, and the numbers from the two cannot be applied to each other. The next episode, 3.3, covers RL post-training and the self-improvement loop.

The public quiz’s 5 questions come with full explanations. Signed-in readers can do three exercises: write down the past observation, condition, output and evidence status for five fictional generation requests; audit eight fictional generated clips and work out the pass rate and the false-alarm rate on real clips; and review a fictional conclusions memo, using paired results from three seeds to judge whether the generated data really helped. Each comes with a step-by-step worked solution, common mistakes and a 20-point rubric.

1. One formula, several conditions: how the paper defines a world model

Both papers are arXiv preprints, R level in this course’s grading, and both are NVIDIA’s reports on its own models. The structures and numbers below are author-reported; this site has not run any of these models. The platform paper’s abstract opens like this:

Physical AI needs to be trained digitally first. It needs a digital twin of itself, the policy model, and a digital twin of the world, the world model.

Section 2 of the paper writes a world foundation model as a mapping: a model W predicts the observation at time t + 1 from the past observations from time 0 to t and the current perturbation. In code-style notation, x̂[t+1] = W(x[0:t], c[t]). The paper says that in Cosmos the past observations are RGB video, and the perturbation can take many forms:

It can be an action taken by the Physical AI, a random perturbation, a text description of the perturbation, etc.

The video’s “give it past frames plus an action or instruction” is this formula. For each model in the paper, ask two things first: what is x, and what is c? The table below is this lesson’s summary of the passages read, not a table from the paper:

Model or post-trainingPast observation xCondition cOutputWhere
Cosmos-1.0-Diffusion-7B / 14B-Text2Worldnonetext promptvideoSection 5, Table 10
Cosmos-1.0-Diffusion-7B / 14B-Video2Worldinput video (current observation)text promptfuture videoSection 5
Cosmos-1.0-Autoregressive-4B / 12Bcurrent videonone; video onlyfuture videoSection 5
Cosmos-1.0-Autoregressive-5B / 13B-Video2Worldpast videotext prompt (inferred from the two-step pre-training in Section 2.2)future videoSection 2.2, Table 10
Camera-control post-trainingnot stated in the passages readcamera posea virtual world you can explore by moving the viewpointIntroduction, Section 6.1
Robot instruction-based video predictionvideolanguage instructionfuture videoSection 6.2
Robot action-based next-frame predictionvideorobot actionnext frame, chained autoregressively into videoSection 6.2 (Bridge)
Autonomous-driving-related tasksnot stated in the passages readnot stated in the passages readnot stated in the passages readIntroduction, Section 6.3
Cosmos-Transfer1not stated in the abstract readmultiple spatial control inputs such as segmentation, depth and edge, weightable by locationworld simulations (video)2503.14492 abstract

Both the diffusion and the autoregressive models turn video into tokens before working on it: the diffusion family uses continuous tokens (Cosmos-1.0-Tokenizer-CV8x8x8) and the autoregressive family uses discrete tokens (Cosmos-1.0-Tokenizer-DV8x16x16). The paper says these tokenizers are causal: the tokens for the current frames do not depend on future observations.

In the platform paper, Transfer first appears as a proposal: Section 2.1 says a world model can be fine-tuned to be conditioned on rendering metadata such as depth or semantic maps, for the Sim2Real use case. Cosmos-Transfer1 is the later paper that does exactly this. Its abstract says:

We introduce Cosmos-Transfer, a conditional world generation model that can generate world simulations based on multiple spatial control inputs of various modalities such as segmentation, depth, and edge.

It allows weighting different conditional inputs differently at different spatial locations.

The following is a fictional teaching example. Robot arm D puts a cup into a basket. To see “the same motion under different lighting”, you use Transfer: c is the segmentation and depth maps of that motion, with higher weight where the cup and basket are and looser control on the background, and the output is new realistic frames. To see “what happens if the gripper lets go 0.2 s early”, you use action-conditioned Predict: x is the frames before the release, c is the edited action sequence, and the output is the predicted frames that follow. The first changes the appearance, the second changes the future. Both give you only frames; whether the cup really falls according to physics in them has to be checked separately.

Limitations

  • Both papers are NVIDIA arXiv preprints, R level; the structures, sizes and evaluation numbers are the authors’ reports on their own models. This site has not replicated them or run any model.
  • This lesson read the platform paper’s HTML v1 abstract, introduction, Section 2, the dataset passage at the start of Section 3, the start of Section 5 with Table 10, and part of the robot evaluation in Section 6.2; only the abstract of Cosmos-Transfer1 was read. The passages read give neither the total number of comparisons nor an interval for 78.3%, and do not describe the Cosmos-1X dataset.
  • The following are this lesson’s summaries, arithmetic or inferences: the conditions table (in which Autoregressive-5B / 13B-Video2World being text-conditioned is inferred from the two-step pre-training in Section 2.2); 78.3 plus 13.0 making 91.3; the 68% and 27% FVD reductions; the relation between the training-data mix and your scene; setting Transfer weights by region; and the whole set of acceptance rules in section 4.
  • The gripper example in section 1, and robot D, team Q, requests R1–R6, clips G1–G8, the audit thresholds, the pixel areas, the real clips and the success counts in the exercises, are fictional; they are not data from any paper or system.
  • The numbers in this lesson belong only to the Cosmos 1.0 platform paper and Cosmos-Transfer1; the Cosmos 3 covered in the site’s radar is a different version.
  • Paper calculations, simulation and audits of generated frames are not hardware safety validation.

Links

Watch this lesson (English video) · 3.2