Lesson video
Narration is AI-generated. Video chapters, transcript and sources
Transcript
Transcript
UNIT-01 asks
A robot can't keep breaking parts in the real world, over and over. So where does it practise? One answer is to practise in a virtual world. This episode is about NVIDIA's Cosmos. How it builds a world, and uses it to generate training data.
What is a world model?
First, what is a world model? NVIDIA's paper puts it plainly. Physical AI has to be trained in a digital world first. It needs two digital twins. One of the robot itself, and one of the world it lives in. That second one is the world model. A model of the world does one thing. Give it past frames, plus an action or an instruction. It predicts what the next moment looks like. One thing to be clear about. A world model is not a rendering engine. It doesn't run on rules people wrote. It learns physics from video. There's one more key part, the video tokenizer. It squeezes video into a string of compact tokens. That's what lets the model digest it. Models like this are fed on oceans of video. NVIDIA's paper says it used twenty million hours of raw video. Cut into about a hundred million clips. Why so much? Because the variety of the data decides how many kinds of physics this virtual world has seen. The more it has seen, the less reality can trip it up. Its ambition is to be a general simulator of the physical world. Any robot can run through it first. Run through it first, then go to the real robot.
Predict: predicting the future
The first tool is Cosmos Predict. As the name says, it predicts the frames that come next. It also has pre-training and post-training. First, oceans of video for general knowledge. Then a little target data, for fine-tuning. How does it connect to a robot? Feed the action to the model as a condition. It predicts the frames that follow from that action. There are two approaches. NVIDIA's paper describes the diffusion line first. It comes in two sizes, seven billion and fourteen billion. The other line is autoregressive. Like a language model, it guesses one piece after another. Each line has its strengths. Models like this can go on clip after clip. Continuing autoregressively. Short clips roll into a long video. How much compute did training take? According to NVIDIA's paper, ten thousand H. one hundred chips, for three full months. What good is it for robots? NVIDIA reports a human evaluation of robot instruction video prediction. The fine-tuned model got seventy-eight point three percent overall preference, against thirteen percent for the baseline. The paper also lists some possible uses. Planning, for one. Let the world model rehearse a few actions, then pick the best one. It could be the examiner, and the environment, so a policy is tested and trained inside it first. But the paper says it has not yet shown results for these uses. If that works, we could practise and test the policy in an imagined world first!
Transfer: transfer and sim2real
Why do we need Transfer too? Simulation is cheap, but it looks fake. Reality looks right, but it's slow and expensive. The second tool is Cosmos Transfer. NVIDIA's paper says you give it segmentation maps, depth and edges. It generates the matching realistic frames. Its control is adaptive. Different regions can get different strengths. Wherever you want it strict, it stays strict. What's the use? Frames from simulation often look fake. Transfer can repaint them to look real. The paper calls this world-to-world transfer. The same scene, in a new style or new weather, generated again. The paper also uses it to enrich self-driving data. Back to Predict, which the paper fine-tunes to follow camera commands. Like in a game, forward, back, turn left, turn right. So you can walk into the generated world. The paper also fine-tunes Predict for driving-related tasks. What it generates is driving footage. How fast? The Transfer paper says with a full rack of top-end GPUs, it can reach real-time generation at best. NVIDIA also gave it guardrails. Harmful inputs and outputs both get blocked. That's required homework before release. Join the two tools and you get a data pipeline. Simulation gives the structure. Transfer gives the look. Predict fills in the future. Generated data is cheap and varied. As many lighting setups and placements as you want, you generate. A real robot could never go that fast. Back to our example. Putting the cup into the basket, rehearsed first in a generated world. Change the placement, and try again. The paper also proposes using a model of the world as the starting point for a policy. First learn how things in the world move, then learn how to act. Here too, the paper has no results yet. What's still unsolved: however real the frames look, that isn't real physics. The paper itself says the world-model problem is still far from solved. It makes silly mistakes, too. Objects appear from nowhere, and vanish into nothing. Or they defy gravity, and pass right through the table. How do you tell if a model of the world is any good? That alone is hard. Human ratings are subjective, and metrics may not match either. So Cosmos gives us two things. Predict predicts the future, Transfer changes the style. Together, they make training data. Remember one line. It's a data amplifier, not a source of ground truth. It can save real-robot time, but not the checking. The longer-term use is a closed loop. A policy tries and fails in the world, and uses the feedback to improve itself. That's exactly next episode. Let's recap, three points. One, a model of the world predicts the future. Two, Predict generates, Transfer transfers. Three, looking real isn't the same as the right physics. Practise again and again in an imagined world, then come back to reality. That's why we build worlds! So how does a policy get stronger by trial and error? See you next time!
Evidence and sources
Practice and self-check
- Interactive quiz5 questions, explained as you answer
- One-page summaryprintable; opens in a new tab
- Download the reproduction labzip: one command, with this lesson's recorded run
The quiz is public and explains each answer as you go; answers to the guide's exercises are for signed-in members.

The video for episode 3.2 covers how NVIDIA’s Cosmos uses world models to make training data: a world model takes past frames plus an action or instruction and predicts the next moment; Cosmos Predict continues a video forward, and Cosmos Transfer generates realistic frames from control inputs such as segmentation, depth and edges; however real the frames look, that is not the same as the right physics. The number cards in the video come from two NVIDIA arXiv papers: the Cosmos World Foundation Model Platform (2501.03575) and Cosmos-Transfer1 (2503.14492). This page is the reading and exercises for the episode. It does not repeat the transcript; instead it does three things the video only touches on. It writes the paper’s definition as one formula and spells out, for each kind of generation, what it is conditioned on and what it outputs. It puts the paper’s two robot evaluations, the 78.3% human preference and the four frame metrics on the Bridge dataset, back with their own models, data and scoring, to see what they measure and what they do not. And it sets acceptance rules for a fictional batch of generated cup-placing data. One rule runs through the whole page: a use the paper lists is not a result the paper reports, and an evaluation number belongs only to the model, dataset and scoring that produced it.
The video, transcript and sources for this lesson are at the top of this page; the season plan is in the course outline. The previous lesson, Physical AI Lab 3.1: Inside VLAs — Encoding One Cup Motion into 256-Bin Tokens, Then Labelling the Scope of Three Models’ Parameter Counts and Rates, took the policy model apart. This lesson turns the other way: it leaves the policy alone and builds the world and the data the policy practises on. Why simulation is cheap, and which experimental conditions simulation numbers need, was covered in Physical AI Lab 0.4: Ten Thousand Tries in a Virtual World — Putting the Conditions Back on Three Simulation Numbers. Narrowing the gap between simulation and the real robot with domain randomization and system identification was covered in Physical AI Lab 2.5: Sim2real with Domain Randomization and System Identification — Putting Two Papers’ Numbers Back in Their Setups, Then Working Out Friction by Hand from One Push of a Cup; Cosmos Transfer deals with another side of it: simulated frames look fake. The acceptance rules in section 4 use paired comparisons and intervals, worked out in Physical AI Lab 1.8: Evaluation — Splitting a “Net Gain of 2” into a Paired Table, Then Checking the Failure Labels and Random Seeds and Physical AI Lab 2.8: Reproducible Evaluation — Working Out Wald and Wilson Intervals by Hand, Sizing a Run from 1/√n, Then Bounding the Failure Rate after Zero Failures; this lesson does not derive them again. This lesson reads the Cosmos 1.0 platform paper and Cosmos-Transfer1. The site’s radar post Frontier Radar | NVIDIA Cosmos 3 on AWS: reasoning, video and actions in one forward pass, and the validation conditions attached to instance sizing is about Cosmos 3, a different version, and the numbers from the two cannot be applied to each other. The next episode, 3.3, covers RL post-training and the self-improvement loop.
The public quiz’s 5 questions come with full explanations. Signed-in readers can do three exercises: write down the past observation, condition, output and evidence status for five fictional generation requests; audit eight fictional generated clips and work out the pass rate and the false-alarm rate on real clips; and review a fictional conclusions memo, using paired results from three seeds to judge whether the generated data really helped. Each comes with a step-by-step worked solution, common mistakes and a 20-point rubric.
1. One formula, several conditions: how the paper defines a world model
Both papers are arXiv preprints, R level in this course’s grading, and both are NVIDIA’s reports on its own models. The structures and numbers below are author-reported; this site has not run any of these models. The platform paper’s abstract opens like this:
Physical AI needs to be trained digitally first. It needs a digital twin of itself, the policy model, and a digital twin of the world, the world model.
Section 2 of the paper writes a world foundation model as a mapping: a model W predicts the observation at time t + 1 from the past observations from time 0 to t and the current perturbation. In code-style notation, x̂[t+1] = W(x[0:t], c[t]). The paper says that in Cosmos the past observations are RGB video, and the perturbation can take many forms:
It can be an action taken by the Physical AI, a random perturbation, a text description of the perturbation, etc.
The video’s “give it past frames plus an action or instruction” is this formula. For each model in the paper, ask two things first: what is x, and what is c? The table below is this lesson’s summary of the passages read, not a table from the paper:
| Model or post-training | Past observation x | Condition c | Output | Where |
|---|---|---|---|---|
| Cosmos-1.0-Diffusion-7B / 14B-Text2World | none | text prompt | video | Section 5, Table 10 |
| Cosmos-1.0-Diffusion-7B / 14B-Video2World | input video (current observation) | text prompt | future video | Section 5 |
| Cosmos-1.0-Autoregressive-4B / 12B | current video | none; video only | future video | Section 5 |
| Cosmos-1.0-Autoregressive-5B / 13B-Video2World | past video | text prompt (inferred from the two-step pre-training in Section 2.2) | future video | Section 2.2, Table 10 |
| Camera-control post-training | not stated in the passages read | camera pose | a virtual world you can explore by moving the viewpoint | Introduction, Section 6.1 |
| Robot instruction-based video prediction | video | language instruction | future video | Section 6.2 |
| Robot action-based next-frame prediction | video | robot action | next frame, chained autoregressively into video | Section 6.2 (Bridge) |
| Autonomous-driving-related tasks | not stated in the passages read | not stated in the passages read | not stated in the passages read | Introduction, Section 6.3 |
| Cosmos-Transfer1 | not stated in the abstract read | multiple spatial control inputs such as segmentation, depth and edge, weightable by location | world simulations (video) | 2503.14492 abstract |
Both the diffusion and the autoregressive models turn video into tokens before working on it: the diffusion family uses continuous tokens (Cosmos-1.0-Tokenizer-CV8x8x8) and the autoregressive family uses discrete tokens (Cosmos-1.0-Tokenizer-DV8x16x16). The paper says these tokenizers are causal: the tokens for the current frames do not depend on future observations.
In the platform paper, Transfer first appears as a proposal: Section 2.1 says a world model can be fine-tuned to be conditioned on rendering metadata such as depth or semantic maps, for the Sim2Real use case. Cosmos-Transfer1 is the later paper that does exactly this. Its abstract says:
We introduce Cosmos-Transfer, a conditional world generation model that can generate world simulations based on multiple spatial control inputs of various modalities such as segmentation, depth, and edge.
It allows weighting different conditional inputs differently at different spatial locations.
The following is a fictional teaching example. Robot arm D puts a cup into a basket. To see “the same motion under different lighting”, you use Transfer: c is the segmentation and depth maps of that motion, with higher weight where the cup and basket are and looser control on the background, and the output is new realistic frames. To see “what happens if the gripper lets go 0.2 s early”, you use action-conditioned Predict: x is the frames before the release, c is the edited action sequence, and the output is the predicted frames that follow. The first changes the appearance, the second changes the future. Both give you only frames; whether the cup really falls according to physics in them has to be checked separately.
Sign in to keep reading: the rest of this lesson, its self-check questions and the answers.
Limitations
- Both papers are NVIDIA arXiv preprints, R level; the structures, sizes and evaluation numbers are the authors’ reports on their own models. This site has not replicated them or run any model.
- This lesson read the platform paper’s HTML v1 abstract, introduction, Section 2, the dataset passage at the start of Section 3, the start of Section 5 with Table 10, and part of the robot evaluation in Section 6.2; only the abstract of Cosmos-Transfer1 was read. The passages read give neither the total number of comparisons nor an interval for 78.3%, and do not describe the Cosmos-1X dataset.
- The following are this lesson’s summaries, arithmetic or inferences: the conditions table (in which Autoregressive-5B / 13B-Video2World being text-conditioned is inferred from the two-step pre-training in Section 2.2); 78.3 plus 13.0 making 91.3; the 68% and 27% FVD reductions; the relation between the training-data mix and your scene; setting Transfer weights by region; and the whole set of acceptance rules in section 4.
- The gripper example in section 1, and robot D, team Q, requests R1–R6, clips G1–G8, the audit thresholds, the pixel areas, the real clips and the success counts in the exercises, are fictional; they are not data from any paper or system.
- The numbers in this lesson belong only to the Cosmos 1.0 platform paper and Cosmos-Transfer1; the Cosmos 3 covered in the site’s radar is a different version.
- Paper calculations, simulation and audits of generated frames are not hardware safety validation.
Links
- Physical AI Lab: a physical AI course where every number can be checked (course outline)
- Physical AI Lab 3.1: Inside VLAs — Encoding One Cup Motion into 256-Bin Tokens, Then Labelling the Scope of Three Models’ Parameter Counts and Rates
- Physical AI Lab 0.4: Ten Thousand Tries in a Virtual World — Putting the Conditions Back on Three Simulation Numbers
- Physical AI Lab 2.5: Sim2real with Domain Randomization and System Identification — Putting Two Papers’ Numbers Back in Their Setups, Then Working Out Friction by Hand from One Push of a Cup
- Physical AI Lab 1.8: Evaluation — Splitting a “Net Gain of 2” into a Paired Table, Then Checking the Failure Labels and Random Seeds
- Physical AI Lab 2.8: Reproducible Evaluation — Working Out Wald and Wilson Intervals by Hand, Sizing a Run from 1/√n, Then Bounding the Failure Rate after Zero Failures
- Frontier Radar | NVIDIA Cosmos 3 on AWS: reasoning, video and actions in one forward pass, and the validation conditions attached to instance sizing
- Frontier Radar | WorldFailBench: when a world model evaluates a policy, check failure reason, stage and boundary, not just success rate
- NVIDIA: Cosmos World Foundation Model Platform for Physical AI (arXiv 2501.03575, HTML v1)
- NVIDIA: Cosmos-Transfer1: Conditional World Generation with Adaptive Multimodal Control (arXiv 2503.14492)



