Lesson video
Narration is AI-generated. Video chapters, transcript and sources
Transcript
Transcript
UNIT-01 asks
To teach a robot to pick up a cup, you first have to show it lots of demonstrations. So who makes them? This episode sorts out the three sources of a robot's action data: teleoperation, human video, and synthesis. Each one has its own cost.
Teleoperation
The most direct way is for a person to lead. Someone does the task once, through the robot, and the motion is recorded as it goes. A common setup uses two arms, a leader and a follower. You move the leader arm, and the robot's follower arm moves with it. That's how human skill gets passed across. Here's an example: the SO-101 pick-and-place dataset. Fifty demonstrations, each with two cameras. Thirty frames a second, plus readings from six joints. Two cameras: one looking down, one from the side. The six joint readings are the action labels. Watch the picture, and you can see what it's learning. Skip ahead to the moment it grasps the brick. The gripper slowly closes, and all six curves move together. This is what one clean grasp looks like. This method is the most reliable, and the motion is clean. But it's slow and expensive. Every single episode has to be recorded by a real person. So what can you do? One paper pooled everyone's data. Twenty-one institutions, twenty-two kinds of robots, and more than five hundred skills. So one lab at a time is too slow. Put it all together, and there's finally enough!
Human video
There's an even cheaper way. Skip the robot, and just watch people work. People wear cameras and film first-person video. Why focus on people? Because people are working every single day. Video like this is almost endless. How big can it get? In early 2026, one paper used more than twenty thousand hours of this video to pre-train robots. The same paper found something else. The more human video it was fed, the better the model learned. It's almost like a law. But raw video isn't enough. First you have to label the actions, so the model has something to learn from. That step alone takes real work. There's another hurdle. A human hand isn't a robot's hand. The video has no precise joint angles, and no forces. You can learn from watching people, but first you have to translate human hands into robot hands!
Synthesis
The third path is to make data in simulation. Remember the simulator from last episode? Let it act things out on its own, and it can produce a lot. How? First, a real person does a few demonstrations. Then the system automatically moves them into new scenes and new positions. The upside is variety. A new position, a new object, all generated automatically. Real people recording one by one could never keep up. How much does it save? Researchers used one system to turn about two hundred human demonstrations into more than fifty thousand, covering eighteen tasks. The savings are real. But no matter how much you synthesise, one question remains. Does it actually look like the real thing? I get all three paths now. But is more data always better? The answer: not necessarily. Data doesn't just need to be plentiful. It needs to be varied, and cover all kinds of situations. Piling up quantity alone stops paying off. Line the three paths up. Teleoperation is the most precise, human video the most plentiful, synthesis the most flexible. None of them can do it all alone. In practice, the three are often mixed. How to balance them, how to guarantee quality: there's still no settled answer. On screen, we step through a demonstration frame by frame. The two cameras and the six joint curves all move together. Where it drifts off, you can see at a glance. Let's recap, three points. One: teleoperation is the most direct, with clean motion, but it's slow and expensive. Two: human video comes in huge volume, but human hands aren't robot hands. Three: synthesis can multiply data, but you must keep it true to life. So data doesn't fall from the sky. People lead it one episode at a time, and machines help multiply it! Now we have data. But how does a robot actually learn actions from demonstrations? See you next time!
Evidence and sources
Practice and self-check
- Interactive quiz5 questions, explained as you answer
- Browser labno install: drag and tune it yourself
- One-page summaryprintable; opens in a new tab
The quiz is public and explains each answer as you go; answers to the guide's exercises are for signed-in members.
The episode 1.5 video sets out three sources of robot action data. Teleoperation gives the cleanest motion, but it is slow and expensive. Human video comes in far greater volume, but a human hand is not a robot’s hand, and the video still needs labelling and aligning. Synthesis can multiply a few real demonstrations into huge amounts, but it has to stay true to life. The video cites four pieces of evidence: the SO-101 pick-and-place dataset has 50 demonstrations, each with two cameras at 30 frames a second; Open X-Embodiment pools data from 22 kinds of robot into one dataset; EgoScale used over 20,854 hours of first-person human video; MimicGen generated more than 50,000 demonstrations from about 200 human ones. This page is the episode’s reading and exercises. Rather than repeat the transcript, it first opens the SO-101 dataset’s metadata file and works out how long the 50 episodes last in total and which numbers each frame stores. It then looks at why data from the three sources cannot simply be poured into one training set: their units, action spaces and what counts as “one demonstration” all differ. Finally, a set of hypothetical numbers separates “how many were generated” from “how many situations are covered”. One rule runs through the whole page: a number describes only its own dataset, in its own unit.
The video, transcript and sources for this lesson are at the top of this page; the season plan is in the course outline. The previous lesson, Physical AI Lab 1.4: A Map of Simulators — Counting the Cost of the Timestep, Finding Where Contact Splits Two Runs, and Pinning Three Speed-ups to Their Baselines, ended by asking where the action data a robot practises on comes from in the first place. This lesson picks up there. The scale figures for Open X-Embodiment and EgoScale were already covered in Physical AI Lab 0.2: Why Now? The Three Curves of Data, Models, and Compute. This lesson does not repeat them and adds only two points relevant to this episode: how the action spaces of multi-robot data are aligned, and which quantity EgoScale’s scaling relation is about.
The public quiz’s 5 questions come with full explanations. Signed-in readers can work through three exercises: read a fictional dataset’s metadata, work out its durations and split it into training and test sets by whole episode; keep a ledger for a fictional plan that mixes four kinds of data, and find the action spaces that have the same number of dimensions but different meanings; review a fictional memo that cites this episode’s four pieces of evidence and rewrite it sentence by sentence. Each comes with a step-by-step solution, common mistakes and a 20-point rubric.
1. Four pieces of evidence, and what each one counts
The video’s four figures come from four documents, and each counts something different in a different unit:
| In the video | Source | What it counts | Unit | Level |
|---|---|---|---|---|
| 50 episodes, two cameras, 30 frames a second | Metadata of LeRobot’s svla_so101_pickplace dataset (pinned commit f641879) | Demonstration episodes of one pick-and-place task (the video calls them teleoperated; the metadata does not record how they were collected) | Episodes; frames per second | O (dataset metadata) |
| 22 kinds of robot, 21 institutions, 527 skills | Open X-Embodiment (arXiv 2310.08864) | Robot embodiments, collaborating institutions and skills pooled into one dataset | Kinds; institutions | R (authors’ report) |
| Over 20,854 hours | EgoScale (arXiv 2602.16710) | Action-labelled first-person human video | Hours | R (authors’ report) |
| More than 50,000 generated from about 200 | MimicGen (arXiv 2310.17596) | Human demonstrations and automatically generated demonstrations | Demonstrations; across 18 tasks | R (authors’ report) |
The corresponding original sentences from the three papers, each excerpted separately:
We assemble a dataset from 22 different robots collected through a collaboration between 21 institutions, demonstrating 527 skills ( 160266 tasks).
We train a Vision Language Action (VLA) model on over 20,854 hours of action labeled egocentric human video, more than 20 times larger than prior efforts, and uncover a log linear scaling law between human data scale and validation loss.
We use MimicGen to generate over 50K demonstrations across 18 tasks with diverse scene configurations, object instances, and robot arms from just ~200 human demonstrations.
Three points to keep in mind when reading this table:
- Different units cannot be ranked against each other. Episodes, hours, kinds and demonstrations are four different units. Dividing 20,854 hours of human video by 50 robot episodes does not tell you “how many times more” data one has: every frame of the robot episodes stores joint readings, while human video, as the video puts it, has no precise joint angles and no forces.
- “22 kinds” counts robot embodiments, not demonstrations. Section III of the Open X-Embodiment paper says separately that the dataset contains 1M+ real robot trajectories, pooled from 60 existing datasets from 34 robotic research labs around the world. 21 institutions and 34 labs are also two different counts; do not mix them up.
- The 50,000-plus are generated; the human demonstrations number only about 200. The video’s number card labels MimicGen’s result as in simulation, reported by the paper. A generated demonstration and a human-recorded one are not the same kind of “demonstration”; section 4 works through this again.
Sign in to keep reading: the rest of this lesson, its self-check questions and the answers.
Limitations
- The figures from Open X-Embodiment (arXiv 2310.08864, 2023-10-13), MimicGen (arXiv 2310.17596, 2023-10-26, CoRL 2023) and EgoScale (arXiv 2602.16710, 2026-02-18) are authors’ reports, level R; EgoScale is a preprint. This site has not independently reproduced any of them. This lesson relies on the abstract and opening sections of Open X-Embodiment (the excerpt of section IV breaks off after roll) and on the abstracts of MimicGen and EgoScale, not on supplementary material.
- The SO-101 values come from
meta/info.jsonof the svla_so101_pickplace dataset at pinned commit f641879, level O. This lesson read only that metadata file and did not download or check the data files or videos. The metadata describes fields and counts; it does not describe the quality of the demonstrations, whether they succeeded, how they were recorded, or what collection cost. - 238.78 frames, 7.96 s, 397.97 s, 23,878 images, 307,200 pixels, 71,634 numbers and roughly 250 times are this lesson’s arithmetic on the figures listed, not numbers stated by the sources. In the log-linear relation, a and b are symbols written for this lesson, and 200, 2,000 and 20,000 hours are examples, not EgoScale data points.
- Team T, cups C1–C4 and plans G1 and G2 in section 4, and cup_place_demo, team K, robot arm K1, data S1–S4 and memo M1–M4 in section 5, are fictional exercises, not data from any dataset, team or system. Holding out by factor, splitting by whole episode and “keeping a ledger” are teaching suggestions, not conclusions from any source; neither paper calculations nor simulation results are hardware safety validation.
- This article has not checked the implementation of the video’s lab page (frame-by-frame browsing of an SO-101 demonstration).
Links
- Physical AI Lab: a physical AI course where every number can be checked (course outline)
- Physical AI Lab 1.4: A Map of Simulators — Counting the Cost of the Timestep, Finding Where Contact Splits Two Runs, and Pinning Three Speed-ups to Their Baselines
- Physical AI Lab 1.1: Coordinate Frames and Kinematics — Working Out by Hand How a Two-Link Arm Reaches the Cup
- Physical AI Lab 0.2: Why Now? The Three Curves of Data, Models, and Compute
- Hugging Face: lerobot/svla_so101_pickplace dataset
- meta/info.json of svla_so101_pickplace (pinned commit f641879)
- Open X-Embodiment: Robotic Learning Datasets and RT-X Models (arXiv 2310.08864)
- EgoScale (arXiv 2602.16710)
- MimicGen (arXiv 2310.17596, CoRL 2023)
