Lesson video
Narration is AI-generated. Video chapters, transcript and sources
Transcript
Transcript
UNIT-01 asks
People have studied robots for decades. So why, in just the last couple of years, can they suddenly use their hands? The answer is three curves, all rising at the same time: data, models, and compute. This episode, we take them one by one.
Data
Teaching a robot is like training an apprentice. It has to watch many, many times. But in the past, every lab kept only a small pile of data from its own robots. In 2023, one paper pooled the data of 21 institutions. Twenty-two kinds of robots, and more than 500 skills. The bigger step was learning by watching people work. In early 2026, one paper used more than twenty thousand hours of video, shot by people wearing cameras while they worked, to lay the groundwork for robot models. Robots really can learn from people!
Models
Data isn't enough. You also need a model that can digest it. The large models behind chat AI have seen oceans of pictures and text. They know a cup, and they know a table. In 2023, a paper from Google had an idea. Write the robot's actions as a string of words too, so the model looks at the picture and simply says the action. The research team reports that in six thousand real-robot trials, even with things it had never seen, it still did better. By 2026, models like this can be downloaded openly and used commercially. In April, a chip company released a model for humanoid robots, with three billion parameters. And it was pre-trained on that very human video: more than twenty thousand hours of it. This is where data and models join up. A model that describes pictures has learned to use its hands!
Compute
Last comes compute. Training a model takes computing, and so does the robot's own body while it works. Now, training. The training compute of frontier AI models grows four to five times a year. That's the figure from a research group. Robot models stand on the shoulders of these large models. The computing module on the robot is getting stronger fast, too. Officially, at a specific precision, its peak compute is seven and a half times the last generation. With the compute in place, large models can finally ride on robots, seeing, thinking, and acting all at once. Taken one at a time, none of the three curves is new. What's new is that they're rising at the same time, pushing each other forward. But a rising curve doesn't mean the problem is solved. As we saw last time: when something goes wrong halfway through, robots still often can't recover. Let's recap, three points. One, data: from each lab keeping its own, to learning by watching people. Two, models: large models that have seen oceans of pictures and text have learned to output actions. Three, compute: it has caught up, both for training and on the robot itself. So robots didn't suddenly get smart. Three conditions finally matured together! So what actually happens inside one second of a robot's life? See you next time!
Evidence and sources
Practice and self-check
The quiz is public and explains each answer as you go; answers to the guide's exercises are for signed-in members.
Robots have been studied for decades, so why is it only in the last couple of years that they suddenly seem able to do things with their hands? The answer in episode 0.2 of Physical AI Lab is three curves: data, models, and compute, rising over the same period and pushing each other along. This article is the text version of the episode. It lists the source of every number and marks which ones are self-reported by authors or vendors.
Note: this article can be read on its own and reuses the example from episode 0.1: pick up the cup on the table without knocking over the water bottle next to it.
1. Data: from separate collections to watching people work
Teaching a robot is like training an apprentice: first it has to watch many demonstrations. The old problem was that each lab collected only a small amount of data from its own robot, with different formats, robot models, and tasks, so the data could not be shared.
Pooling data across institutions. The Open X-Embodiment paper, submitted in October 2023, pooled data from 21 institutions. According to the paper, the dataset spans 22 different robots and 527 skills (160,266 tasks); the authors trained RT-X models on it and reported positive transfer across platforms.
Learning directly from people at work. However much robot data is pooled, it still has to be collected one teleoperated episode at a time. A bigger step is learning from human video. The EgoScale paper, submitted in February 2026, pretrains a vision-language-action model on 20,854 hours of action-annotated egocentric video (video recorded from a person’s own point of view with a head-worn camera). According to the paper, this is more than 20 times larger than previous comparable work; data scale and validation loss follow a log-linear scaling relationship; and on a 22-degree-of-freedom dexterous hand, the final policy’s average success rate is 54% higher than a baseline without human-data pretraining (the paper does not say whether this is relative or in percentage points). 20,854 hours is about 2.4 years of nonstop playback.
Sign in to keep reading: the rest of this lesson, its self-check questions and the answers.
Limitations
- All three numbers from papers (22 robots, 20,854 hours, 6,000 trials) are self-reported by the authors; EgoScale is a 2026 preprint and has not yet been peer-reviewed.
- That GR00T N1.7 “allows commercial use” comes from the wording of NVIDIA’s blog post; check the license on the model page for the specific terms before using it.
- The “three curves” are a framework to aid understanding, not a proof of causation. Progress in robot capabilities also depends on factors this episode does not cover, such as the robot hardware itself, simulation, and evaluation methods.
Links
- Open X-Embodiment: Robotic Learning Datasets and RT-X Models (arXiv 2310.08864)
- EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data (arXiv 2602.16710)
- RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control (arXiv 2307.15818)
- NVIDIA Isaac GR00T N1.7 (Hugging Face blog)
- Epoch AI: Training compute of frontier AI models grows by 4-5x per year
- NVIDIA: Jetson Thor Now Available
