Lesson video
Narration is AI-generated. Video chapters, transcript and sources
Transcript
Transcript
UNIT-01 asks
Show a robot a few demonstrations, and it's got it? Then why, after watching so many times, does it sometimes still not learn? This episode is about imitation learning: how robots learn actions from demonstrations, why copying isn't that simple, and how many demonstrations are enough.
Copying
Start with the simplest idea. See the cup on the left, reach left. Gripper lined up, close it. Put simply, it's matching what you see to what you do. This most basic approach is called behaviour cloning. See enough examples, and the patterns emerge on their own. Just like learning to ride a bike. But there's a trap hiding here. Once the robot drifts off course, it ends up somewhere the demonstrations never went. It knows even less there, so it drifts further and further. Here's an analogy. Copy one word wrong while transcribing, and everything after gets more and more off. A drifting robot snowballs the same way. So the hard part of copying is that mistakes snowball!
Two fixes
Researchers have tried many tricks. Here are two well-known ones. The first is called action chunking. Instead of one step at a time, it outputs a short stretch at once. A short stretch at a time, and the robot is less likely to lose focus halfway. Less jitter, and errors don't snowball so easily. That's the idea behind ACT. The paper reports that it learned six real tasks. On ones like opening a condiment cup and slotting a battery, with only ten minutes of demonstrations, success reached eighty to ninety percent. The second fix: swap the brain. Use a diffusion model as the policy. It's good at handling one scene with several correct ways to act. Say you're going around an obstacle. Left or right, both work. The old approach takes the average, gets stuck in the middle, and crashes. A diffusion policy can learn each way separately. This is called Diffusion Policy. The paper reports that across twelve tasks, compared with the previous best methods, it did nearly forty-seven percent better on average. The two fixes take different routes, but at heart both are imitation. Both are after the same thing: how to keep errors in check. Chunking or diffusion, both are ways to stop mistakes from snowballing!
How many are enough
There's one more practical question: how many demonstrations do you need? Your gut says the more, the better. Is that true? One paper studied exactly this. It found that once each scene has enough demonstrations, adding more does very little. What really works is diversity. More environments and more objects beat piling up numbers. The same paper reports that with enough diverse data, two tasks reached about ninety percent success in new environments with unseen objects. Draw it as a curve: a steep rise at first, then it flattens out. Diversity is the key to lifting the whole curve. So when you collect data, don't just keep recording more. Ask one more question: have I covered all kinds of situations? So it's not the more, the better. It's the more complete, the better! Try it on screen. Drag the number of demonstrations, then drag scene diversity. Watch how the success rate climbs along the curve. With few demonstrations, the policy drifts off halfway, marked in amber. With enough varied demonstrations, it finishes steadily, and turns green. But be careful. These success rates were all measured in tests. Switch to a scene it has never seen, and they may drop again. How do you know it has truly learned? You have to test it on enough unseen situations. That's the craft of evaluation, and we'll give it its own episode later. In the end, imitation learns to copy. Faced with something the demonstrations never showed, it easily gets lost. That's when it has to try things itself. Let's recap, three points. One: imitation learning means copying demonstrations, but mistakes snowball. Two: action chunking and diffusion policies make it steadier. Three: more demonstrations aren't always better; diversity matters more. So if it watches many times and still can't learn, maybe the demos weren't varied enough, or the mistakes snowballed! If all it can do is imitate, what about things it's never seen? Let it learn by trial and error. See you next time!
Evidence and sources
Practice and self-check
- Interactive quiz5 questions, explained as you answer
- Browser labno install: drag and tune it yourself
- One-page summaryprintable; opens in a new tab
The quiz is public and explains each answer as you go; answers to the guide's exercises are for signed-in members.
The episode 1.6 video is about imitation learning. The simplest approach is behaviour cloning: learning “what you see, what you do” by copying demonstrations. The difficulty is that once the robot drifts off course, it enters states the demonstrations never covered, and the errors snowball. Action chunking (ACT) and Diffusion Policy are two well-known fixes. And more demonstrations are not always better; diversity matters more. The video cites three pieces of evidence: the ACT paper reports 80–90% success on real tasks such as opening a condiment cup and slotting a battery, with about 10 minutes of demonstrations; the Diffusion Policy paper reports an average improvement of 46.9% over the previous best methods across 12 tasks; and a data scaling paper reports that, once enough diverse data had been collected, two tasks reached about 90% success in new environments with unseen objects. This page is the episode’s reading and exercises. Rather than repeat the transcript, it first puts the three numbers back into the experimental conditions of their own papers. It then uses hypothetical numbers to work out three things by hand: how an error rolls out of the range the demonstrations cover, how chunking changes the number of “decisions”, and why two correct ways around an obstacle can be averaged into a collision. Finally, it splits “how many demonstrations” into three separate counts: environments, objects and demonstrations per combination. One rule runs through the whole page: a success rate holds only for the paper, the set of tasks and the evaluation that produced it.
The video, transcript and sources for this lesson are at the top of this page; the season plan is in the course outline. The previous lesson, Physical AI Lab 1.5: Where Does Data Come From? Working Out Demonstration Time from SO-101 Metadata, Then Separating the Units and Action Spaces of Three Sources, ended by asking how, once the data exists, a robot actually learns actions from demonstrations. This lesson picks up there. Section 4 of 1.5 already used a hypothetical synthesis plan to separate “how many were generated” from “how many situations are covered”. This lesson does not repeat that; instead it looks at what a paper that systematically varied the number of environments, objects and demonstrations in the real world does and does not say.
The public quiz’s 5 questions come with full explanations. Signed-in readers can work through three exercises: work through a fictional drift log, find the step at which the error leaves the range the demonstrations cover, and check the number of policy queries under chunking and temporal ensembling; work out why a fictional set of left and right detours around an obstacle gets averaged into a collision, and review two proposed fixes; and review a fictional demonstration-collection plan and its claims about evaluation. Each comes with a step-by-step solution, common mistakes and a 20-point rubric.
1. Three pieces of evidence and their conditions
The three numbers come from three arXiv papers, all authors’ reports (level R). In the order the video uses them:
| What the video says | Source | Conditions in the source | What the source does not say | Level |
|---|---|---|---|---|
| About 10 minutes of demonstrations, 80–90% success on real tasks such as opening a condiment cup and slotting a battery | Zhao, Kumar, Levine and Finn, Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware (ACT / ALOHA, arXiv 2304.13705, 2023-04-23) | 6 fine manipulation tasks in the real world; the 80–90% is attached to the examples given, opening a translucent condiment cup and slotting a battery; the introduction says “10 minutes or 50 demonstration trajectories”; the ALOHA bimanual teleoperation system, with teleoperation and data recording both at 50 Hz | The quoted sentence does not say every one of the 6 tasks reached 80–90%; the quoted text does not state whether the 10 minutes are per task or in total | R (authors’ report) |
| Nearly 47% better on average than the previous best methods across 12 tasks | Diffusion Policy (arXiv 2303.04137, 2023-03-07) | 12 tasks from 4 robot manipulation benchmarks, compared with the state-of-the-art robot learning methods of the time, average improvement 46.9% | The abstract does not say whether 46.9% is a relative improvement or percentage points, nor which of the 12 tasks are in simulation and which on real robots | R (authors’ report) |
| With enough diverse data, about 90% for two tasks in new environments with unseen objects | Lin, Hu et al., Data Scaling Laws in Imitation Learning for Robotic Manipulation (arXiv 2410.18647, 2024-10-24) | Four data collectors worked for one afternoon; the introduction names the two tasks as Fold Towels and Unplug Charger, each tested in 8 new environments and objects; the policy is a Diffusion Policy | It is not a result for all tasks; the two tasks used to derive the scaling laws (Pour Water and Mouse Arrangement) are a separate set of experiments | R (authors’ report) |
The corresponding sentences in the three papers, each excerpted separately:
ACT allows the robot to learn 6 difficult tasks in the real world, such as opening a translucent condiment cup and slotting a battery with 80-90% success, with only 10 minutes worth of demonstrations.
The synergy between these two parts allows learning of 6 fine manipulation skills directly in the real-world, such as opening a translucent condiment cup and slotting a battery with 80-90% success, from only 10 minutes or 50 demonstration trajectories.
We benchmark Diffusion Policy across 12 different tasks from 4 different robot manipulation benchmarks and find that it consistently outperforms existing state-of-the-art robot learning methods with an average improvement of 46.9%.
With four data collectors working for one afternoon, we collect sufficient data to enable the policies for two tasks to achieve approximately 90% success rates in novel environments with unseen objects.
The first two sentences both come from the ACT paper, the first from the abstract and the second from the introduction. Three things to note when reading them:
- The three percentages are not the same kind of quantity. 80–90% is a success rate on the real tasks given as examples; 46.9% is an average improvement over “the previous best methods”; about 90% is a success rate for two tasks in new environments with new objects. Their tasks, baselines and evaluations all differ, so they cannot be ranked against each other, nor used to say which method is “better”.
- “10 minutes” and “50” are two measures in the same sentence. If both describe the same set of demonstrations for one task (the quoted text does not say so explicitly), each trajectory averages about 600 ÷ 50 = 12 s, or about 12 × 50 = 600 timesteps at the 50 Hz recording rate. This is this lesson’s arithmetic, and section 2 uses it.
- The two fixes were later combined. The data scaling paper models its data with a Diffusion Policy and also adopts the temporal ensemble proposed in ACT to smooth the actions. The video says the two fixes take different routes but are both imitation at heart; in this paper, they are not an either-or choice either.
Sign in to keep reading: the rest of this lesson, its self-check questions and the answers.
Limitations
- The figures from ACT / ALOHA (arXiv 2304.13705, 2023-04-23), Diffusion Policy (arXiv 2303.04137, 2023-03-07) and Data Scaling Laws in Imitation Learning (arXiv 2410.18647, 2024-10-24) are authors’ reports, level R. This site has not independently reproduced any of them.
- This lesson relies on the abstract of Diffusion Policy, the abstract and opening sections of the ACT paper (the excerpt breaks off in the training pipeline of section IV) and the abstract and opening sections of the data scaling paper (the excerpt breaks off partway through stating the maximum score), not on supplementary material. The k that ACT actually uses, the weights of its temporal ensemble, per-task success rates and how the data scaling paper converts scores into success rates are not in that text.
- The 12 s and 600 steps in section 1 assume that “10 minutes” and “50” describe the same set of demonstrations for one task, which the source does not state explicitly; the 1,600 demonstrations in section 4 are this lesson’s multiplication from the authors’ example. These are this lesson’s arithmetic, not figures the sources state directly.
- The covered ranges, deviation update rules, bump sizes, k = 50, the four joint predictions, obstacle dimensions, left/right proportions and the 4 × 4 × 100 plan in sections 2–4, and R6, demonstration set D, fixes A and B, team D, P1–P3, C1–C5 and all scores in section 5, are made up for the exercises; the computed 0.8 cm, 0.75 cm, step 43, 0.15 s, −2.4 cm, 79%, 0.75, 0.78 and so on hold only under those assumptions. The deviation model is a teaching simplification and does not describe the dynamics of any real policy.
- “Full marks on every step counts as completed”, holding out test environments by factor and the histogram check are teaching suggestions, not definitions or conclusions from any source. Paper calculations, the browser lab page and simulation are not hardware safety validation.
- The curve on the episode’s lab page is labelled illustrative; this page has not checked the lab page’s implementation.
Links
- Physical AI Lab: a physical AI course where every number can be checked (course outline)
- Physical AI Lab 1.5: Where Does Data Come From? Working Out Demonstration Time from SO-101 Metadata, Then Separating the Units and Action Spaces of Three Sources
- Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware (ACT / ALOHA, arXiv 2304.13705)
- Diffusion Policy (arXiv 2303.04137)
- Data Scaling Laws in Imitation Learning for Robotic Manipulation (arXiv 2410.18647)
