Lesson video
Narration is AI-generated. Video chapters, transcript and sources
Transcript
Transcript
UNIT-01 asks
Can a robot learn the way a kid learns to ride a bike? Just by trying, and trying, until it gets it? This episode is about reinforcement learning. No demonstrations, only rewards. The robot works out its own moves by trial and error.
Rewards and trial and error
Last time, imitation learning meant watching a person's demonstration and copying it. This time is different. There's no demonstration. In reinforcement learning, the robot moves on its own, and the environment gives it a score. That score is called a reward. Right moves earn points; wrong moves lose them. What it has to learn is a policy. Look at the situation in front of it, and know how to move next. The robot isn't after the score for one step. It wants the total for a whole round. We call that the return. Here's the hard part: when a round succeeds, which steps deserve the credit? The robot has to sort that out, slowly. Take reaching for the cup. The closer to the cup, the higher the score. A steady pick-up earns a big score. Knock over the bottle, and you lose points. At first, the policy moves at random. Most of the time, it can't reach. Once in a while it happens to get near the cup, and it earns a few points. It makes the high-scoring moves more likely. After a while of practice, it reaches the cup more and more often. Keep practising, and the motion gets smooth too. Reach out, go around the bottle, pick up the cup, all in one go. So nobody taught it. It stumbled onto the right moves, again and again, and remembered them!
Trying many times in simulation
The trouble with trial and error: you need a lot of tries. A real robot can't afford that. It's too slow, and it breaks easily. So most of the trial and error moves into simulation. If it breaks, just start over. It costs nothing, and hurts no one. In twenty twenty-one, a paper ran an experiment. Thousands of copies of a robot dog practised at the same time, on a single graphics card. Each copy tries on its own. The falls and the successes all flow together, and feed one shared policy. The paper reports that on flat ground, the robot dog learned to walk in under four minutes. On uneven terrain, it took only twenty minutes. The paper also says the policy trained in simulation was moved onto a real robot dog. It was tested, and it really worked. There's one more trick to trial and error. Try moves you haven't tried yet. Don't always stick with the safest one you know. Plot the return of every round as a curve. When it climbs, the robot is getting better. But it wobbles, and it can stall halfway.
Rewards are hard to get right
Reinforcement learning has one upside. It doesn't copy a demonstration, so it may find a better way than a person would. But rewards are hard to write. Score only the moment of success, and it's too rare. Learning is very slow. So people often add points for "getting close" too, guiding it the right way. This is called reward shaping. Shaping cuts both ways. Write the reward just a little wrong, and the robot will find the loophole. Say you only reward "the cup moved." The robot might just knock the cup off the table, and still get a high score. So rewards need testing and rewriting, again and again. And someone has to watch the replays, to see if it really did the job. Compare it with imitation learning. Imitation learns fast, but rarely beats the teacher. Reinforcement can go further. But it's slow, and you have to be careful with rewards. On the page, drag the progress bar and watch the reward curve climb. Then switch to a bad reward, and watch it find the loophole. Three things are still unsolved. Rewards are hard to get right; trial and error takes a lot of tries; and there's still a gap between simulation and reality. Let's recap, three points. One: reinforcement learning runs on rewards and trial and error, not demonstrations. Two: trial and error takes many tries, so thousands of copies practise together in simulation. Three: rewards are hard to get right, and a crooked one gets hacked. So it wasn't taught. It found its own way, bit by bit, by chasing rewards! But how good is it, really? How much does one success rate leave unsaid? See you next time!
Evidence and sources
Practice and self-check
- Interactive quiz5 questions, explained as you answer
- Browser labno install: drag and tune it yourself
- One-page summaryprintable; opens in a new tab
The quiz is public and explains each answer as you go; answers to the guide's exercises are for signed-in members.
The episode 1.7 video is about reinforcement learning. There are no demonstrations, only rewards: the robot works out a policy by trial and error (look at the situation in front of it, and know how to move next). What it is after is not the score for one step but the return, the total for a whole round. Trial and error takes a great many tries, so it moves into simulation. And rewards are hard to get right; write one a little wrong and the robot finds the loophole. The video’s real evidence comes from one paper: Rudin et al., Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning (arXiv 2109.11978), reports that training thousands of simulated robots in parallel on one workstation graphics card produces walking policies for a quadruped in under four minutes on flat terrain and in twenty minutes on uneven terrain. This page is the episode’s reading and exercises. Rather than repeat the transcript, it does two things the video does not go into. First, it works out the return of one episode for a hypothetical cup, to see how shaping that rewards “getting close” can make “waiting next to the cup” score higher than “picking it up”. Then it opens two sections of the same paper that the video does not cover: how an automatic terrain curriculum makes the difficulty follow the policy, and how the paper tells apart two ways an episode can end, “time ran out” and “the robot fell”. One rule runs through the whole page: a return holds only for its own reward definition, and a training time holds only for its own hardware, task and set-up.
The video, transcript and sources for this lesson are at the top of this page; the season plan is in the course outline. The previous lesson, Physical AI Lab 1.6: Imitation Learning — Working Out How an Error Rolls Out of the Demonstrations’ Coverage, Then Putting the ACT, Diffusion Policy and Data Scaling Numbers Back in Their Own Conditions, ended by asking what a robot that can only imitate does when it meets something it has never seen. This lesson picks up there. The same paper’s “thousands of robots” and “under four minutes”, with their experimental conditions, were already covered point by point in Physical AI Lab 0.4: Ten Thousand Tries in a Virtual World — Putting the Conditions Back on Three Simulation Numbers, which also worked through a parallel set-up with the paper’s batch formula; Physical AI Lab 1.4: A Map of Simulators — Counting the Cost of the Timestep, Finding Where Contact Splits Two Runs, and Pinning Three Speed-ups to Their Baselines then checked its baseline alongside two other speed-up figures. This lesson does not repeat that; it adds only what concerns reinforcement learning itself: rewards and returns, the curriculum, and resets.
The public quiz’s 5 questions come with full explanations. Signed-in readers can work through three exercises: compute returns for three replays under two fictional rewards, find the one being gamed and put a cap on the shaping term; apply the paper’s curriculum rules to a fictional training log to decide level changes, and explain why the return can fall while the policy gets better; and review three ways of handling time-out resets, computing the critic’s training targets. Each comes with a step-by-step solution, common mistakes and a 20-point rubric.
1. One paper, two numbers, and what the video does not go into
The paper’s authors are Nikita Rudin, David Hoeller, Philipp Reist and Marco Hutter, of ETH Zurich and NVIDIA; it was submitted to arXiv on 2021-09-24 and is level R (authors’ report). The video’s two numbers correspond to two sentences in the abstract, each excerpted separately:
In addition, we present a novel game-inspired curriculum that is well suited for training with thousands of simulated robots in parallel.
The parallel approach allows training policies for flat terrain in under four minutes, and in twenty minutes for uneven terrain.
The “thousands” in the first sentence is the same thing as the video’s “thousands of copies”, but the subject of that sentence is actually a curriculum, which section 3 of this lesson unpacks. Three more things to note when reading these sentences (the hardware and simulation conditions already covered in 0.4 are not repeated):
- What is learned. Section 3 of the paper states the task: a quadruped must walk across uneven surfaces, slopes, stairs and obstacles while following base-heading and linear-velocity commands. “Learning to walk” means training such a policy in simulation. Most simulation and real-world experiments use the ANYbotics ANYmal C; in simulation, policies were also trained for ANYmal B, ANYmal C with an attached arm, and Unitree A1.
- Faster than what. The abstract says this is a speedup of multiple orders of magnitude compared to previous work. The introduction lists several earlier quadruped reinforcement learning results: blind policies robust to highly uneven ground, 12 hours of training; perceptive locomotion combining learning with optimal control, 82 and 88 hours; a fully learned approach, 120 hours; and says this paper’s approach trains a perceptive policy in under 20 minutes on a single GPU. These are the authors’ summaries of other work, with different tasks, hardware and perception set-ups, so they cannot be divided to get a single “times faster” figure.
- The real-robot step. The abstract says the policies were finally transferred to the real robot to validate the approach; the video accordingly says “it was tested, and it really worked”. The excerpt this lesson relies on stops at the start of section 3.2 and does not include the real-world results. The stopwatch in the video is drawn with a one-hour dial; 4 minutes and 20 minutes are just positions on that dial, not additional measurements.
Sign in to keep reading: the rest of this lesson, its self-check questions and the answers.
Limitations
- The training times, curriculum rules, reset handling and the description of Figure 3 in Learning to Walk in Minutes (arXiv 2109.11978, 2021-09-24) are authors’ reports, level R. The authors are from ETH Zurich and NVIDIA, and Isaac Gym, the simulation environment used, is NVIDIA’s. This site has not independently reproduced any of them.
- This lesson relies on the paper’s abstract and the text up to the start of section 3.2, not on the definitions of the reward terms and actions, the real-world results or the supplementary material; the effect of the reset handling on the total reward and critic loss is in the supplementary material, which this lesson has not read.
- The 12, 82, 88 and 120 hours in the introduction are the authors’ summaries of other work, each with its own task, hardware and perception set-up; this lesson has not checked those original papers.
- Reward R0, the two replays, k = 0.1 points/cm and 30 cm in section 2; the 0.3 m/s and the three distances in section 3; γ = 0.8 and 4.6 in section 4; and A7, replays a–c, V1, V2, the 4-level stairs, robots A–F, the dashboard figures, team W, γ = 0.9, 9.0, I1–I3 and the evenly staggered episode starts in section 5 are made up for the exercises. The computed 13, 16, 10.8, 0.476, 0.333, 0.167, 4.68, 9.1, 40 batches, 100 robots and so on hold only under those assumptions and are not data from the paper or any system.
- “Walking straight towards an edge, 4 m to the border” is this lesson’s simplification from the 8 m tiles and the centre start. The paper’s excerpt does not explain how “walking past the border” is determined, nor what happens when the level-up and level-down conditions both hold.
- Capping the shaping term, rewarding how much closer it got, the replay-review rule and comparing on fixed terrain are teaching suggestions, not the paper’s methods or conclusions. Paper calculations and simulation results are not hardware safety validation; trial and error on a real robot needs separate safety measures, which this lesson does not cover.
- Segment 30 of the video is a scripted performance of lab page 1.7-rl; this page has not checked the lab page’s implementation.
Links
- Physical AI Lab: a physical AI course where every number can be checked (course outline)
- Physical AI Lab 1.6: Imitation Learning — Working Out How an Error Rolls Out of the Demonstrations’ Coverage, Then Putting the ACT, Diffusion Policy and Data Scaling Numbers Back in Their Own Conditions
- Physical AI Lab 0.4: Ten Thousand Tries in a Virtual World — Putting the Conditions Back on Three Simulation Numbers (the same paper’s experimental conditions and batch calculation)
- Physical AI Lab 1.4: A Map of Simulators — Counting the Cost of the Timestep, Finding Where Contact Splits Two Runs, and Pinning Three Speed-ups to Their Baselines
- Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning (arXiv 2109.11978)
