Lesson video
Narration is AI-generated. Video chapters, transcript and sources
Transcript
Transcript
UNIT-01 asks
A robot's success rate sounds pretty high. But would you trust it, based on that one number alone? This episode, we take one success rate apart, and look at three things it doesn't tell you. One number is easy to remember, easy to compare, and easy to put in a headline. But it's too clean. It covers up all the questions you should ask. How do you tell whether a policy is any good? We run it many times, and count how often it succeeds. That's evaluation. A success rate is the number of successes, divided by the total number of tries. So first you need to know how many tries there were.
Under what conditions?
Here's a real example. Someone took a robot arm policy and ran a public stress test on it. Thirty runs in all. The same task, under three conditions. Normal light, dim light, and a moved camera. Each run has a fixed budget. At most one hundred sixty actions, eight seconds of simulated time. Then it stops. Light first. In dim light, compared with normal light, a different set of runs succeeds. They also paired the runs. Same starting state, one run under each of the three conditions. That way you can compare them one to one. Now the results. With the camera moved, successes went up by two, net. But split it apart, and it's three got better and one got worse. So a "net gain" can quietly cancel out the better and the worse! So the first thing is to ask. Under what conditions was this success rate measured?
Failures come in kinds
Thirty runs isn't many. With a small sample, good or bad luck alone can push a success rate up or down. Look at the denominator first. In those same thirty runs, fourteen failed. Every one of them stopped only because it ran out of its action budget. Stopping only when the steps run out means it didn't give up. It just didn't finish within the allowed actions. But failing isn't the same as being stuck. In the records, the end of the arm was still moving. In that last stretch, it moved forty-four point eight to one hundred thirty-eight point six millimetres. One "failure" label lumps several situations into one. Stuck and not moving, still trying, or just short. Those are really very different. Take one set of seeds. In normal light, it succeeded. But in dim light and with the moved camera, it didn't finish. It used up all its steps. So from the success rate alone, you can't even tell how it failed! And don't forget how demos work. The video you get to see is often cut from the best run.
One result doesn't count
Reinforcement learning is full of randomness. Starting position, exploration, seed. Change one, and the result may be different. Same method, different seed, and the score jumps up and down. Run it only a few times, and it may well be just luck. A paper warns about exactly this. Drawing conclusions from just a few runs makes it easy to get it wrong. That experiment did one good thing. It reran the whole set exactly as before, and all thirty results came out the same. But that only shows it can be reproduced under the same conditions. With a different task or a different seed, it may not hold. Run it a few more times, and report how high and how low the score goes. Then others know how steady the number is. So a serious evaluation runs several times, and reports a range, not just a single number. We should be clear, too. This is a small controlled experiment, not an official benchmark. Real deployment is more complicated. So when you see a success rate, ask four more questions. What conditions? How many runs? How did it fail? Can it be repeated? On the page, pick a condition and choose a set of seeds. Then lay out all thirty results, one cell at a time. Let's recap, three points. One: for a success rate, ask under what conditions it was measured. Two: failures come in kinds, so look at how it failed. Three: one result doesn't count. Repeat it, and look at the spread. Don't be fooled by a pretty success rate. Ask a few more questions about how it was really measured! That wraps up the hands-on level. Want to build a grasping scene yourself, in simulation? See you next time!
Evidence and sources
Practice and self-check
- Interactive quiz5 questions, explained as you answer
- Browser labno install: drag and tune it yourself
- One-page summaryprintable; opens in a new tab
The quiz is public and explains each answer as you go; answers to the guide's exercises are for signed-in members.
The episode 1.8 video takes one success rate apart and asks about three things it does not tell you: under what conditions it was measured (a total can cancel gains against losses), what kind of failure it counts (one “failure” label hides how the robot failed), and whether the result can be repeated (change the seed and the score jumps, so repeat, look at the spread and report an interval). The running example is robot-reel’s SmolVLA Stress Lab: one task, three conditions, 30 closed-loop simulation tests. Chapter three also cites Henderson et al., Deep Reinforcement Learning that Matters (arXiv 1709.06560). This page is the episode’s reading and exercises. Rather than repeat the transcript, it lays out the calculations behind those numbers. First it works out from several README records how many successes each condition had, sets the “net gain of 2” out as a paired table, and counts where the p-value of the README’s exact sign test comes from. Then it looks at which 1 mm criterion the “still moving” failure label rests on, and whether the label changes with the threshold. Finally it separates “an exact rerun matched” from “a new seed makes the score jump”. One rule runs through the whole page: a success rate belongs only to its own task, conditions, budget and number of trials, and a test speaks only for its own set of pairs.
The video, transcript and sources for this lesson are at the top of this page; the season plan is in the course outline. The previous lesson, Physical AI Lab 1.7: Reinforcement Learning — Working Out One Episode’s Return to See How a Reward Gets Gamed, Then Reading the Paper’s Terrain Curriculum and Time-out Resets, ended by asking how good the robot really is, and how much a single success rate leaves unsaid. This lesson picks up there. The stress lab as an open-source project is introduced in Robot Reel: turning robot policy simulations into frame-by-frame verifiable replays and 3D scenes; this lesson uses only the records that concern evaluation method. A disclosure: robot-reel is published by NoteFlowAI, the same organisation behind this course. It is treated below as level O (the project’s own account), and this site has not reproduced it independently.
The public quiz’s 5 questions come with full explanations. Signed-in readers can work through three exercises: keep the books on a fictional evaluation of 12 paired starts and compute an exact sign test by hand; relabel 8 fictional failures at 4 stall thresholds and review two conclusions; and split 10 fictional seeds of one method into two groups and review three report sentences. Each comes with a step-by-step solution, common mistakes and a 20-point rubric.
1. Two sources, and what each one says
The video’s robot-reel numbers all come from the project README; this lesson reads the main-branch version fetched on 2026-10-10. The “paper warning” in chapter three comes from an arXiv preprint by Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup and David Meger (v1, submitted 2017-09-19), from McGill University and Microsoft Maluuba. The relevant README sentences, each excerpted separately in source order:
SmolVLA runs the same task under reference lighting, reduced light and a shifted camera. Explore 30 real closed-loop trials across ten paired initial states.
The camera condition’s net gain of two successes includes three gains and one loss.
All 14 unsuccessful trials reached the action limit.
Every trial remained above the 1 mm stall threshold: end-effector travel over the final tenth of each episode ranged from 44.8 mm to 138.6 mm.
One repeat of the full plan on the same L40S matched 30 / 30 outcomes, action counts, recorded robot states and actions, and 360 / 360 rendered frames consumed by policy calls.
Fixed budget: 160 actions / 8 simulation seconds per trial.
The sentence from the “Random Seeds and Trials” section of Henderson et al.:
We perform 10 experiment trials, for the same hyperparameter configuration, only varying the random seed across all 10 trials. We then split the trials into two sets of 5 and average these two groupings together.
In one table; the “what the source does not say” column matters just as much:
| In the video | Source | Conditions in the source | What the source does not say | Level |
|---|---|---|---|---|
| 30 runs, three conditions | robot-reel README | SmolVLA on one task; reference lighting, reduced light, shifted camera; each of 10 paired initial states run under all three; recorded with NVIDIA L40S / CUDA inference | The README states this is a controlled diagnostic, not an official LIBERO benchmark score; there is only one task | O (project’s own account) |
| 160 actions / 8 seconds | Same | The fixed budget per test | Task progress and success with a larger budget; the README says these require separate evaluation | O |
| Net gain of 2 = 3 gains, 1 loss | Same | Paired outcomes for the camera condition | The README says this single task does not establish general robustness or causation | O |
| 14 failures, end effector still moving 44.8–138.6 mm | Same | Every failure reached the action limit; travel over the final tenth of each episode stayed above the 1 mm stall line | It shows motion at the cut-off only, not task progress | O |
| Exact rerun, 30 / 30 matched | Same | One repeat of the full plan on the same L40S | The README says it does not establish general determinism or, by itself, causal attribution | O |
| 10 runs split into two groups of 5 | Henderson et al. | Same hyperparameters, only the random seed varied; the paper studies policy gradient methods on MuJoCo continuous-control tasks | The paper says no specific number of trials can be recommended | R (authors’ report) |
Three things to note when reading this table:
- The two sources study different kinds of randomness. robot-reel evaluates the test results of an already trained policy (SmolVLA) under a fixed budget; Henderson et al. discuss the randomness of reinforcement-learning training itself, citing environment stochasticity and stochasticity from random number generators in the learning process (for example weight initialisation). The video’s chapter three puts both under “one result doesn’t count”, but the paper’s variance cannot be carried over directly to robot-reel’s success rates.
- All the robot-reel numbers come from one task in one simulation. The README also states that its 30 trial rows and 20 paired rows on Hugging Face are the pilot’s tabular evidence, not a training dataset or an official benchmark.
- The README changes. It is a file on the main branch, not a pinned commit; this lesson relies on the version fetched on 2026-10-10.
Sign in to keep reading: the rest of this lesson, its self-check questions and the answers.
Limitations
- The robot-reel README’s numbers are level O, the project’s own account, published by NoteFlowAI, the same organisation behind this course; this site has not reproduced them independently. This lesson reads the main-branch README fetched on 2026-10-10; it did not read the files it links, such as docs/stress.md, nor open the lab page or the Hugging Face dataset. The README states this is a controlled diagnostic, not an official LIBERO benchmark score; it covers one task, runs in simulation and uses NVIDIA L40S inference.
- The 5, 4 and 7 successes for reference lighting, reduced light and shifted camera are this lesson’s derivation from three README records; “lost” and “gained” are read as relative to reference lighting, which the quoted README text does not spell out. The README mentions that the lab page gives per-condition confidence intervals; the quoted text has no values or method, and this lesson did not compute them.
- Deep Reinforcement Learning that Matters (arXiv 1709.06560v1, 2017-09-19) is a level-R preprint; its experiments are policy gradient methods (TRPO, DDPG, PPO, ACKTR) on MuJoCo environments from OpenAI Gym, and the Table 3 numbers are authors’ reports. This lesson relies on an excerpt of the main text up to the start of the Figure 5 caption, not the supplemental material. The paper discusses randomness in training, not the success rates of a trained policy in a stress test.
- The sign-test combination counts, “at least 6 discordant pairs”, “44.8 mm is above 5 mm so no label changes”, the interval widths and so on are this lesson’s arithmetic and explanation. This site has not run any of the README’s commands.
git clonefetches the main branch and is not pinned to a commit; the README’s PyPI installation instructions pin robot-reel==0.19.1, while--paired-exactrequires 0.17.1 or later. - Section 5’s policies P and Q, conditions R and N, seeds 01–12, f1–f10, the 200 and 300 actions, method M, baseline N, team T and all returns, batches and sentences are fictional, for practice only; the computed 58.3%, 0.03125, 0.0625, 2040, 1654, 1847, 2390, 68.3% and so on hold only under these assumptions. The “usable / needs rewriting / not usable as representative” split and the experimental conventions are teaching suggestions, not conclusions of any source. Neither simulation evaluation nor paper calculation is a hardware safety validation.
- This page has not checked the implementation of the episode’s interactive page.
Links
- Physical AI Lab: a physical AI course where every number can be checked (course outline)
- Physical AI Lab 1.7: Reinforcement Learning — Working Out One Episode’s Return to See How a Reward Gets Gamed, Then Reading the Paper’s Terrain Curriculum and Time-out Resets
- Robot Reel: turning robot policy simulations into frame-by-frame verifiable replays and 3D scenes
- Frontier Radar | Beyond Binary Success: sequential tests that separate robot policies with fewer real-world rollouts
- Frontier Radar | Is Success All You Need?: two VLAs with similar success rates can succeed in very different ways
- robot-reel README (SmolVLA Stress Lab; main branch, fetched 2026-10-10)
- Deep Reinforcement Learning that Matters (arXiv 1709.06560v1)
