Physical AI Open Source in Action R01 | LeRobot v0.7.0: install it in a sandbox, train twice, read the evaluation

R01: we install LeRobot v0.7.0 in an isolated sandbox and train on simulated PushT for 2,000 and 20,000 steps. Evaluation: 0 of 3 and 1 of 10 episodes succeed. The intervals overlap heavily, so this doesn't show the policy has learned the task.

In this episode we install LeRobot, an open-source robot learning framework, and use it to train a policy on a simulated task: pushing a T-shaped block onto a T-shaped target (PushT). We look at the evaluation results from two training runs, then at two evaluation replays, one that succeeds and one that fails. What you’ll see is one complete run through the workflow. Nothing here shows that the policy has learned the task.

This episode is a simulation demo, not a real-robot experiment. All numbers come from the two runs’ train.log and the evaluation JSON. This page is the written version of episode 1 of the Physical AI Open Source in Action series.

Video

The voice is AI-generated (Serena).

Why LeRobot

LeRobot is Hugging Face’s robot learning library, released under Apache-2.0. We used v0.7.0, pinned to commit 410c095e. Data, policies, training and evaluation all run from one set of commands. We ran training and evaluation end to end in a sandbox.

Setting up the environment

We installed dependencies with uv, on Python 3.12 or later. The repository ships a lock file that pins every version; its sha256 starts with d569cc07.

The official way is to install lerobot itself, following the official installation guide, with three extras: training, pusht and diffusion. Install all three together. If any one is missing, you get an ImportError at run time.

Our sandbox has isolation requirements, so we exported the dependencies from the lock file and ran from the source directory through PYTHONPATH. You don’t need to do that. Follow the official instructions.

Three things about the training command

First, push_to_hub is on by default, which means the training config asks to push the results to the Hugging Face Hub. Put --policy.push_to_hub=false in the command. Second, --help only lists the full set of options after you pick a policy type, so we pass --policy.type=diffusion first and then look up option names. Third, pin the dataset version. For lerobot/pusht that’s revision b1c3ecba, data format v3.0, 206 episodes and 25,650 frames.

Policy and settings

The policy is a diffusion policy. The vision part is a ResNet-18 that starts from ImageNet-pretrained weights, so we did not train it from scratch. The rest of the policy is trained on PushT data. Both runs used a batch size of 8 and random seed 1000, and both used that single seed.

Run one: 2,000 steps

The first run trained for 2,000 steps and took about 4 minutes 40 seconds, including two evaluations, at step 1,000 and step 2,000. At step 2,000 we evaluated 3 episodes, and none succeeded. The 95% Wilson interval for that result runs from 0% to 56.2%. Zero successes in 3 episodes does not tell you the success rate is zero.

At step 1,000, also over 3 episodes, the average max reward was 0.053. At step 2,000 it was 0.17.

This metric measures how far the block gets pushed into the target region. For each episode we take the highest reward it reached, then average across episodes. The reward at each step is the coverage divided by 0.95, clipped to between 0 and 1. Once coverage passes 95%, the episode counts as a success and ends immediately. So 1 means success, and 0.96 corresponds to about 91% coverage, which is still a failure. This metric is not a success rate.

Run two: 20,000 steps

The second run trained for 20,000 steps on an L40S and took about 32 minutes (not counting the final evaluation). That works out to about 10 steps per second, which we calculated as 20,000 steps divided by about 1,927 seconds.

At step 20,000 the run evaluated 10 episodes. One succeeded, a 10% success rate, with a 95% interval of 1.8% to 40.4%. The average max reward was 0.70.

Training stepsEval episodesSuccessesSuccess rate95% interval (Wilson)Average max reward
2,000300%0% to 56.2%0.17
20,00010110%1.8% to 40.4%0.70

The two success-rate intervals overlap heavily, so we can’t say the success rate went up from 0% to 10%. What we can say is that the average max reward went from 0.17 to 0.70: the policy pushed the block closer to the target. That’s what we saw in these two runs, and it doesn’t carry over to other step counts or other seeds.

Evaluation replays: one success, one failure

The evaluation recorded 4 videos. Video 0 is the only success out of the 10 episodes; video 1 is a failure. Videos 2 and 3 also failed. The video above shows videos 0 and 1.

Seven pitfalls

We ran into 7 pitfalls in this episode. Three are covered here: push_to_hub is on by default; --help needs a policy type first; and the training and diffusion extras have to be installed together. The other four are the storage domains to allow in a network-restricted environment, the settings a GPU needs inside an isolated environment, two opencv packages overwriting each other, and parallel downloads hitting the proxy’s connection limit. The details are not covered in this episode.

Limits and notes

Two training runs, one seed, and 13 evaluation episodes in total: 3 in the first run and 10 in the second. The conclusion is a small one. The workflow runs, and the numbers match the logs, but the policy isn’t reliable yet. To judge whether it’s any good, you’d need more steps, more episodes and more seeds.

The terminal in the video shows excerpts replayed from the training logs, not screen recordings. The text is taken from the logs, with paths replaced by generic names and the timestamp prefix at the start of each line removed (a date inside a line is part of the log itself). Long lines are cut off with an ellipsis, and some lines in between are removed. The result cards and the comparison chart in the video are illustrations drawn from the measured numbers.

Next time we’ll take another repository and install it, run it and check it the same way.

Evidence

Where every number in the video comes from. Feel free to check.

  • 0/3 Run one (2,000 steps): 0 of 3 evaluation episodes succeeded, 95% Wilson interval 0%–56.2%, average max reward 0.17 (0.053 at step 1,000), about 4 minutes 40 seconds including two evaluations (O · official, first-hand). Source: the training log from this run (evaluation summary lines; log not published), 2026-11-11
  • 1/10 Run two (20,000 steps): 1 of 10 evaluation episodes succeeded, 95% Wilson interval 1.8%–40.4%, average max reward 0.70, about 32 minutes of training (not counting the final evaluation), about 10 steps per second calculated as 20,000 steps divided by about 1,927 seconds (O · official, first-hand). Source: the training log from this run (evaluation summary lines; log not published), 2026-11-11
  • 410c095e LeRobot v0.7.0, commit 410c095eacc0011d998a73383eb94abed2715af0, Apache-2.0 (O · official, first-hand). Source: huggingface/lerobot, 2026-11-05
  • b1c3ecba Dataset lerobot/pusht, revision b1c3ecbae7f244acc039a3dbc255a00dad1372b9 (206 episodes, 25,650 frames) (O · official, first-hand). Source: Hugging Face dataset lerobot/pusht, 2026-01-29
  • d569cc07 First 8 hex digits of the lock file’s sha256 (O · official, first-hand). Source: the lock file shipped with the LeRobot v0.7.0 repository, 2026-11-05
  • 0.95 Definition behind the average max reward: reward per step = coverage ÷ 0.95, clipped to 0–1; coverage above 95% counts as success (O · official, first-hand). Source: source code of the PushT environment gym-pusht, 2026-10-12
  • 8 · 1000 Batch size 8, random seed 1000 (same in both runs, one seed only) (O · official, first-hand). Source: the training config from this run (train_config.json; not published), 2026-11-11
  • Log replay The terminal in the video is an excerpt replayed from the two training logs, not a screen recording; paths replaced, timing rebuilt from the logs (O · official, first-hand). Source: production notes for this episode, 2026-11-13