Physical AI Lab 2.2: Training ACT with LeRobot — Mapping Dataset Metadata to the Model’s Inputs, Then Checking the Training and Evaluation Commands Item by Item

The episode 2.2 video uses LeRobot to go once through turning demonstrations into a policy. The input is the image the arm sees and its joint angles right now; the output is the positions it should reach next. The model is ACT (Action Chunking with Transformers), which predicts a short chunk of future actions at once. The data is svla_so101_pickplace, a public dataset from the LeRobot community holding 50 real-robot pick-and-place demonstrations on an SO-series arm. Training is one command; evaluation means going back to the real robot. The episode’s number cards come from two public sources: the dataset’s meta/info.json and the ACT page of the LeRobot documentation (about 80 million parameters, often a high success rate with just 50 demonstrations, about a few hours for 100k steps on a single GPU). This page is the episode’s reading and exercises. Rather than repeat the transcript, it reads the two sources side by side: which ACT input or output each metadata field corresponds to, what each flag of the documented training command does and leaves unsaid, and where the evaluation example and the training data disagree on robot type and cameras. One rule runs through the page: a command or a number holds only for the data and the documentation version that state it, so match every item before you run anything.

The video, transcript and sources for this lesson are at the top of this page; the season plan is in the course outline. The previous lesson, Physical AI Lab 2.1: Building a Grasping Scene in MuJoCo — Working Out the Mass, Units, Timestep and Collision Rules the XML Leaves Unwritten, ended by asking: now that the scene exists, how does the robot learn to act from data? This lesson picks up there. The number of episodes, frames and total duration of this dataset were already worked out from the same metadata in Physical AI Lab 1.5: Where Does Data Come From? Working Out Demonstration Time from SO-101 Metadata, Then Separating the Units and Action Spaces of Three Sources. The experimental conditions behind the ACT paper’s “about 10 minutes of demonstrations, 80–90% success” were covered in Physical AI Lab 1.6: Imitation Learning — Working Out How an Error Rolls Out of the Demonstrations’ Coverage, Then Putting the ACT, Diffusion Policy and Data Scaling Numbers Back in Their Own Conditions. Stating conditions, pairs and trial counts behind a success rate was practised in Physical AI Lab 1.8: Evaluation — Splitting a “Net Gain of 2” into a Paired Table, Then Checking the Failure Labels and Random Seeds. This lesson does not repeat them; it covers only the LeRobot documentation page the video newly relies on, and what has to be checked between it and the metadata.

The public quiz’s 5 questions come with full explanations. Signed-in readers can work through three exercises: map the metadata fields to ACT’s inputs and outputs, work out how long several chunk lengths last, and write a training command for a fictional dataset; use fictional timings to keep a training budget, separating steps, batch size and samples; and review a fictional real-robot evaluation plan before it runs, then rewrite its report. Each comes with a step-by-step solution, common mistakes and a 20-point rubric.

1. What the two sources say

The dataset metadata. meta/info.json is a JSON file. The table lists the fields and values it records; it is a summary of metadata facts, not quoted sentences:

The episode’s two number cards correspond to two passages in the metadata, each excerpted independently:

"total_episodes": 50

"gripper.pos"

The second is the last entry in the name lists of action and observation.state: the first five entries are arm joints and the last is the gripper, which together make the video’s “six position values”.

The ACT page of the LeRobot documentation. The three sentences behind the episode’s three number cards, each excerpted independently:

Lightweight : Only ~80M parameters, making it efficient and easy to work with

Data Efficient : Often achieves high success rates with just 50 demonstrations

Expect a few hours for 100k training steps on a single GPU

The page also says that ACT is a lightweight and efficient policy for imitation learning, especially well suited to fine-grained manipulation; that because of its fast training, low computational requirements and strong performance it is the first model recommended when starting out with LeRobot; and that it comes from the paper Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware by Zhao et al. Its training tips add two more points: ACT’s default hyperparameters work well for most tasks, and batch size should start at 8 and be adjusted to the GPU’s memory.

Three things to keep in mind when reading these sources:

• Both are level O, but of different kinds. The metadata is the dataset file’s own record; the documentation is the LeRobot team describing its own model. This site has not run the training or the evaluation and has not checked any of these items.
• The two 50s are not the same thing. The dataset has 50 demonstrations; the documentation says ACT often reaches a high success rate with just 50 demonstrations. The latter is a general statement in the documentation that names no task, robot or evaluation conditions; it is not a prediction for this dataset, and the episode’s number card notes that it was not measured in the video. The ACT paper has its own conditions for “10 minutes or 50 demonstrations”; see 1.6.
• “A few hours” gives no GPU model, batch size or task. It is a time reference for 100k steps, not a number of steps training must reach, nor a condition for a valid evaluation.

Limitations

• meta/info.json was read from the dataset’s main branch, fetched on 2026-10-14; the main branch may change. Lesson 1.5 read the same metadata at a pinned commit, and the episode count, frame count, frame rate and number of cameras it reports match this lesson. The metadata is level O; this lesson did not download or check the data files or videos, and the metadata does not describe the recording method, demonstration quality or any hardware detail beyond the robot_type field.
• The ACT page of the LeRobot documentation is level O vendor documentation, read at an unversioned address and fetched on 2026-10-14; the page offers version switching including main, v0.6.1, v0.5.1 and v0.4.4. This lesson does not bind a LeRobot version, and command names and flags may change between versions. About 80 million parameters, 50 demonstrations and a few hours for 100k steps are general statements in the documentation, not measurements by this site and not benchmarks on this dataset; “a few hours” names no GPU model.
• In the extracted page text, ${HF_USER} has spaces on both sides and –wandb.enable= is followed by a space, and in the evaluation command the comment on the –task line shares a line with –duration=60. This lesson removed those spaces from the training command and lists everything else as the page shows it. The documentation’s text says lerobot-record while its example uses lerobot-rollout; this lesson does not decide which matches the current version.
• The meanings of –output_dir, –job_name and –policy.repo_id, chunks_size referring to a grouping of data files, action being the training target, and the assumptions that one sample corresponds to one frame and that actions execute at the dataset’s frame rate are all the lesson’s inferences or assumptions; neither the documentation nor the metadata says so.
• k = 30 and the k values in the exercises, cup_pick_v1, all timings, the checkpoint interval, team W’s plan and report, and the trial and success counts are fictional, for practice only; 1.0 s, 180, 67.0 passes, 4.17 h, 33.8 min, 85%, 45% and the like hold only under these assumptions and are not the performance of any system.
• This site has not run the episode’s notebook, the training command or the evaluation command. The evaluation command drives a real robot arm; hardware safety measures are outside this lesson, and neither paper calculations nor simulation results are hardware safety validation. The dataset name contains so101 but robot_type is so100_follower, and this site has no first-hand source confirming the recording hardware, so both this page and the video say only “an SO-series arm”.

Links

• Physical AI Lab: a physical AI course where every number can be checked (course outline)
• Physical AI Lab 2.1: Building a Grasping Scene in MuJoCo — Working Out the Mass, Units, Timestep and Collision Rules the XML Leaves Unwritten
• Physical AI Lab 1.5: Where Does Data Come From? Working Out Demonstration Time from SO-101 Metadata, Then Separating the Units and Action Spaces of Three Sources
• Physical AI Lab 1.6: Imitation Learning — Working Out How an Error Rolls Out of the Demonstrations’ Coverage, Then Putting the ACT, Diffusion Policy and Data Scaling Numbers Back in Their Own Conditions
• Physical AI Lab 1.8: Evaluation — Splitting a “Net Gain of 2” into a Paired Table, Then Checking the Failure Labels and Random Seeds
• Hugging Face: meta/info.json of lerobot/svla_so101_pickplace
• LeRobot documentation: ACT (Action Chunking with Transformers)