Lesson video
Narration is AI-generated. Video chapters, transcript and sources
Transcript
Transcript
UNIT-01 asks
It's the same arm. So how come you say one sentence, and it just does it? This episode, we fine-tune an open base model. We teach it to understand a sentence, then go and grab things. Its name is SmolVLA.
From language to action
Last episode's ACT knows only one task. Say it differently, and it's lost. It never learned language. Why only one? It learns from that one set of demos. Never taught anything else, it can't do anything else. Here's what we want instead: give it a sentence, plus what it sees. And it outputs the action to take. For a robot, that sentence is the task. It has to understand first, then act. This kind of model has a name: vision-language-action model, or VLA for short. It takes in three things together: the image, the robot's state, and your instruction. Miss one, and the grasp goes wrong. The benefit is direct. New wording, new goal, no retraining. It can still give it a try. Say, put it in the box one moment, put it on the left the next. The same model can try either one. How well it works depends on whether the training data has similar instructions. But the base has never seen your arm or your table. So it still needs fine-tuning. The docs recommend recording about fifty demos.
What is SmolVLA
Our base model is SmolVLA's open base version. It comes from Hugging Face, and you can download it directly. It stands on the shoulders of a model that has seen oceans of pictures and text. So it already knows a cup and a box. Language, it has down. In other words, the big model does the job of knowing the world. It only has to focus on learning actions. It's not big. The official blog says it's a compact model with four hundred fifty million parameters. The paper reports that it's very small, yet it keeps up with models ten times its size. How does it produce actions? With an action expert. It turns random noise, step by step, into smooth motion. Why this approach? Because it can put out a whole stretch at once, instead of popping out one at a time. Smooth motion keeps the robot from shaking. Shaking can knock things over, or miss the grasp. It has another trick: asynchronous inference. While it carries out one stretch, it works out the next in the background. The benefit? The team says responses get about thirty percent faster, and tasks finished per unit of time can double. So one sentence is enough to direct it, and it's a small model that runs on a small GPU!
Fine-tuning it ourselves
Open this episode's notebook. Install the LeRobot package, with the extras for this model, and check the GPU works. That GPU check matters. Without a GPU, fine-tuning is impractically slow. No GPU at hand? Rent one in the cloud. The data is last episode's SO-101 pick-and-place. Same data, a smarter model. No need to record it all again. We swap in a new model to compare. Same data, so where is the VLA stronger? Train it and see. First, look at the data. Every demo comes with a one-sentence task description. That is the language the model must learn to understand. The key step: fine-tune from the base, don't train from scratch. The base has seen lots of manipulation. It saves data, and time. In fine-tuning, most of what the base can do stays. It only adjusts slightly, on your data. It's like a master taking on a new apprentice. The groundwork is there. It just needs pointers for your particular job. One command. Tell it where the base is, where the data is, and to run on the GPU. Those three are the key settings. Leave the rest on their defaults for now. On the first run, GPU memory may run out. Lower the batch size. It's slower, but it fits on a small GPU. Set the batch size and steps small first, for a smoke test. Once it runs through, scale up following the docs. How long does a full fine-tuning run take? The docs say twenty thousand steps: about four hours on a single A100 GPU. While training, keep an eye on the loss as usual. And evaluation? Put the model back on the real robot, and give it the same sentence. And don't evaluate just once. Try several phrasings and several layouts to judge it right. Only then do the numbers hold up.
Results and limits
Now, the results. One number says a lot. It's about pre-training on community data. The team's experiments show: after pre-training, the success rate on SO-100 jumps to seventy-eight point three percent. Pre-training means first building a foundation on a huge pile of manipulation, then learning your small task. The deeper the foundation, the faster it learns. This is a big trend too: a big model lays the foundation, a little data finishes the job. It saves money and effort. So the order is: pre-train to build the foundation, then fine-tune with your small amount of data. The team says asynchronous inference also makes it more robust. Push it, or move an object, and it carries on. On a real robot, that matters a lot. So async isn't just about speed. It helps the robot keep up when something unexpected happens. But async has a cost. It needs a policy server running in the background, one more layer to set up. Small projects can skip it for now. And one thing isn't solved. Give it an object it's never seen, and it can still fail. Another trap. The wilder your request, the more likely it goes off the rails. Keep instructions close to what it learned. In the end, it's just a student. It answers well only what you've taught it. Language brings flexibility, but generalization still comes from data. The more, and the more varied, the steadier. Let's recap, three points. One: VLA models turn what they see and one sentence together into actions. Two: the SmolVLA model. It is compact, with four hundred fifty million parameters, and it can be fine-tuned on a small GPU. Three, the team says asynchronous inference reacts faster and copes better with disturbances, but generalization still comes from data. So changing the task with one sentence works because the world it has seen is big and varied enough! Then teaching it to walk takes countless practice runs in simulation. How do you train that? See you next time!
Evidence and sources
Practice and self-check
- Interactive quiz5 questions, explained as you answer
- One-page summaryprintable; opens in a new tab
- Download the notebookszip in Chinese, English and Japanese; run in Jupyter or upload to Colab
The quiz is public and explains each answer as you go; answers to the guide's exercises are for signed-in members.
The episode 2.3 video swaps last episode’s ACT for SmolVLA. SmolVLA is a vision-language-action model (VLA): it turns the image, the robot’s state and one language instruction together into actions. The video makes three points: a VLA also takes the instruction as an input; SmolVLA is compact (about 450 million parameters) and is fine-tuned from the open base smolvla_base rather than trained from scratch; and asynchronous inference computes the next stretch while executing the current one, but on objects it has never seen, generalization still depends on data. The episode’s number cards come from three public sources: Hugging Face’s SmolVLA blog post, the SmolVLA page of the LeRobot documentation and the SmolVLA paper (arXiv 2506.01844). This page is the episode’s reading and exercises. Rather than repeat the transcript, it does three things the video only touches on: it shows what that one instruction is in the model, in the data and in the evaluation command; it puts each number back in its own experiment or documentation conditions; and it uses a fictional action queue to work out when asynchronous inference avoids idle time and when it still stalls. One rule runs through the page: a number describes only the model, platform and setup that produced it, and the instruction used in evaluation must trace back to the sentence written into the data when it was recorded.
The video, transcript and sources for this lesson are at the top of this page; the season plan is in the course outline. The previous lesson, Physical AI Lab 2.2: Training ACT with LeRobot — Mapping Dataset Metadata to the Model’s Inputs, Then Checking the Training and Evaluation Commands Item by Item, ended by asking: to make it understand a sentence before it acts, how do you train it? This lesson picks up there. How to read lerobot-train flags such as –dataset.repo_id, –policy.device and –output_dir, and how to keep separate accounts of steps, batch size and samples, were covered in 2.2. Stating conditions, pairs and trial counts behind a success rate was practised in Physical AI Lab 1.8: Evaluation — Splitting a “Net Gain of 2” into a Paired Table, Then Checking the Failure Labels and Random Seeds. This lesson does not repeat them; it covers only what SmolVLA adds: how a language instruction enters the model, the data recipe and commands for fine-tuning from the base, and asynchronous inference.
The public quiz’s 5 questions come with full explanations. Signed-in readers can work through three exercises: plan the number of demonstrations, the instruction strings and the fine-tuning commands for a fictional “put it in the left or right box” task; build a ledger of what eight numbers from the public sources measure, recompute them and judge four claims; and work out the time margin of a fictional asynchronous action queue, then review three deployment claims. Each comes with a step-by-step solution, common mistakes and a 20-point rubric.
1. What the three sources say
The Hugging Face blog post and the LeRobot documentation are the SmolVLA team describing its own model, level O in this course’s grading; the paper is a preprint, level R, and this lesson read only its abstract. All the numbers they give are author-reported, and this site has reproduced none of them. The passages behind the episode’s number cards follow, each excerpted independently.
The blog post:
Today, we introduce SmolVLA , a compact (450M), open-source Vision-Language-Action model for robotics that runs on consumer hardware.
Supports asynchronous inference for 30% faster response and 2× task throughput .
After pretraining on community-collected data, performance jumps to 78.3% , a +26.6% absolute improvement .
The paper’s abstract:
Despite its compact size, SmolVLA achieves performance comparable to VLAs that are 10x larger.
The documentation (the first sentence refers to a similar dataset of only 25 episodes):
it was not enough leading to a bad performance
Training the model for 20k steps will roughly take ~4 hrs on a single A100 GPU.
The same sources hold several related numbers. The table puts them next to their conditions; it is the lesson’s summary of the sources, not quoted sentences:
| Number | Source and level | What it measures | Conditions the page states | Not stated in the text this lesson read |
|---|---|---|---|---|
| 450M | Blog, O | Parameter count of the whole SmolVLA model | Compact, open source, runs on consumer hardware | — |
| About 100M | Blog, O | Parameter count of the action expert (one of two core components) | A compact transformer trained with flow matching | — |
| 10× | Paper abstract, R | Performance comparable to VLAs about 10 times larger | Evaluated on a range of simulated and real-world robotic benchmarks | The abstract names neither the models compared nor the specific benchmarks |
| About 30% (9.7 s vs 13.75 s) | Blog, O | How much faster async completes a task on average than sync | Real-world tasks; both modes reach similar success (about 78%) | This passage names no robot and no trial count; the TL;DR calls it response |
| 2× (19 vs 9 cubes) | Blog, O | Completions within a fixed time | Fixed-time settings | The fixed duration |
| 78.3% (51.7% without pretraining) | Blog, O | Success rate on SO100, with and without community-data pretraining | The page lists it as the result in Table 1 | The table’s contents and trial counts |
| About 50 episodes (25 not enough) | Docs, O | Recommended number of demonstrations to record as a starting point | Enough demonstrations for each variation | Not a guarantee of success |
| 20k steps in about 4 hours | Docs, O | Time to fine-tune from smolvla_base | Single A100; steps should be tuned to performance and use case | This sentence gives no batch size (the command example on the same page uses 64) |
Three things to keep in mind when reading this table:
- “30%” and “2×” are two different measures. The results passage says async completes tasks about 30% faster (9.7 s vs 13.75 s) and doubles completions within a fixed time (19 vs 9 cubes). The first is an average time, the second a count, and the page gives no fixed duration, so neither can be derived from the other.
- “+26.6%” is in percentage points. 78.3 − 51.7 = 26.6 is an absolute difference; the relative improvement has to be worked out separately.
- Do not merge the two “78%” figures. The 78.3% from the pretraining comparison and the “about 78% in both modes” from the sync/async comparison come from two different results in the blog post; the text this lesson read does not say whether they come from the same set of trials, so record them separately.
Sign in to keep reading: the rest of this lesson, its self-check questions and the answers.
Limitations
- The SmolVLA blog post is level O and was published on 3 June 2025; the SmolVLA page of the LeRobot documentation is level O and was read at an unversioned address; both were fetched on 2026-10-17. The blog’s commands use a script path and the documentation uses lerobot-train, so command names and flags may change between versions; this lesson binds no LeRobot version. The paper is a level-R preprint, and this lesson read only its abstract.
- 450M, about 100M, 10×, about 30%, 2×, 78.3%, about 50 episodes and 20k steps in about 4 hours are all author-reported; this site has reproduced none of them. The blog text this lesson read does not include the tables’ contents, so the trial counts behind these results are unknown, and the sync/async passage names no robot.
- When the page text was extracted, spaces appeared after several equals signs in the commands and on both sides of ${HF_USER}, and –policy.type was split apart; this lesson removed those spaces. The documentation’s sentence about training with a notebook without a GPU is cut off, and this lesson has not filled it in.
- The following are the lesson’s inferences or assumptions, not statements in the sources: reading 70% as the remaining queue’s share of the chunk length; counting one image from each camera as one frame; that fusion changes the actions actually executed; that a new chunk is usable on arrival; and that “10 episodes per variation” can serve as a planning reference.
- Teams M and V, block_two_boxes, the positions and target boxes, the smoke-test values, chunk lengths, execution rates, delays, server locations and the reports in the exercises are fictional; 1.17 s, 19.4%, 80 episodes, 87.5%, 46.7% and the like hold only under these assumptions and are not the performance of any system.
- This page has not run the episode’s notebook or the training or evaluation commands. The evaluation command and the policy server drive a real robot arm; hardware safety measures are outside this lesson’s scope, and neither paper calculations nor simulation results are hardware safety validation.
Links
- Physical AI Lab: a physical AI course where every number can be checked (course outline)
- Physical AI Lab 2.2: Training ACT with LeRobot — Mapping Dataset Metadata to the Model’s Inputs, Then Checking the Training and Evaluation Commands Item by Item
- Physical AI Lab 1.8: Evaluation — Splitting a “Net Gain of 2” into a Paired Table, Then Checking the Failure Labels and Random Seeds
- Hugging Face blog: SmolVLA: Efficient Vision-Language-Action Model trained on Lerobot Community Data
- LeRobot documentation: SmolVLA
- SmolVLA paper (arXiv 2506.01844)
