Physical AI Lab 3.5: Fine-Tuning GR00T in Practice — Checking the Fine-Tuning Command and the Modality Wiring Item by Item, Then Using a CPU Stand-In to Separate Open-Loop Error from Closed-Loop Success

Fine-Tuning GR00T in Practice — Checking the Fine-Tuning Command and the Modality Wiring Item by Item, Then Using a CPU Stand-In to Separate Open-Loop Error from Closed-Loop Success. English video, reading, self-checks and sources in one lesson, updated 2026-11-15.

Lesson video

Narration is AI-generated. Video chapters, transcript and sources

Transcript

Transcript

UNIT-01 asks

You can download a big humanoid model. But it's never seen your robot. How do you make it listen to you? This episode has one goal. Take the open GR00T model, and fine-tune it to a new robot. From data to deployment, all the way through.

Meet GR00T

GR00T is NVIDIA's open humanoid model. It looks at images, listens to instructions, and outputs actions directly. One model, a whole set of skills. It isn't tied to one kind of robot. The same model can move to different bodies. That's what we call cross-embodiment. It also has open weights. So you can change it, and build it into your product. NVIDIA says it has three billion parameters. NVIDIA also says its license allows commercial use. Inside, it has two parts. One does the thinking, one generates actions in real time. One slow, one fast; that's the dual system. Fine-tuning is easy, because it already learned a lot in pre-training. You don't start from a blank page. A high starting point means fast training. Think of it like this. It's an apprentice who has seen the world. You just teach it your house rules. NVIDIA says it was pre-trained on more than twenty thousand hours of video. Mostly first-person footage from cameras people wore while they worked. Even more important is a pattern. NVIDIA says that growing the human video from one thousand hours to twenty thousand more than doubled the average completion rate. So NVIDIA says, from one thousand to twenty thousand hours, the more it watches us work, the more skilful its hands! NVIDIA also says the benefit is concrete. Even a dexterous hand with twenty-two degrees of freedom can do fine work.

Prepare your data

The first job in fine-tuning: put your data in GR00T's format. It uses Le Robot's format. That's how the model can read it. Better not to build the format from scratch. Before you start, check the repo's instructions for conversion scripts and examples. If they're there, adapt them; no need to start from zero. Each piece of data needs images, the robot's state and actions, plus the task instruction. Miss one, and the model can't line them up. All aligned: that's good data. There's also a small file called modality. It says which of the robot's joints each number in the action belongs to. It's like a wiring diagram. If the config is wrong, training still runs. But the learned actions won't line up. So check it first. Practise on example data first, just a few episodes. A robot arm puts a cube into a bowl. A simple task, good for getting started. Once the data is ready, we can start training. But how big a machine does training need?

One command to train

The good news is that fine-tuning doesn't mean training from scratch. You just keep training a little on your own data. It saves time, and it saves data. It's a single command. Point it at the base model and your data. Then add the robot tag and the modality config. There aren't many parameters, but each one has to be right. Base, data, tag, config: you need all four. One wrong, and the whole run is wasted. A single GPU can run it. Which GPU, and how much memory, follow the repo's instructions. Check before you start. The rest is waiting for the loss to come down. Keep an eye on the curve while it trains, and don't let it blow up. Going steadily down is right. After a while, the weights are saved. These weights are your robot's own brain. Training from scratch takes ages, but just a little more training works. That saves so much!

Evaluate and deploy

When training's done, how do you know if it's any good? Start with an open-loop evaluation. Open-loop means using recorded data. The model predicts the actions. You compare them with the real ones. The gap is plain to see. Plot both sets of actions as curves. In our illustration, one line is the prediction and one is the truth. The closer the two lines, the more it has learned; that's the most direct signal. To measure the gap, you can use the mean squared error. It's especially sensitive to big deviations, so it catches the wild predictions. Don't just look at one number. Try more episodes and seeds, then decide. A number has to hold up when repeated. But open-loop is only a necessary condition. Accurate predictions don't mean the robot can actually get the job done. That has to be said clearly. To really confirm it, connect it to a robot and run closed-loop. Run it in simulation first; that's safer. Once it's stable, move to the real robot. Deployment is simple too. Start a policy server. The robot sends images and gets actions back. A clean interface. For more speed, you can use TensorRT. Or run it on an edge device. It depends on where you deploy. One more thing, to be honest. Train twice on the same data, and the results jitter a little. Don't mistake noise for improvement.

Hands-on: run the example

Let's go through it. First, clone the repo together with its submodules. Miss the submodules, and you'll get errors later. Set up the environment and log in to Hugging Face. Then request the access the repo's instructions list. Without it, loading fails. Then, on the example data, run the fine-tuning command. For the robot tag, pick the new-embodiment option. While it trains, keep the training log open, and watch the loss and throughput. If it blows up, stop early. When it's done, run an open-loop evaluation. Plot the prediction and the truth together: do the two lines stick together? You can tell at a glance. Did the reproduction succeed? See whether the two lines are clearly closer. And compared with the untuned model, how much better is it? What's still unsolved is this. Good open-loop results don't guarantee a stable closed loop. On a real robot, errors pile up step by step. Let's recap, three points. First, NVIDIA's GR00T is an open humanoid model you can use commercially. Second, before fine-tuning, your data must be in the LeRobot format. Third, keep training with one command, then check with an open-loop evaluation. Understand it, prepare the data, run one command, and you can make a big model your own too! But these success rates, these improvements, how do you tell if they're real or exaggerated? See you next time!

Evidence and sources

3 billion · parameters: GR00T N1.7 is an open 3 billion parameter vision-language-action model (sees images, takes instructions, outputs continuous actions) [O] · NVIDIA · Hugging Face blog “GR00T N1.7”

Commercial use · licence: NVIDIA describes GR00T N1.7 as open and commercially licensed [O] · NVIDIA · Hugging Face blog “GR00T N1.7”

20,854 hours · of human egocentric video used to pre-train GR00T N1.7 (EgoScale, over 20 task categories) [O] · NVIDIA · Hugging Face blog “GR00T N1.7”

More than double · NVIDIA says: going from 1,000 to 20,000 hours of human video more than doubles average task completion (dexterity scaling law) [O] · NVIDIA · Hugging Face blog “GR00T N1.7”

22 DoF · hands: this dexterity law lets 22 degree-of-freedom dexterous hands do contact-rich fine tasks [O] · NVIDIA · Hugging Face blog “GR00T N1.7”

Practice and self-check

The quiz is public and explains each answer as you go; answers to the guide's exercises are for signed-in members.

Bar chart titled "CPU stand-in: fine-tuning above scratch, far below expert (measured here)", with a subtitle saying it shows closed-loop success in percent, 200 test starts per seed, measured here, not GR00T. Four horizontal bars drawn from zero, top to bottom: base used as is, mean of 3 seeds, 0.0%; trained from scratch, mean of 3 seeds, 10.7%; fine-tuned from base, mean of 3 seeds, 33.2%, which is highlighted; scripted expert, same test starts, 98.5%. A source line runs along the bottom.
Closed-loop success in this course's CPU stand-in experiment (pretrained on planar three-joint arm A, transferred to arm B), 200 held-out test starts per model per seed; the bars are means over 3 training seeds, calculated by this lesson. The highlighted bar is the fine-tuning result this article discusses; training from scratch is lower and the scripted expert higher. Per-seed values are 24.5% to 41.5% for fine-tuning and 9.0% to 12.0% from scratch, with no overlap; they are in the table in the text. This is not GR00T, has no vision input and is not a real-robot result. Measured here.

The video for episode 3.5 fine-tunes the open GR00T N1.7 model to a new robot. It first introduces the model, then puts the data into the LeRobot format and writes the modality config, then runs one command that continues training a little from the pretrained model, and finally checks the result with an open-loop evaluation, with the warning that good open-loop results do not mean a stable closed loop on a real robot. All the number cards in the video come from NVIDIA’s GR00T N1.7 blog post on Hugging Face. This page is the reading and exercises for the episode. It does not repeat the transcript; it does three things the video mentions without working through. It goes through the fine-tuning command in the blog post flag by flag, separating what the blog states, what you must supply and what you have to look up in the repository’s instructions. It treats the modality config as a wiring diagram, checked against recorded ranges before training and reported per dimension during evaluation. And it uses a small stand-in experiment that this course ran on a CPU to show what open-loop error on the training, validation and test sets, and closed-loop success, each tell you. One rule runs through the whole page: open-loop error and closed-loop success are two different quantities and must be reported separately; a number belongs only to the model, data and setup that produced it, and the stand-in’s numbers are not GR00T results.

The video, transcript and sources for this lesson are at the top of this page; the season plan is in the course outline. The previous lesson, Physical AI Lab 3.4: Long-Horizon Tasks — Working Out Per-Step and Whole-Chain Success by Hand, Seeing Which Quantity Progress Memory Changes, Then Reviewing a Routing Ablation That Swaps Only the Upper Layer, ended by asking: how do you tune that low-level policy to your own robot? This lesson picks up that question. The structure and parameters of GR00T N1’s dual system were taken apart in Physical AI Lab 3.1: Inside VLAs — Encoding One Cup Motion into 256-Bin Tokens, Then Labelling the Scope of Three Models’ Parameter Counts and Rates. How LeRobot dataset metadata maps to a model’s inputs, and how to keep steps and batch size apart in the accounting, were covered in Physical AI Lab 2.2: Training ACT with LeRobot — Mapping Dataset Metadata to the Model’s Inputs, Then Checking the Training and Evaluation Commands Item by Item. The command for fine-tuning a VLA from an open base model, and the scope of its numbers, were covered in Physical AI Lab 2.3: Fine-Tuning SmolVLA — Planning Demonstrations for an Instruction, Checking What Each Number Measures, Then Timing the Asynchronous Action Queue. Paired tables and intervals were worked out in Physical AI Lab 1.8: Evaluation — Splitting a “Net Gain of 2” into a Paired Table, Then Checking the Failure Labels and Random Seeds and Physical AI Lab 2.8: Reproducible Evaluation — Working Out Wald and Wilson Intervals by Hand, Sizing a Run from 1/√n, Then Bounding the Failure Rate after Zero Failures. This lesson does not repeat any of that; it covers only the fine-tuning entry point in the GR00T N1.7 blog post, how to check the modality config, and how far apart open loop and closed loop are. The next episode, 3.6, is about how to read papers and check their numbers.

The public quiz’s 5 questions come with full explanations. Signed-in readers can do three exercises: follow the blog’s command to write a fine-tuning run card for a fictional robot arm, working out the number of samples and splitting the data by whole episodes; review a draft modality config against fictional recorded ranges, then compute per-dimension open-loop error; and use the record of this course’s stand-in experiment to review a fictional evaluation memo and rewrite its conclusion. Each comes with a step-by-step worked solution, common mistakes and a 20-point rubric.

1. What the GR00T N1.7 blog post says, and what it leaves out

The blog post is NVIDIA’s introduction to its own model, published on April 17, 2026, and is level O in this course’s scheme. The structure, numbers and uses below are all the vendor’s reports; this site has not downloaded or run GR00T N1.7. The post opens by saying this release is an Early Access version and describes it as an open humanoid VLA with a commercial licence; the original wording is:

open, commercially licensed

The video’s “3 billion parameters” card corresponds to this sentence:

GR00T N1.7 is a 3B-parameter open reasoning Vision-Language-Action (VLA) model that maps visual observations and natural language instructions to continuous robot actions.

The post says it uses a dual-system design: System 2 is a vision-language model with a Cosmos-Reason2-2B backbone, and System 1 is a 32-layer diffusion transformer that takes the VLM’s output and the live robot state and denoises them into motor commands (lesson 3.4 already quoted the System 2 sentence). The most useful lines for fine-tuning are the stated inputs and outputs:

Inputs: RGB image frames (any resolution) + language instruction + robot proprioceptive state (joint positions, velocities, EEF poses)

Outputs: Continuous-value action vectors mapped to the robot’s degrees of freedom

Read together, these say that each episode of your data must supply images, an instruction and proprioceptive state, and that each dimension of the action vector the model outputs must map to one of the robot’s degrees of freedom. The video calls the file that spells this out the modality config; the blog post does not explain that file’s format. The fine-tuning entry point gets a single sentence:

N1.7 supports fine-tuning on custom embodiments using the LeRobot dataset format.

The post also says the model was validated on Unitree G1, Bimanual Manipulator YAM and AGIBot Genie 1, and is supported on NVIDIA Ampere, Hopper, Lovelace, Blackwell and Jetson platforms; that pre-training used 20,854 hours of human egocentric video; and that going from 1k to 20k hours of human video more than doubled average task completion. These are all NVIDIA’s reports; the post gives no task list and no trial counts.

For anyone doing the work, what the post leaves out matters just as much: the GPU memory needed, how long the 2000-step fine-tune takes, success rates on any robot after fine-tuning, which tag name a new embodiment should use, and the file format of the modality config. Inference performance is only a link (“4 denoising steps, single camera view”); there are no numbers in the body. All of this has to be looked up in the repository’s instructions and cannot be inferred from the post.

Limitations

  • The GR00T N1.7 blog post is NVIDIA’s introduction to its own model, level O. The parameter count, validation platforms, 20,854 hours of pre-training video, the “more than doubles” completion rate and the “drop-in swap” are all vendor reports. This site has not downloaded or run GR00T N1.7; the commands here are copied from the blog post and were not run.
  • The blog post gives no memory requirement, fine-tuning duration, post-fine-tuning success rate, tag name for a new embodiment or modality config format; follow the repository’s instructions for these. The video’s “a single GPU can run it” corresponds to --num-gpus 1 in the blog’s command; this site has not verified it.
  • The blog’s install steps clone the repository’s default branch without pinning a version. Flag names may change as the repository is updated; this is this lesson’s inference.
  • The stand-in experiment in section 4 is this course’s own CPU run (2026-10-02, measured here): a planar three-joint arm, a small MLP policy, no vision input; it is not GR00T and not a real robot. This text transcribes only the summary numbers from the run record, not the run script or the per-seed Wilson intervals.
  • The following are this lesson’s calculations, summaries or suggestions: the three-seed means and the ratios, the flag table in section 2, the checking order in section 3, and the next steps in item 4 of Task 3.
  • Arm R, its data, limits, draft config and per-dimension errors, colleague Q’s memo and the numbers in each transfer exercise are fictional. Neither paper calculations, the stand-in experiment nor simulation results are hardware safety validation.

Links

Watch this lesson (English video) · 3.5