Physical AI Lab 3.1: Inside VLAs — Encoding One Cup Motion into 256-Bin Tokens, Then Labelling the Scope of Three Models’ Parameter Counts and Rates

Inside VLAs — Encoding One Cup Motion into 256-Bin Tokens, Then Labelling the Scope of Three Models’ Parameter Counts and Rates. English video, reading, self-checks and sources in one lesson, updated 2026-11-08.

Lesson video

Narration is AI-generated. Video chapters, transcript and sources

Transcript

Transcript

UNIT-01 asks

Look at a picture, hear an instruction, then act. Do three robot models all do it the same way? This episode, we take apart three vision-language-action models. We look at their vision, their language backbone and their action head, and where they differ.

One shared skeleton

First, what they share. A vision-language-action model, or VLA for short. Give it an image and one instruction, and it outputs the robot's actions directly. We'll follow one task through the whole episode: put the cup on the table into the basket. The same task, through three models. Compared with a chat model, a VLA has one hard constraint. Its output has to drive motors. So it must understand meaning, and geometry too. Taken apart, it has three blocks. The first sees the world, the second understands language, and the third generates actions. This episode, the star is block three. One more key word: cross-embodiment. People want one model to control a robot arm, two arms, even a humanoid. In the past, a robot was split into stages, perception, planning and control. A VLA tries to merge them into one block, learned end to end. They are Open V-L-A, π-zero and GR00T N1. From Stanford and other institutions, Physical Intelligence, and NVIDIA. The papers say Open V-L-A and GR00T have been publicly released.

Action heads: three routes

The action head is how the model turns an idea into a real sequence of actions. Three models, three different routes. First, Open V-L-A. It treats actions as text. The paper says it splits each action dimension into two hundred fifty-six bins, or buckets. The upside is simplicity. The way a language model writes words is the way it writes actions. The π-zero paper says this approach struggles with high-frequency, highly dexterous tasks. How does flow matching work? Give the model a cloud of noise, then remove the noise step by step. The noise ends up as one smooth motion. Next, π-zero. It doesn't output tokens. It uses flow matching instead. The paper calls it a diffusion-style way to generate actions. Next to the language model, it adds a small action expert. The paper says action chunking plus flow matching lets control run at up to fifty hertz. Third, NVIDIA's GR00T N one. The whole model is a dual system, one slow and one fast. The paper says the slow one is the vision-language backbone, which looks and understands; the fast one is the action module that generates actions. Here's a metaphor we can use. The slow system is like the brain, thinking slowly. The fast system is like the cerebellum, moving quickly. Still, it's only an analogy. That fast system is a diffusion transformer. The paper says it is also trained with flow matching. It outputs closed-loop actions at up to one hundred twenty hertz. So Open V-L-A predicts tokens, while the other two both use flow matching. π-zero adds an action expert, and GR00T adds its own fast system. Why is continuous generation smoother? Because motion is a continuous quantity. Popping it out word by word is actually awkward. The π-zero paper says action chunking plus flow matching lets it handle fast, fine work. The paper demonstrated this on folding laundry. So the action head is not a standard part. The route you pick decides how steady the hand is!

Vision and language backbones

One block further in are seeing the world and understanding language. Here, the three models stand on the shoulders of different open-source giants. OpenVLA's language backbone is Llama two. The paper says its eyes fuse two sets of visual features together. Why two pairs of eyes? The paper says one leans toward meaning, the other toward space. A working robot needs both. The paper says the whole model has seven billion parameters. It was trained on about nine hundred seventy thousand real robot demonstrations. Pi-zero is more compact. The paper says its backbone is the open model PaliGemma. Add that small action expert. The paper says the two together come to about three point three billion parameters. The third model's backbone comes from NVIDIA. The paper says it uses the Eagle two vision-language model. This Eagle two is also built from parts. The paper says it was fine-tuned from Smol LM two, with SigLIP two as the vision encoder. The paper says the released GR00T N one checkpoint has two point two billion parameters. Smaller than the other two, it mainly targets humanoid robots. Put the three axes side by side: vision encoder, language backbone, action head. That's the map for taking apart any VLA.

Limits and trade-offs

Why not train from scratch? Learning meaning and objects from zero is too costly and too slow. A borrowed large model already knows cups and baskets. One more trade-off. Tokens come one at a time, while chunking plus flow matching can produce a whole chunk at once. That's where latency and precision pull against each other. What counts as strong? Success rate, speed, and how steady it is with objects it has never seen. We'll pick that up in later episodes. Note that more parameters isn't always better. GR00T is the smallest, but small says neither strong nor weak. What fits depends on the task. OpenVLA and GR00T being open to download means something more. You can download them yourself, fine-tune them yourself, even check them yourself. Standing on open models has another upside. Every step the community takes, VLAs can pick up along the way. What's still unsolved is this. The names we took apart don't mean any one of them is stronger. Which is better on a real robot has to be tested one by one. Change the robot or the setting, and the ranking may change again. Let's recap, three points. One, every VLA looks at images, understands language, then outputs actions. Two, action heads: OpenVLA predicts tokens one at a time, and the other two use flow matching. Three, the vision and language backbones each stand on different open models. Can't remember the names? No problem. Remember the three blocks, and that's enough! But where does these models' training data come from? Next time, we go and build a world. See you next time!

Evidence and sources

256 bins · OpenVLA's action head: each action dimension is discretised into 256 bins and predicted as tokens [R] · arXiv 2406.09246 · OpenVLA

Flow matching · π-zero's action head: on a pre-trained vision-language model, flow matching (a variant of diffusion) generates continuous actions [R] · arXiv 2410.24164 · π0

50 hertz · With action chunking plus flow matching, π-zero runs control at up to 50 hertz (per the paper; peak, for dexterous tasks like folding laundry) [R] · arXiv 2410.24164 · π0

Fast + slow systems · GR00T N1 as a whole is a dual system: the slow System 2 (vision-language module) understands, and the fast System 1 (diffusion transformer action module) outputs actions in real time [R] · arXiv 2503.14734 · GR00T N1 (NVIDIA)

120 hertz · GR00T N1's fast system (a diffusion transformer) outputs closed-loop actions at up to 120 hertz (paper) [R] · arXiv 2503.14734 · GR00T N1 (NVIDIA)

Flow matching · GR00T N1's fast system (a diffusion transformer, DiT) is also trained with flow matching to generate actions (paper) [R] · arXiv 2503.14734 · GR00T N1 (NVIDIA)

Llama 2 · OpenVLA's language backbone is Llama 2; its vision encoder fuses DINOv2 and SigLIP features [R] · arXiv 2406.09246 · OpenVLA

7 billion · parameters in OpenVLA (7B); trained on about 970,000 real robot demonstrations (paper) [R] · arXiv 2406.09246 · OpenVLA

PaliGemma · is π-zero's vision-language backbone; its weights are initialised from this open model (paper) [R] · arXiv 2410.24164 · π0

3.3 billion · π-zero has about 3.3 billion parameters in total (a 3 billion VLM + a 300 million action expert, paper) [R] · arXiv 2410.24164 · π0

Eagle-2 · GR00T N1 uses NVIDIA's Eagle-2 as its vision-language backbone (paper) [R] · arXiv 2503.14734 · GR00T N1 (NVIDIA)

SmolLM2 + SigLIP-2 · Eagle-2 is fine-tuned from the SmolLM2 language model and the SigLIP-2 image encoder (paper) [R] · arXiv 2503.14734 · GR00T N1 (NVIDIA)

2.2 billion · GR00T-N1-2B has about 2.2 billion parameters in total, about 1.34 billion of them in the vision-language backbone (paper) [R] · arXiv 2503.14734 · GR00T N1 (NVIDIA)

Practice and self-check

The quiz is public and explains each answer as you go; answers to the guide's exercises are for signed-in members.

Table diagram on a light background, titled "Action heads and total parameters of three VLAs", with a rule under the title and a source line at the bottom. A shaded header band carries two column headings, "Action head" and "Reported total parameters"; a vertical rule separates the left column of row names from them. Thin lines separate three rows, the first and third shaded: OpenVLA, discrete tokens with 256 bins per dimension, 7B; π0 (π-zero), flow matching plus a 300M action expert, 3.3B; the released GR00T-N1-2B, a System 1 DiT trained with flow matching, 2.2B (VLM: 1.34B).
The action-head design and total parameters each paper reports. The three totals are counted differently: OpenVLA’s 7B is the nominal size the paper gives; π0’s 3.3B is the 3B PaliGemma plus a 300M action expert; GR00T’s 2.2B belongs only to the released GR00T-N1-2B, of which the VLM is 1.34B. Row order is not a ranking, and parameter counts do not indicate strength; the three papers do not compare these models on the same robot. All numbers are author-reported. Sources: OpenVLA (arXiv 2406.09246), π0 (arXiv 2410.24164), GR00T N1 (arXiv 2503.14734).

The video for episode 3.1 moves up to L3 and takes apart three vision-language-action models (VLAs): OpenVLA, π0 (π-zero) and GR00T N1. The video makes three points. All three share one skeleton: they look at images, understand language and output actions directly, with three blocks inside for seeing the world, understanding language and generating actions. Their action heads take three routes: OpenVLA cuts each action dimension into 256 bins and treats them as tokens, π0 uses flow matching with an action expert, and GR00T N1’s fast system is a diffusion transformer trained with flow matching. Their vision and language backbones stand on different open models (Llama 2, PaliGemma, Eagle-2), and neither the names nor the parameter counts say which model is stronger. The video’s number cards come from three arXiv papers: OpenVLA (2406.09246), π0 (2410.24164) and GR00T N1 (2503.14734). This page is the episode’s reading and exercises. It does not repeat the transcript. Instead it does three things the video only touches on: it works through “cutting into 256 bins” on a fictional cup-placing motion, to see what sets the bin width and what happens to a value outside the range; it puts the integration steps and the timings that the two flow-matching action heads report back on their own hardware and settings; and it breaks the three totals, 7B, 3.3B and 2.2B, into the blocks each paper names, to see what each one counts. One rule runs through the page: a parameter count, rate or timing belongs only to the paper, model version and hardware that reported it. Numbers from the three papers cannot be converted into one another or assembled into a leaderboard.

The video, transcript and sources for this lesson are at the top of this page; the season plan is in the course outline. The previous lesson, Physical AI Lab 2.8: Reproducible Evaluation — Working Out Wald and Wilson Intervals by Hand, Sizing a Run from 1/√n, Then Bounding the Failure Rate after Zero Failures, ended by asking what the large model that sees, thinks and acts actually looks like inside. This lesson answers that. How π0 re-plans once every 25 actions on a 50 Hz robot, and how the fast and slow systems are layered, were put back in their own systems in Physical AI Lab 0.3: See, Think, Act — Putting Each Rate Back in Its Own System. An action expert trained with flow matching and the asynchronous action queue were covered in Physical AI Lab 2.3: Fine-Tuning SmolVLA — Planning Demonstrations for an Instruction, Checking What Each Number Measures, Then Timing the Asynchronous Action Queue. How long an action chunk should be was worked out in Physical AI Lab 2.6: To the Edge — Working Out the Control-Rate Ceiling from a Latency Log, Telling Whether Quantization Saves Size or Time, Then Sizing an Action Chunk. This lesson does not repeat those. It covers only how the three papers themselves describe the three blocks, and the scope you need to label when you read their numbers. Lesson 0.3 cited a vendor blog about GR00T N1.7; this lesson reads the GR00T N1 paper, and its GR00T numbers belong only to GR00T-N1-2B. The next episode, 3.2, covers world models that generate and transfer training data with Cosmos.

The public quiz’s 5 questions come with full explanations. Signed-in readers can do three exercises: encode three dimensions of a fictional cup-placing action as 256-bin indices and work out the reconstruction and out-of-range errors; break the three models’ totals into blocks, account for integration steps and timings, then judge four statements; and review a fictional VLA selection memo, putting each sentence back in the conditions of the original papers, rewriting it, and listing what to measure on your own robot. Each comes with a step-by-step worked solution, common mistakes and a 20-point rubric.

1. How the three papers describe the three blocks

All three are arXiv preprints, rated R in this course; the structures and numbers below are author-reported, and this site has not reproduced them or run any of the models. GR00T N1 is NVIDIA’s report on its own model. The original sentences behind the video’s number cards follow, each quoted separately.

OpenVLA (Kim et al., authors from Stanford University, UC Berkeley and other institutions, arXiv 2406.09246), from the abstract:

we introduce OpenVLA, a 7B-parameter open-source VLA trained on a diverse collection of 970k real-world robot demonstrations.

OpenVLA builds on a Llama 2 language model combined with a visual encoder that fuses pretrained features from DINOv2 and SigLIP.

From the method section (HTML v1), on actions:

we discretize each dimension of the robot actions separately into one of 256 bins.

The paper’s architecture figure caption splits the model into three parts: a vision encoder that concatenates DINOv2 and SigLIP features, a projector that maps visual features into the language embedding space, and a Llama 2 7B language backbone; given an image observation and a language instruction, the model predicts 7-dimensional robot control actions. It is built on the Prismatic-7B vision-language model: a 600M-parameter visual encoder, a 2-layer MLP projector and a 7B-parameter Llama 2. Input image patches pass separately through both encoders, and the resulting features are concatenated channel-wise; the paper says adding DinoV2 features helps spatial reasoning. The video’s “one leans toward meaning, the other toward space” matches the related-work section: low-level spatial information from DINOv2 combined with higher-level semantics from SigLIP.

π0 (Physical Intelligence, arXiv 2410.24164), from the abstract:

We propose a novel flow matching architecture built on top of a pre-trained vision-language model (VLM) to inherit Internet-scale semantic knowledge.

From the introduction, the framework figure caption and the model section (HTML v1):

This enables our model to control robots at frequencies of up to 50 Hz for dexterous tasks such as laundry folding

The VLM backbone weights are initialized from PaliGemma , providing representations learned from large-scale Internet pre-training.

PaliGemma is an open-source 3 billion parameter VLM that offers a convenient tradeoff between size and performance. We add 300M parameters for the action expert (which is initialized from scratch) for a total of 3.3 billion parameters.

The “This” in the first sentence refers to the previous sentence in the paper: an action chunking architecture with flow matching, which the paper calls a variant of diffusion. The framework figure caption says the model consists of a larger VLM backbone and a smaller action expert that processes robot states and actions. Section IV says image encoders embed the image observations into the same space as the language tokens. The paper also says PaliGemma was used for convenience and because its comparatively small size helps real-time control, while the framework works with any pre-trained VLM. The authors also trained π0-small, a 470M-parameter model without VLM initialization, for ablation comparisons.

GR00T N1 (NVIDIA, arXiv 2503.14734), from the abstract and body (HTML v1):

GR00T N1 is a Vision-Language-Action (VLA) model with a dual-system architecture. The vision-language module (System 2) interprets the environment through vision and language instructions. The subsequent diffusion transformer module (System 1) generates fluid motor actions in real time.

It generates closed-loop motor actions at a higher frequency (120Hz).

We use the NVIDIA Eagle-2 VLM ( Li et al., 2025 ) as the vision-language backbone.

Specifically, our publicly released GR00T-N1-2B model has 2.2B parameters in total, with 1.34B in the VLM.

Eagle-2 is finetuned from a SmolLM2 ( Allal et al., 2025 ) LLM and a SigLIP-2 ( Tschannen et al., 2025 ) image encoder.

The model architecture section adds that GR00T N1 uses flow matching to learn action generation, and that System 1’s diffusion transformer (DiT) reads the image and text tokens output by the VLM through cross-attention, with embodiment-specific encoders and decoders for states and actions of different dimensions. The introduction says System 2 runs at 10 Hz on an NVIDIA L40 GPU.

The table below is this lesson’s summary of the three papers, not original sentences:

BlockOpenVLAπ0GR00T N1
Vision encodingDINOv2 and SigLIP features concatenated channel-wise, 600MImage encoders embed observations in the same space as language tokens; part of PaliGemma, with no separate parameter count in the passages readSigLIP-2 (part of Eagle-2); 224 × 224 per frame, giving 64 image tokens
Language backboneLlama 2 7BPaliGemma (3B, the whole VLM)Eagle-2, fine-tuned from SmolLM2; GR00T-N1-2B uses the LLM’s 12th-layer representations
Action headDiscrete tokens, 256 bins per dimension, trained by next-token prediction300M action expert, flow matching, outputs action chunksSystem 1 DiT, flow matching, 16 actions per chunk
Total parameters reported7B3.3B2.2B (1.34B in the VLM)
What the passages read say is releasedModel checkpoints, fine-tuning notebooks, PyTorch code; a footnote says training data, code and weights for reproducing it are releasedNo statement on whether π0’s own weights are released; only that PaliGemma is an open-source VLMGR00T-N1-2B checkpoint, training data and simulation benchmarks

Two things to note when reading this table. First, “open” means different things in the three papers. OpenVLA’s footnote says that the SigLIP, DinoV2 and Llama 2 models it uses release only their weights, not their training data or code; GR00T states that the released version is GR00T-N1-2B. That is why the video says only that “the papers say OpenVLA and GR00T have been publicly released”, rather than “three open-source models”. Second, “backbone” covers different scopes: π0’s VLM backbone includes the image encoders, OpenVLA lists its vision encoder separately, and GR00T’s 1.34B is the whole VLM.

Limitations

  • All three papers are arXiv preprints, rated R; the structures, parameter counts, rates and timings are author-reported, and this site has not reproduced them or run any model. GR00T N1 is NVIDIA’s report on its own model. This lesson read OpenVLA’s abstract, introduction, related work and section 3 up to the start of the training-data section; π0’s abstract, introduction, related work, overview, the start of section IV and the passage on inference and parameter counts; and GR00T N1’s abstract, introduction, model architecture section and the start of the training-data generation section.
  • π0’s inference times are in Appendix A-D of the paper, which this lesson did not read; the π0 passages read give no action-chunk length and do not say whether π0’s weights are released.
  • The following are this lesson’s conventions, arithmetic or inferences: rounding OpenVLA indices down, reconstructing at the bin centre and clipping out-of-range values; the quantile scheme clipping about 2% of training actions per dimension; a GR00T chunk of 16 actions covering 133 ms when executed consecutively at 120 Hz, with sampling taking about 48% of that; the parameter remainders and shares; the network call counts.
  • The +4.2 mm, the −10 to +10 mm range and the outliers in section 2, and robot D, team Q, model Z, all quantiles, target actions and memo sentences in the exercises, are fictional and are not data from any paper or system.
  • The three papers do not compare these three models on the same robot and task, and this lesson does not make that comparison either. Paper calculations and simulation results are not hardware safety validation.
  • This lesson’s GR00T numbers belong only to the GR00T N1 paper and GR00T-N1-2B; GR00T N1.7, cited in lesson 0.3, is a different version.

Links

Watch this lesson (English video) · 3.1