
An operator teleoperating a robot usually does not say what they intend while they work. The preprint arXiv 2611.16419 (2026-11-12), by authors at ETH Zurich and the Chinese University of Hong Kong, asks whether a vision-language model (VLM) can watch a short stretch of teleoperation and name both the action the operator is performing (such as “pick up the apple”) and the mission that action serves (such as “put the apple on the plate”), and then pass the mission to a VLA as its language instruction so the robot can finish the job. For this the authors built the VI AMO dataset, fine-tuned a model called VIM, and ran human-to-robot handoff experiments on a real Franka arm. All numbers below are reported by the authors [R]; this site has not run the code or reproduced the results.
Two levels of intent: the current action and the mission it serves
The paper splits intent into two levels. The ongoing action (OA) is the single atomic manipulation step being performed. The ongoing mission (OM) is a run of consecutive atomic actions that serve one immediate goal: “pick up the apple” followed by “place the apple in the basket” together form “put the apple in the basket”. The authors argue that both levels are needed because the same action can serve different missions: after picking up an apple, the robot might put it in a basket or hand it to a person. VOSA and CASPER, which the paper cites, represent intent as the target object or position of the current step, or as a skill chosen from a predefined library; they do not explicitly infer the mission behind the action.
VIM takes two inputs: n time-ordered frames, possibly from several cameras, and a text execution context listing the actions already completed within the current mission. When the list is empty, the context reads “Robot has no previous actions.” The output follows a fixed format, “Ongoing action: …” and “Ongoing mission: …”, generated together in one autoregressive pass. Both are open-vocabulary natural language, and no candidate labels are supplied during generation. A predicted mission can therefore serve directly as the instruction for a language-conditioned policy, without a fixed skill library. The difficulty is timing: at handoff only part of the action has been seen, so the model has to commit from a few frames and the record of completed actions.
On this site, Physical AI Lab 2.3 covers how a VLA turns images, robot state and a language instruction into actions, and Physical AI Lab 3.4 covers the upper layer that gives instructions to a low-level policy in long-horizon tasks. This paper changes where the instruction comes from: a person does not write it and a planner does not issue it; it is inferred from what the operator has already done.
The VI AMO dataset: labels mostly drafted by models
VI AMO has 6,600 episodes from 11 robot data sources (including Bridge V2, DROID, Fractal (RT-1) and TACO Play), with 13,653 temporally aligned action annotations and 9,702 mission annotations, split by episode into 5,500 for training, 550 for validation and 550 for testing. DROID contributes 6,372 of the action annotations and Bridge V2 4,950, about 82.9% together (this site’s calculation from the paper’s Table I).
Annotation has five stages:
- Action decomposition: GPT-5.6 Sol lists ordered atomic-action descriptions from the original dataset’s episode instruction and first-frame images.
- Mission grouping: GPT-5.6 Sol judges each pair of adjacent actions. If they clearly continue toward a shared goal, they join one mission; otherwise a boundary is kept. Ungrouped actions form single-action missions.
- Coarse localization: Qwen3.7-Plus estimates a rough time window for each action from sparsely sampled frames.
- Boundary refinement: consecutive frames are taken around each coarse boundary, together with gripper opening and closing events where available, and Qwen3.7-Plus decides the frame at which the action is completed.
- Human review: reviewers check and correct action descriptions, mission groupings and time boundaries in around 50% of the episodes.
VIM uses Qwen3.5-27B as its backbone, with LoRA (rank 16, α 16) on the attention and MLP modules of both the vision and language parts. Only the low-rank parameters are updated, and the loss covers only the OA and OM text. The training set has 11,481 actions; with 4 frames per camera and 6 temporal sampling plans per action, this gives 68,886 training examples. Training runs for one epoch in BF16 and takes about 13.44 GPU-hours on a single A100 80GB.
Test-set results: the same backbone before and after fine-tuning
The 550 test episodes do not overlap with training and contain 1,091 actions and 781 mission instances. Each action is evaluated at 9 progress points (0% to 95% of its duration), with 4 frames sampled uniformly from action onset to that point. There are four metrics: token-level F1; SBERT, the cosine similarity of all-mpnet-base-v2 sentence embeddings; cosine similarity from the SigLIP text encoder; and Top-10, which ranks all unique test labels (726 action labels and 539 mission labels) by SigLIP similarity to the prediction and counts how often the correct label is among the first 10. The table below is taken from the paper’s Table II and averages the 9 progress points:
| Model | Action SBERT | Mission SBERT | Mission Top-10 (%) |
|---|---|---|---|
| GPT-5.6 Luna | 0.581 | 0.554 | 39.33 |
| GPT-4o | 0.563 | 0.470 | 29.91 |
| Qwen Base (27B), not fine-tuned | 0.581 | 0.564 | 37.27 |
| Qwen Base (9B) | 0.529 | 0.496 | 28.32 |
| RynnBrain 1.1-9B | 0.505 | 0.499 | 35.49 |
| RoboBrain 2.5-8B | 0.316 | 0.315 | 19.29 |
| VIM | 0.825 | 0.821 | 78.77 |
None of the baselines was trained on VI AMO. The most direct comparison uses the same backbone: the untuned Qwen3.5-27B scores 0.564 mission SBERT and 37.27% mission Top-10, and the fine-tuned VIM scores 0.821 and 78.77%. Action Top-10 also rises from 39.14% for the best baseline to 76.59%. The two models post-trained for robotics, RynnBrain and RoboBrain, do not beat the general-purpose VLMs, which the authors read as a sign that inferring operator intent from motion is a distinct problem. The diagram in this article draws only four bars, for GPT-4o, the untuned Qwen3.5-27B, GPT-5.6 Luna and VIM; the other 3 baselines in the table are not drawn.
Across progress, VIM’s action and mission SBERT rise from 0.776/0.776 at 0% to 0.855/0.849 at 95%, with smaller gains after 70%; at 0%, when only the action onset has been seen, the scores are already 0.776/0.776. In the ablations, removing the wrist camera lowers the scores to 0.798/0.798, removing the text context to 0.767/0.788, and replacing the 4-frame history with a single frame at 0% to 0.793/0.792.
The authors also split the 1,091 test actions by whether their labels occur in training: 621 have both labels seen, 386 (35.4%) have both unseen, and the remaining 84 have one unseen. On the seen part VIM scores 0.902/0.903 against 0.587/0.566 for the strongest baseline; on the unseen part VIM falls to 0.726/0.717, against at most 0.570/0.564 for the baselines. “Unseen” here is decided by the normalized label string, so a reworded description of a seen label also counts as unseen; it does not mean an entirely new action or mission (this site’s inference).
Real-robot handoff: VIM proposes a mission, the operator confirms, π0.5 continues
The platform is a Franka arm with a parallel-jaw gripper, one OAK-D camera in front and one on the wrist, and a Rokoko Smartglove for teleoperation: relative hand motion moves the end effector, and the thumb-index pinch distance opens and closes the gripper. The handoff works as follows:
- VIM reads the history since the most recent human takeover, samples 4 synchronized front/wrist image pairs uniformly, and predicts the ongoing action and mission.
- The operator reviews the predicted mission and, if it matches their goal, confirms it, handing control to the VLA.
- A fine-tuned π0.5 uses the predicted mission as its instruction and issues actions at a nominal 15 Hz to finish the manipulation.
- The operator can take over again at any time; the glove mapping is re-anchored and the robot follows the operator again. Each later handoff repeats the same procedure.
VIM runs on an RTX 5090 with TensorRT-Edge-LLM, quantized to NVFP4; the paper says this was done to fit the available memory. Here quantization saves memory, which can be read alongside the separate size and time accounts in Physical AI Lab 2.6. π0.5 is fine-tuned on newly collected demonstrations, 130 for the all-fruits placement task and 100 for each of the others, trained jointly for 10,000 steps on 4 A100 80GB GPUs.
The 5 experiments fall into four families, each testing a different intent cue: apple placement and banana placement share a scene and differ only in the target fruit; egg placement needs the commonsense pairing of object and receptacle (egg to tray); trash sweeping requires reading the goal from the tool the robot reaches for; and all-fruits placement needs repeated inference and continuation. None of these scenarios is in VIM’s fine-tuning data. The comparison conditions use GPT-4o or GPT-5.6 Luna as the intent module, plus full teleoperation; each experiment is run 5 times per condition.
The authors report that VIM needs less human teleoperation time than either GPT setup in every experiment, with a mean reduction of 34.6% relative to GPT-4o and 39.8% relative to GPT-5.6 Luna. In trash sweeping, neither GPT model produced a usable intent, so that experiment stayed fully teleoperated. Mean latency per prediction is 0.70–0.81 s for VIM across experiments, 3.09–3.53 s for GPT-4o and 1.92–2.27 s for GPT-5.6 Luna, reductions of 76.9% and 63.8% (the abstract summarizes this as roughly 70% on average). The authors explain that many manipulation actions last only a few seconds, so delays above 2 seconds let the inferred intent fall behind the robot’s actual state.
Three points matter when reading these results. First, all conditions use the same VLA, so only the intent module is compared. Second, the text gives only the percentages relative to the two GPT baselines; savings relative to full teleoperation and absolute times are plotted in Figure 5, and the text read here gives no values for them. Because both GPT baselines fell back to full teleoperation in trash sweeping, that experiment likely widens the average gap (this site’s inference). Third, VIM is a quantized model on a local GPU, while the GPT baselines are API calls, so the latency gap includes differences in deployment, not only in the models (this site’s inference).
Limitations
- This is an arXiv preprint, all numbers are reported by the authors [R], and nothing has been independently reproduced. The test-set results come without intervals, standard deviations or multiple seeds. The real-robot part has only 5 trials per experiment per condition, and its means and standard deviations appear only in Figure 5; the text read here gives no values.
- The reference answers used for evaluation were drafted by GPT-5.6 Sol and localized by Qwen3.7-Plus, and humans reviewed only about 50% of the episodes. VIM was fine-tuned on these labels and learned their wording; the untuned baselines did not, so the text-similarity metrics may favor VIM (this site’s inference; the paper does not discuss it).
- Test episodes do not overlap with training episodes but come from the same pool of sources (10 of the 11; HYDRA has no test episodes). The real-robot scenarios are not in VIM’s fine-tuning data, but π0.5 was fine-tuned specifically for those experiment tasks with 100–130 demonstrations each.
- Limitations the authors state: the model still makes prediction errors and has no calibrated confidence, so knowing when to trust a prediction remains an open problem; the dataset evaluation assumes known action boundaries and does not assess online boundary detection during live teleoperation; and the VLA executes more slowly than a skilled operator, so less manual control time does not always mean faster task completion.
- The operator confirms the predicted mission before handoff as part of the paper’s procedure. The paper does not report how often operators rejected a wrong prediction, and the real-robot experiments are not a safety validation.
- The paper says the dataset, the annotation pipeline and the VIM weights “will be released” on a project page; this site has not opened that page and has not checked whether they are public.
Links
- I See What You Want: Decoding Hierarchical Operator Intent from Visual Teleoperation History (arXiv 2611.16419)
- Full paper in HTML
- Physical AI Lab 2.3: Fine-Tuning SmolVLA — Planning Demonstrations for an Instruction, Checking What Each Number Measures, Then Timing the Asynchronous Action Queue
- Physical AI Lab 2.6: To the Edge — Working Out the Control-Rate Ceiling from a Latency Log, Telling Whether Quantization Saves Size or Time, Then Sizing an Action Chunk
- Physical AI Lab 3.4: Long-Horizon Tasks — Working Out Per-Step and Whole-Chain Success by Hand, Seeing Which Quantity Progress Memory Changes, Then Reviewing a Routing Ablation That Swaps Only the Upper Layer
- Physical AI Lab: a physical AI course where every number can be checked



