Can a general-purpose multimodal model control a robot arm directly, with a handful of tools and no trained action policy? In the preprint Large Multimodal Models for Direct Robot Control: GPT-6 as a Beginning (arXiv 2610.28259, 2026-10-20), authors from Nanjing University, Nanyang Technological University, Peking University, the Institute of Automation of the Chinese Academy of Sciences and other institutions use their own tool framework, GPT-GEAR, to have GPT-6 Astra run closed-loop manipulation, over more than 2,000 trials in simulation and the real world. The authors report a 76.0% success rate on RoboTwin 2.0 and 26.92% on RoboDojo, but only 11.1% on their new geometry-constrained benchmark LIBERO-GeoMotion and 40% overall in zero-shot real-robot runs. For engineers, the more useful part is three ablations: the form in which depth, touch and demonstrations reach the model changes the result considerably. All numbers below are author-reported [R]; this site has not reproduced them.
GPT-GEAR: five tools and an action interface of relative poses only
The paper frames manipulation as a partially observable decision problem. On each turn the model reads RGB images and proprioceptive state, plus whatever the experimental condition adds (camera parameters, depth maps or tactile signals), then picks a tool and supplies its arguments. The five tools are:
- Observe: reads the current observation record or a past one. Reading history is read-only: it does not recapture images, advance the simulator or consume control steps.
- Reference: before the first action, supplies a prevalidated demonstration package, which can be video keyframes, textual steps, action trajectories or a combination.
- Act: the only tool that changes the physical state. For each arm it takes a relative translation Δp (in meters), a rotation increment Δq as a unit quaternion, a gripper target g ∈ [0, 1] and a limit on solver steps.
- Status: queries the environment’s task-completion predicate.
- Finish: ends the interaction.
The implementation details are specific. GPT-6 Astra is the only planning component, with no other robot-specific model; all experiments run through Codex with reasoning effort set to high. Translation and rotation increments are limited to 0.01 m and 0.05 rad per step, a single Act request is bounded by 0.20 m of displacement and π/2 rad of rotation, and by default each Act call allows up to 10 native control steps. Assuming the 0.01 m limit applies to the length of each step’s translation vector (the paper says “norm” only for the 0.20 m request bound, so the per-step limit could also apply to each axis), these numbers imply (this site’s calculation) that one Act call can move at most 0.10 m under the default setting. The paper states that motion completion and residual pose errors are returned to the model as feedback, and the model decides how to correct on the next call. The controller checks kinematic and joint constraints and rejects the step for both arms if either command is invalid; the paper states that these checks do not guarantee collision-free motion or successful grasping, and task-dependent motion planning remains the model’s responsibility.
One evaluation detail needs attention. In simulation, success is decided at termination by the environment’s official predicate, independently of whether the model claims to be done; Status queries that same predicate. When the model judges the task complete it calls Status, and if the check fails, execution continues until the budget runs out. The text this site read does not give the size of that budget, nor does it say what determines completion in the real-robot experiments.
Three ablations: how information is given matters more than whether it is given
Each ablation uses a small set of tasks with a different sample size; the table lists only the key rows. Total tokens are the average input plus output tokens per episode.
| Ablation (tasks × trials per task) | Condition | Success rate | Avg. action rounds | Avg. total tokens |
|---|---|---|---|---|
| Depth (LIBERO-Plus, 4 × 10) | RGB only | 47.5% | 34.61 | 1,046,755 |
| RGB + camera parameters | 60.0% | 26.32 | 986,066 | |
| RGB + dense depth map rendered from ground truth | 67.5% | 24.23 | 1,101,271 | |
| RGB + back-projection point-query tool | 60.0% | 27.00 | 1,377,672 | |
| Demonstrations (LIBERO-Plus, 5 × 10) | No demonstration | 64.0% | 23.00 | 1,187,814 |
| First and last frames only | 46.0% | 21.18 | 1,448,025 | |
| Video sequence frames | 64.0% | 20.72 | 1,442,084 | |
| Textual steps | 78.0% | 21.30 | 1,052,921 | |
| Action trajectory + video sequence frames | 82.0% | 19.92 | 1,222,021 | |
| Action trajectory + textual steps | 82.0% | 21.00 | 1,191,807 | |
| Action trajectory + video frames + textual steps | 86.0% | 20.58 | 1,372,547 | |
| Touch (UniVTAC, 5 × 20) | RGB only | 14.0% | 10.57 | 423.90k |
| RGB + tactile images | 30.0% | 10.53 | 646.06k | |
| RGB + three-axis force values as text | 50.0% | 9.44 | 451.19k | |
| RGB + tactile images + force text | 48.0% | 9.75 | 553.36k |
Depth. Adding only camera intrinsics and extrinsics raises success from 47.5% to 60.0%; the authors attribute this to camera parameters helping the model relate image positions to the robot workspace. Requiring the model to estimate the depth of target object centers before acting gives 62.5%; in three examples, the mean absolute errors against ground truth are 1.91, 3.81 and 3.40 cm. Accurate point coordinates do worse than a dense depth map: each back-projection query returns the 3D coordinates of only one point, so the model must still choose points and relate them to objects and free space, and it uses 31.6% more tokens than RGB alone and 25.1% more than the depth-map condition. The authors also note that some tasks still fail even with ground-truth depth, so depth is not the only bottleneck.
Demonstrations. Giving only the first and last frames drops success to 46.0%, below no demonstration at all; the authors suggest that sparse images introduce ambiguity when the demonstrated scene differs from the current one. Textual steps mainly help task understanding, while action trajectories add motion-level information: adding a trajectory to any of the visual or textual contexts raises success. The authors stress that demonstrated actions are not replayed verbatim but adapted to the current scene; the full combination’s 86.0% costs about 180,000 more tokens than trajectory plus text at 82.0%. All demonstrations in this ablation are expert trajectories for the same task under different scene perturbations.
Touch. Readings from the same sensor work better as text force values (x, y and z components) than as tactile images: 50.0% versus 30.0% success, with fewer tokens. The authors’ explanation is that tactile images encode taxel responses rather than scene appearance, which makes them harder for the model to interpret; giving both together (48.0%) brings no further benefit. Even in the best condition, half of the episodes fail.
When reading these gaps, convert them back into trial counts: each depth condition has 40 trials, so one trial is 2.5 percentage points and 60.0% versus 62.5% is a single trial; each demonstration condition has 50 trials, so one trial is 2 points; each touch condition has 100 trials. The text this site read gives no confidence intervals or significance tests. Physical AI Lab 1.8: Evaluation practices judging whether a gap holds up when trials are limited; for what depth cameras and tactile sensors actually output, see Physical AI Lab 1.3: Sensors.
Five benchmarks: ahead in several, clearly behind in one
The main experiments combine the capabilities: only RGB and camera parameters are given, with depth estimation of targets required before acting; demonstrations combine sampled video frames, textual steps and action trajectories; on UniVTAC, tactile feedback and ground-truth depth maps are also provided. The five benchmarks contain 164 tasks. Because closed-loop evaluation is costly, the authors did not run the full official protocols: 20 trials per task on RoboCasa GR1 and UniVTAC, 10 on the other three. Some comparison methods were re-evaluated by the authors under this protocol; the rest use officially reported values.
- LIBERO-Plus: 83.50%, close to the 83.90% officially reported for π0.5; the authors’ re-evaluated starVLA-OFT and starVLA-π reach 84.75% and 85.50%.
- RoboTwin 2.0: 76.00%, 9.20 percentage points above the re-evaluated second place, OpenWAM-α.
- UniVTAC: 72.50%, above the reported FTP-1 (62.90%) and starVLA (56.10%); note that the model received touch and ground-truth depth here as well.
- RoboDojo: 26.92% success and a process score of 35.17, both the highest in the comparison and above the reported DM0.5 (19.34%, 24.90); the authors also say both absolute values remain low.
- RoboCasa GR1: 18.96%, above the reported π0 (13.60%), below the reported π0.5 (37.00%), and well below the re-evaluated starVLA-OFT (57.29%) and reported OpenWAM-α (60.50%). This is the only benchmark with dexterous hands; the authors hypothesize that the higher degrees of freedom and multiple contact relationships contribute, while noting that aggregate results cannot establish the cause.
The paper calls these zero-shot results, meaning no benchmark-specific policy training and no parameter updates; the model still receives demonstrations in context. The comparison methods were all trained on their own robot data. Leading on several benchmarks shows that a general model can already match trained policies on some of them; it does not show that it is stronger under equal conditions.
Broken down by capability: it understands, but does not execute precisely
The authors label the 164 tasks with six capability dimensions according to task requirements (not execution outcomes); a task can carry several labels. Aggregate success rates: task understanding 73.45% (426/580), generalization 59.20% (817/1380), memory 56.67% (34/60); long-horizon execution 33.03% (218/660), high-order operation (orientation changes, rotation, pouring, motion around articulated joints) 34.62% (135/390), precision 36.19% (380/1050). Memory covers only 6 tasks, all from RoboDojo. The authors themselves state that these comparisons reflect differences in task requirements and difficulty rather than isolated model abilities; their conclusion is a recurring gap between knowing what to do and physically getting it done.
To probe that gap directly, the authors introduce LIBERO-GeoMotion: built on LIBERO assets, it minimizes semantic complexity and requires precise pose alignment, coordinated translation and rotation, and anticipation of an action’s physical consequences. GPT-6 Astra’s zero-shot success rate is 11.1%. On real hardware, the authors report that the model reliably completes basic pick-and-place, but its overall zero-shot success rate is only 40%, with some constrained tasks failing entirely; persistent failures involve unsafe grasp configurations, motion feasibility and precise alignment, and substantial inference latency further hinders practical deployment.
Compared with the VLA work this site has covered before, this paper starts from a different point: it trains no policy and instead measures how far a general model gets on reasoning and feedback alone. Is Success All You Need? showed on the same LIBERO-Plus perturbation suite that policies with similar success rates can differ greatly in execution quality; this paper’s action rounds and token counts are another record beyond success rate. How demonstrations shape a policy is covered from the training side in Physical AI Lab 1.6: Imitation Learning; here, demonstrations go only into the context, and no parameters change.
Limitations
- This is an arXiv preprint; all numbers are author-reported [R], and this site has not reproduced them. The text this site read ends at Section 5.2.3 (memory); it did not include the body sections on LIBERO-GeoMotion’s task design, the real-robot experiments or latency data. Those parts rest only on statements in the abstract and introduction, so this article gives no task count for LIBERO-GeoMotion, no real-robot task count and no latency figures.
- Benchmark evaluation uses reduced trial counts, not the full official protocols; the comparison figure mixes the authors’ re-evaluated values with officially reported ones, and input conditions differ across methods (on UniVTAC, for example, this model also had touch and ground-truth depth).
- “Zero-shot” means no training and no parameter updates; the main experiments still provide demonstration context, and the demonstrations in the ablation are expert trajectories for the same task. In simulation, Status queries the environment’s official predicate, and the model can keep trying within the budget.
- Average tokens per episode across the conditions in the table range from about 420,000 to 1.45 million; these are records of API usage, not measurements of latency or cost.
- The paper lists a project page and a code repository; this site did not open them and does not confirm whether the code is available.
- The controller’s constraint checks do not guarantee collision-free motion; simulated success rates and small-scale real-robot results are not hardware safety validation, and the authors themselves list physical safety as an open problem.
Links
- Large Multimodal Models for Direct Robot Control: GPT-6 as a Beginning (arXiv 2610.28259)
- Full paper (HTML)
- Frontier Radar | Is Success All You Need?: two VLAs with similar success rates can succeed in very different ways
- Physical AI Lab 1.3: Sensors — Working Out Pixels, Depth Range and Grip Force for One Cup
- Physical AI Lab 1.6: Imitation Learning — Working Out How an Error Rolls Out of the Demonstrations’ Coverage, Then Putting the ACT, Diffusion Policy and Data Scaling Numbers Back in Their Own Conditions
- Physical AI Lab 1.8: Evaluation — Splitting a “Net Gain of 2” into a Paired Table, Then Checking the Failure Labels and Random Seeds
- Physical AI Lab: a physical AI course where every number can be checked (course outline)
