The same subtask VLA succeeds on 80% of RoboCasa365’s atomic tasks, yet once those skills are chained into long-horizon tasks that require moving the base, the agentic baseline succeeds on only 19%. The preprint Cracking Mobile Manipulation (arXiv 2610.19600, 2026-10-14) does not train a stronger policy. It first attributes each of 81 failed episodes to “the earliest error that was never corrected and kept the task from completing,” then adds a targeted intervention group at each of the three stages where failures concentrate (four interventions in all), leaving the underlying VLA unchanged throughout. The authors report that success on the development episodes rises from 19.0% to 47.0%, and, with the design frozen, from 18.0% to 49.0% on held-out seeds; from 14.0% to 37.0% on five BEHAVIOR-1K tasks; and from 5/20 to 13/20 on two real-robot tasks. What repays a close reading is the attribution rule and the experiment that adds interventions one at a time: the latter shows how, once one stage is fixed, failures move to the next. All numbers below are author-reported [R]; this site has not reproduced them.
Between 80% and 19%: how a failed episode is attributed
The evaluation covers the 20 of RoboCasa365’s 32 composite tasks that require mobile manipulation. Each contains 2–15 subtasks and involves moving between workspaces for object transport, pick-and-place and interaction with articulated objects. The baseline adapts the hierarchical design of Harness VLA: a high-level planner selects subtask instructions or tools, and one shared subtask VLA handles both navigation and manipulation. That VLA is trained on atomic-task demonstrations and subtask-level segments from the RoboCasa365 training split.
Each task is run five times, giving 100 episodes. The baseline succeeds in 19, leaving 81 failed episodes for analysis. The attribution rule has three parts:
- Review the recorded actions, observations and simulator state to find the failed subtasks.
- If the affected subtask is completed within the next three replanning attempts, the error counts as recovered and is excluded from the primary attribution.
- Otherwise, find in the remaining trajectory the earliest error that stays unresolved and prevents task completion, and mark its time step as t*. The paper’s example: if the robot fails to grasp an object and never picks it up, the failure is attributed to grasping, not to a later placement attempt; only if the grasp is later recovered and placement then fails is it attributed to placement.
Offline diagnosis uses the rollout video, the planner’s interaction history, robot states and simulator object states. Simulator states serve only for diagnosis and are not available to the online agent. Candidate labels come from three independent Codex diagnoses, which human reviewers then check against the rules above. On a random subset of 30 failed episodes, three reviewers agreed unanimously on 26, with Fleiss’ κ = 0.86.
According to the paper’s Figure 2, the primary failures in the 81 failed episodes are attributed to task planning (21.0%), navigation and base positioning (25.9%) and local manipulation (53.1%). In episodes that is 17, 21 and 43, totalling 81 (this site’s conversion). The authors describe the three regimes as follows:
- Task planning: the necessary low-level skills are available, but the planner invokes them for the wrong target or at the wrong stage; once early actions change the scene, later decisions may no longer hold, because task progress is not tracked reliably.
- Navigation and base positioning: the robot reaches the correct area but stops at a base pose from which the next manipulation is difficult or infeasible. The authors argue that navigation should be judged not only by whether the robot reaches the target area, but also by whether the pose it arrives at affords the upcoming interaction.
- Local manipulation: planning and navigation succeed, but the final interaction still fails because of target localization errors, grasp misalignment or inaccurate end-effector positioning.
The authors stress that these failures are sequential in execution: planning errors can keep the robot from the right workspace, a poor base pose can keep a correct subtask from being executed, and small geometric errors can still derail the final interaction.
Four interventions, each matched to a failure stage
The subtask VLA stays fixed, and all agentic methods use Kimi K3 as the high-level agent. The four interventions:
| Failure stage | Intervention | How it works | Boundary stated in the paper |
|---|---|---|---|
| Task planning | Skill library (from demonstration videos) | Gemini 3.5 Flash matches the task instruction to annotations in the corresponding benchmark’s training data, retrieves a demonstration and summarizes it as a “skill card”: task goal, subtask order, representative keyframes and completion criteria. At each step the planner conditions on both the skill card and the current execution context (current observation and action history) | The card gives task-level guidance while the planner still adapts its actions to the current scene |
| Task planning | Adaptive visual memory | Stores the current episode’s observations; when the planner issues a query, Qwen3.8-27B ranks the stored observations against it and returns the 16 most relevant frames with a short trajectory summary, for example “Where was the basket last seen?” | Only past observations are eligible; no future frames or simulator states |
| Navigation and base positioning | Accessibility check and base adjustment | Before switching from navigation to manipulation, the agent calls check_stance: the target’s name is a text query for Grounding DINO, SAM refines the mask, and valid depth pixels in the mask are back-projected with calibrated camera parameters into the robot frame; by target distance the stance is classed as reachable, marginal or out of reach. If marginal, approach_xy applies a small local base correction with the heading fixed; if out of reach, goto_stance selects a new stance within manipulation range and plans a path with A* on a 2D occupancy grid built from the current depth observations, inflating obstacles by 0.1 m as a safety margin, and a PD controller follows the waypoints | Distance thresholds are estimated from successful training trajectories and fixed before evaluation; the check considers only target distance and does not validate joint limits, inverse kinematics or full-body collisions |
| Local manipulation | End-effector adjustment | For each grasping or placement subtask, the same Grounding DINO and SAM pipeline yields a target mask whose valid depth pixels are back-projected into 3D; the median of these points is the target position, reducing sensitivity to depth outliers and missing depth; a local surface normal estimated from the points sets the end-effector direction, and a bounded correction moves toward an offset pose along it | Translation is limited to 0.3 m per call; only the local pose changes, not the subtask VLA |
Adding interventions one at a time: how failure moves to the next stage
During development, every configuration is tested on the same episode seeds and simulator initial states, so each added component can be compared episode by episode with the previous configuration. In the order of the paper’s Figure 4, the skill library, adaptive visual memory, base adjustment and end-effector adjustment are added in turn. The text gives: 19.0% for the baseline, 29.0% just before base adjustment, 40.0% after it and 47.0% with everything added. So the two planning components together add 10 percentage points, base adjustment 11 and end-effector adjustment 7 (this site’s arithmetic; the text gives no figure for the skill library alone). The authors note that base adjustment is the largest increment in this sequence.
The paper’s Figure 5 counts each failure type over the same 100 matched episodes. As the authors describe it, the earlier planning interventions mainly reduce failures in target selection and task progression, while base adjustment substantially reduces navigation and base-positioning failures; once the planning and navigation bottlenecks ease, more episodes reach later stages and manipulation becomes the dominant remaining failure type. The authors call this a “moving failure frontier”: improving one stage exposes the next bottleneck in the pipeline. In the fine-grained categories, the failure modes the interventions target directly fall sharply:
- Placement target or receptacle assignment errors: 9 → 0
- Ineffective approaches or poor base positioning: 11 → 1
- Grasp misalignment or empty grasps: 10 → 3
The remaining failures concentrate in incorrect object placement, objects dropped during transport and collision-induced scene disruption. These errors occur late in execution and are not fully addressed by the current geometric corrections.
One point deserves attention: at baseline, local manipulation accounts for the largest share of failures, 53.1%, yet the largest single increment comes from base adjustment. This site’s reading: each failed episode is counted once, at its “earliest unrecovered error,” so the share tells you where failure first occurs, not which stage will pay off most when fixed; once an earlier stage is fixed, some episodes go on to fail at a later one. The paper’s “failure frontier” description is consistent with this reading, but the text does not give the paired numbers of episodes that improved or worsened at each step.
Held-out seeds, BEHAVIOR-1K and the real robot
After fixing the system design, prompts and tool parameters, the authors ran a separate evaluation on five held-out seeds per task: the same task definitions, with different scene layouts and initial object states. All methods in Table 1 use the same episode seeds. Agentic methods are additionally matched on planner budget and retry budget, and Pigey and the authors’ Harness VLA adaptation use the same subtask VLA checkpoint as the proposed method; end-to-end baselines use the released checkpoints and implementations linked from the official RoboCasa365 repository.
| Type | Method | Development (%) | Held-out seeds (%) |
|---|---|---|---|
| VLA | π0 | 2.0 | 2.0 |
| VLA | π0.5 | 3.0 | 4.0 |
| VLA | GR00T N1.6 | 7.0 | 8.0 |
| VLA | GR00T N1.5 | 8.0 | 9.0 |
| VLA | RLDX-1 | 15.0 | 14.0 |
| VLA | Xiaomi Robotics 1 | 44.0 | 45.0 |
| Agentic | CaP-X (Kimi K3) | 10.0 | 11.0 |
| Agentic | Pigey (Kimi K3) | 13.0 | 12.0 |
| Agentic | Harness VLA (Kimi K3) | 19.0 | 18.0 |
| Agentic | Proposed method (Kimi K3) | 47.0 | 49.0 |
The authors report that the held-out improvement is statistically significant under paired analysis (p < 0.001, Appendix E; the reference list shows Appendix E cites McNemar’s test and the bootstrap), and conclude that the gains are not specific to the episodes used during development. Their wording for Xiaomi Robotics 1 is “comparable” performance: 47.0 vs 44.0 and 49.0 vs 45.0; the text this site read gives no significance test between the two.
There are two transfer experiments. On five BEHAVIOR-1K tasks, the low-level policies are trained on BEHAVIOR-1K training data while the intervention design stays the same, and success rises from 14.0% for the corresponding Harness VLA baseline to 37.0%. On two real-robot tasks with a dual-arm mobile manipulator, the full system succeeds in 13 of 20 trials and the agentic baseline in 5. Task definitions and detailed results are in Appendices I and J, which the text this site read does not include.
Ablations: which design choices matter
Each ablation changes one design choice and keeps the rest of the full system unchanged; all are evaluated on the same 100 matched episodes (%):
| Component | Variant | Success rate |
|---|---|---|
| Reference | Full system | 47.0 |
| Skill library | Text-only skill card (VLA training data) | 43.0 |
| Skill library | Text skill card + keyframes (HIW-500) | 40.0 |
| Skill library | Text skill card + keyframes (VLA training data) | 47.0 |
| Visual memory | Uniform history every turn, 8 / 16 / 32 frames | 37.0 / 40.0 / 35.0 |
| Visual memory | Planner-triggered query, 8 / 16 / 32 frames | 42.0 / 47.0 / 45.0 |
| Base adjustment | Without accessibility check check_stance | 36.0 |
| Base adjustment | Without local base correction approach_xy | 42.0 |
| Base adjustment | Without stance replanning goto_stance | 39.0 |
| End-effector adjustment | Target location from the agentic planner | 43.0 |
| End-effector adjustment | Target location from geometric skills | 47.0 |
The authors’ reading: HIW-500 consists of real-world household demonstrations teleoperated with humanoid robots; replacing them with in-domain demonstrations from the RoboCasa365 training split raises success from 40.0% to 47.0%, and dropping the keyframes from the in-domain cards, leaving text only, lowers it to 43.0%, suggesting that both domain alignment and visual context contribute. Planner-triggered queries score higher than uniform history at every frame budget, while more frames do not consistently help; the authors take this to mean that visual memory’s value lies more in retrieving task-relevant evidence. Removing the accessibility check causes the largest drop, and removing stance replanning hurts more than removing the local correction. The end-effector pair differs by 4 points, which the authors themselves call modest.
When reading this table, keep in mind that each percentage point here is 1 of 100 episodes, so 4 points is a net difference of 4 episodes. The paired statistics for the ablations are in Appendix K.1, which the text this site read does not include, so it is not known how many episodes improved or worsened within these differences, or which of them are statistically reliable.
How this relates to existing pages on this site
Physical AI Lab 1.8: Evaluation showed that a single “failure” label hides different ways of failing and that a net gain lets improvements and regressions cancel out. This paper gives an explicit set of rules for failure labels: a recovery window of three replanning attempts, counting only the earliest unrecovered error, and model-proposed candidate labels checked by people, with inter-reviewer agreement reported. Its development-stage evaluation also follows the paired design taught in 1.8: every configuration runs on the same seeds and initial states.
Physical AI Lab 1.1: Coordinate Frames and Kinematics used a workspace ring to decide whether an arm can reach a point, and noted that it computes position only, without joint limits. check_stance likewise sorts a stance into three classes by target distance, and the paper states explicitly that it does not validate joint limits, inverse kinematics or full-body collisions; “reachable” here is a distance class and does not guarantee an inverse-kinematics solution.
Frontier Radar | RoboAware also builds an agentic layer around fixed executors. The approaches differ: RoboAware trains a coordinator from same-state counterfactual branches in simulation, whereas this paper designs its interventions by hand from human-checked failure attribution. The two papers use different benchmarks, tasks and protocols, so their success rates cannot be compared.
Limitations
- This is an arXiv preprint; all numbers are author-reported [R], and this site has reproduced none of them. The authors’ reproducibility statement says code, prompts, configuration files and evaluation episode lists will be released; this site has not checked whether they are public yet, and this article does not link a code repository.
- Candidate failure labels come from Codex and are then checked by people; agreement is reported only on 30 randomly sampled failed episodes (26 unanimous, κ = 0.86). The paper states that generative AI tools were also used for literature collection, parts of the manuscript and parts of the experimental code.
- The development episodes were used both to design the system and to report the step-by-step 19.0% → 47.0% results; the result independent of the design process is 18.0% → 49.0% on held-out seeds. Both cover only RoboCasa365’s 20 mobile manipulation tasks, with five episodes per task.
- BEHAVIOR-1K covers only five tasks, and the real-robot evaluation only two tasks with 20 trials in total; task definitions, the allocation of trials per task and the experimental conditions are in Appendices I and J, which the text this site read does not include. Simulation results are not hardware safety validation.
check_stanceconsiders only target distance and does not validate joint limits, inverse kinematics or full-body collisions; the 0.1 m obstacle inflation in stance replanning is a margin set in the paper, not a safety guarantee. By the authors’ own analysis, late-stage failures such as drops during transport and collision-induced scene disruption are still not fully addressed by the current geometric corrections.- The system relies on several external models: Kimi K3 as the high-level agent, Gemini 3.5 Flash for skill cards, Qwen3.8-27B for ranking visual memory, and Grounding DINO with SAM for target segmentation; the text this site read gives no latency or cost for these calls. The paired statistics for the ablations and the held-out evaluation are in Appendices E and K.1, which the text this site read does not include.
Links
- Cracking Mobile Manipulation: From Failure Diagnosis to Agentic Robot System (arXiv 2610.19600)
- Full paper in HTML
- Physical AI Lab 1.8: Evaluation — Splitting a “Net Gain of 2” into a Paired Table, Then Checking the Failure Labels and Random Seeds
- Physical AI Lab 1.1: Coordinate Frames and Kinematics — Working Out by Hand How a Two-Link Arm Reaches the Cup
- Frontier Radar | RoboAware: learning from same-state counterfactual branches whether modular composition or an end-to-end policy should handle each skill stage
- Physical AI Lab: a physical AI course where every number can be checked (course outline)
