When a large language model generates fine-tuning data for task planning, a common failure is that the generated problem has no solution: the scene graph omits a key object or attribute, the task goal no longer matches the environment, the PDDL planner finds no plan, and the sample is usually thrown away. In the preprint SCOPE (arXiv 2610.31518, 2026-10-22), authors from the Institute of Automation, Chinese Academy of Sciences, and the School of Artificial Intelligence, University of Chinese Academy of Sciences, handle this differently: they treat the planner’s failure as a diagnostic signal, infer the missing precondition, add it to the scene graph and plan again until an executable plan is found. The authors report that LoRA fine-tuning Qwen2.5-7B-Instruct on 10,000 SCOPE-generated samples yields an overall success rate of 63.00% on AI2-THOR scene graphs, against 50.67% for the same amount of data generated with DELTA, a difference of 12.33 percentage points. This article focuses on how the “failure becomes repair” step works, and on what each of the two experiments can and cannot show. All numbers below are author-reported [R]; this site has not reproduced them.
The problem: why generated planning data is often “unsolvable”
The paper splits a task and motion planning (TAMP) fine-tuning sample into three parts: a scene graph describing the environment state, an instruction defining the task goal, and a plan as the solution. Manual annotation gives high quality at a cost that is hard to scale to fine-tuning needs; synthesising data directly with an LLM is cheap, and the authors see two main quality problems. The first is homogeneity: model preferences make the generated scene graphs highly repetitive, so the dataset carries too little information entropy, which limits generalisation. The second is unsolvability: hallucinations leave logical flaws in scene graphs, such as a missing interactive object or a wrong attribute, which make the task physically unsolvable and add a lot of noise to the training data. The authors note that existing pipelines built on PDDL planners are more logically consistent, but produce structurally simple, homogeneous scene graphs and still cannot guarantee solvability.
The generation pipeline: scene, instruction, plan
Scene graphs. To counter the model’s preference for high-frequency room types, SCOPE tracks the cumulative usage of each room type in the generation history, computes the empirical distribution p̂_t(r_i), and sets the next selection probability as P_t(r_i) ∝ exp(−α · p̂_t(r_i)), with α empirically set to 3.0: the more often a room type has been generated, the less likely it is to be chosen next. Within a room type, each generation samples 5–10 pieces of furniture and 20–30 objects from the asset library into the prompt, so the model does not keep reaching for the same common items. The scene graph is not output in one step; it is written as an instruction stream in a lightweight DSL, coarse to fine: AddRoom and SetAgent set the room type and the agent’s initial state, AddFurn places large furniture, and PlaceObj puts objects inside or outside the furniture. The authors say single-step output of a complete scene graph often produces structural errors such as objects without supporting furniture or inconsistent containment relations.
Instructions. Following a chain-of-thought sequence of “target sampling → action chain construction → instruction language embodiment”, the model picks interactive objects from the linearised scene graph, builds a causal action chain to a target state, and writes it as a natural-language instruction; depending on how explicitly the user’s intent is expressed, instructions fall into three levels: simple, complex and vague. In post-processing, set matching checks whether the target objects and locations in the instruction all exist among the scene entities; tasks that fail are discarded.
Plans. SCOPE wraps the robot’s low-level manipulation capabilities into 9 high-level atomic actions: place_on, place_in, open, close, power_on, power_off, fill, pour, slice. The scene graph is parsed into an initial PDDL state, the LLM decomposes the instruction step by step into an action sequence mapped to sub-goals, and, similarly to DELTA’s autoregressive approach, each sub-goal is passed in turn to a classical heuristic search planner. The text we read does not say which LLM generates the scene graphs and instructions.
The key step: turning a planning failure into a repair signal
When a sub-goal cannot be solved, SCOPE does not discard the sample. It traces back to the core action a_i of the failed sub-task and computes the state deficiency that caused the failure, σ_i = f(a_i | s_i): the static attributes or initial states needed to execute that action in the current state but missing from the scene graph. It then merges σ_i into both the current state and the scene graph (s_new = s_old ∪ σ_i, SG_new = SG_old ∪ σ_i) and hands the problem back to the planner. The function f has two layers. The first is a predefined action–precondition mapping table; the authors’ example is adding a “powerable” attribute to an object when planning power_on fails. For long-tail actions not in the table, the restricted object states and the failed action are given to the LLM, which infers the attributes and initial states to add. Once every sub-goal has a valid sub-plan, the sub-plans are concatenated into a full plan Π, which together with the repaired scene graph and the instruction forms one training sample ⟨SG_final, I, Π⟩.
One point matters for reading the results: the repair changes the scene, not the plan. For data generation this is reasonable, since the scene is synthetic anyway and making it consistent with the task is the goal. But it means SCOPE keeps “problems modified until they are solvable”, not solutions to the original problems. In the paper’s Algorithm 1, the repair loop runs “while π_i is a failure”; the text we read gives no iteration limit, no rate at which repair is triggered, no handling for a repair that never succeeds, and no check that the added attributes are commonsense-plausible.
What each of the two experiments measures
The authors built two evaluation sets, each with 100 scene graphs and 300 instructions across the simple, complex and vague levels. Evaluation first uses the PDDLGym and VAL toolchains to check whether an action sequence can be executed, then an LLM-as-a-Judge scores it from 0 to 4. There are three metrics: executability (EX) counts only the fraction of environment actions that execute successfully, regardless of task relevance; the average quality score (AQS) is the mean of the 0–4 scores; success rate (SR) counts only tasks with the full score of 4 as successful. The text we read does not name the judge model.
| Experiment | Source of scene graphs | What is compared | Overall results (SR / EX / AQS) |
|---|---|---|---|
| Table I: planning on noisy scene graphs | Generated by an LLM, with logical hallucinations and missing attributes (LLM-Synthetic Noisy Dataset) | Each method plans directly on the same noisy scene graphs and instructions; no fine-tuning involved | SCOPE 70.67% / 92.67% / 3.28; DELTA 46.33% / 67.00% / 2.25; LLM+P 41.67% / 48.00% / 1.76; LLM-As-Planner 27.00% / 29.67% / 1.84; LLMGenPlan 23.00% / 61.33% / 1.24 |
| Table II: models fine-tuned on different data | Extracted directly from AI2-THOR simulator states, free of logical flaws (AI2-THOR Grounded Dataset) | The same base model, Qwen2.5-7B-Instruct, fine-tuned with identical LoRA settings on 10,000 triplets generated by each method, then the fine-tuned models are evaluated | SCOPE (diversity + correction) 63.00% / 85.33% / 3.08; DELTA data 50.67% / 77.67% / 2.67; LLM-As-Planner data 25.00% / 26.00% / 1.57 |
In Table I, DELTA and LLM+P delegate planning to a PDDL solver, and the authors see their weakness as a complete dependence on the initial state: when the scene graph lacks a precondition, the solver returns no solution or times out. SCOPE reaches 55.00% success on complex tasks against 31.00% for DELTA, and 71.00% against 39.00% on vague tasks. With retries, LLMGenPlan’s executability rises to 61.33% while its success rate is only 23.00%; the authors explain that text-level feedback only filters out invalid action expressions and cannot fix logical errors in the scene graph.
Details worth checking in the numbers
- Table I does not compare exactly the same problem. SCOPE modifies the input scene graph while planning; the other methods must accept the original graph. The text we read does not say whether Table I’s executability check is run against the original noisy scene graph or the repaired one. In data generation, “changing the scene until it is solvable” is the point; but reading this result as “more robust to perception errors at deployment” would need other evidence: in a real scene, a missing attribute may be a perception omission or an object that genuinely lacks that capability, and adding the attribute only makes the plan executable at the symbolic level.
- 12.33 is percentage points, about 37 instructions. Both 63.00% and 50.67% use 300 instructions as the denominator, corresponding to 189 and 152 successes (this site’s conversion). The text we read reports no confidence intervals, repeat counts or random seeds.
- Ablating the two components. With conventional scenes and correction only, overall success is 58.00% and complex-task success 41.00% (28.00% for DELTA data); with diversity only and no correction it is 59.67%, complex 40.00%; with both it is 63.00%, complex 50.00%. The full configuration is not best in every column, though: its executability of 85.33% is below the 87.00% of diversity only, and its vague-task success of 51.00% is slightly below 52.00%. On simple tasks, the SCOPE configurations range from 86.00% to 88.00%, a small spread.
- An inconsistent label in Table II. The text says “DELTA’s success rate increases from 50.67% to 59.67%”, while the 59.67% row in the table is labelled “SCOPE, diversity ✓, correction ✗”. Going by the text, this row seems to be data generated with DELTA-style planning on diversified scenes, but the text we read does not say so explicitly, so readers cannot be sure how this row’s data was generated.
- Failure analysis. The authors sort the fine-tuned model’s 111 failed cases into two groups: cognitive errors, about 60%, including instruction omission (33 cases) and logic and semantic errors (34 cases); and grounding errors, including physical limitations (23 cases), e.g., violating environmental physics, and perceptual hallucinations (21 cases), e.g., interacting with non-existent objects. The four categories add up to 111, exactly the number left after 63.00% of 300 succeed (this site’s check). In other words, even on test scene graphs free of logical flaws, the fine-tuned model still had 21 perceptual-hallucination failures; the paper’s example for this category is interacting with non-existent objects.
How this relates to other content on this site
SCOPE sits in the slower part of “think” described in Physical AI Lab 0.3: See, Think, Act; it outputs high-level action sequences, not joint commands. The two most recent radar articles also deal with action preconditions, but at different points in the pipeline: Frontier Radar | SLoSa uses graded results from a formal verifier as rewards to train the planner directly; Frontier Radar | CausGround induces preconditions and effects from interaction trajectories and hands them to a planner with frozen parameters. SCOPE does not help the planner at inference time; it uses the planner’s failures to repair data, then fine-tunes a model on that data. The three differ in tasks, environments and metrics, so their numbers cannot be compared with each other. For synthetic data as one source of robot data, see Physical AI Lab 1.5: Where Does Data Come From?; for the gap between EX and SR, the variability of an evaluation on 300 instructions with no reported repeats, and how one failure label can hide several kinds of failure, see the methods in Physical AI Lab 1.8: Evaluation.
Limitations
- This is an arXiv preprint; all numbers are author-reported [R], and this site has not reproduced them. The text we read has no confidence intervals, repeat counts or random seeds.
- Both evaluation sets were built by the authors, with 300 instructions each; SR and AQS depend on scores from an LLM judge, and the text we read neither names the judge model nor reports its agreement with human judgement.
- All experiments are in symbolic planning and the AI2-THOR simulator; Figure 3 shows qualitative snapshots of execution in the simulator. There are no real-robot experiments, and this is not a hardware safety validation.
- The text we read gives no iteration limit for the repair loop, no trigger rate, no handling for failed repairs, and no plausibility check on the added attributes; nor does it name the LLM used to generate scene graphs and instructions.
- The label of the 59.67% row in Table II is inconsistent with the text, as described above.
- The paper gives a project page address, scope-page.github.io, saying more demos and information are available there; the text we read does not say whether code or the dataset have been released. This site did not open that page to check its contents while writing.
- “About 37 instructions”, “189 and 152” and “the four categories add up to 111” are this site’s conversions and checks using the 300-instruction denominator.
Links
- SCOPE: Scene-graph Construction Optimized by Plan Error-correction for Robot Long-term Planning Datasets (arXiv 2610.31518)
- Full paper in HTML
- Physical AI Lab 0.3: See, Think, Act — Putting Each Rate Back in Its Own System
- Physical AI Lab 1.5: Where Does Data Come From? Working Out Demonstration Time from SO-101 Metadata, Then Separating the Units and Action Spaces of Three Sources
- Physical AI Lab 1.8: Evaluation — Splitting a “Net Gain of 2” into a Paired Table, Then Checking the Failure Labels and Random Seeds
- Frontier Radar | SLoSa: a formal verifier sorts plans into five outcomes in a fixed order with graded reward intervals, and 4B–14B task planners run locally in INT4 on a Jetson Orin
- Frontier Radar | CausGround: inducing each action’s preconditions and effects from interaction trajectories, so a frozen multimodal planner adds the “open the door before putting it in” step without parameter updates
- Physical AI Lab: a physical AI course where every number can be checked (course outline)
