Frontier Radar | MoMaSplat: reconstruct the scene and the manipulated object separately, re-render whole mobile-manipulation demos from base poses never demonstrated; authors report 37% average success at unseen poses (strongest baseline 17%)

MoMaSplat reconstructs scene and moved object separately to re-render full demos from new base poses; authors report 37% unseen-pose success vs 17%.

Left-to-right timeline diagram: the title “MoMaSplat: source demos to new base poses” at the top and a source line at the bottom. Five numbered circular nodes are arranged in sequence and joined by arrows, each with text below it. Step 1: a static-scene 3DGS from navigation frames, with depth setting the metric scale; step 2: mask the object, estimate its poses with FastVGGT, align with Sim(3) and fit an object 3DGS; step 3: sample base poses within 1.2 × 1.2 m of the original base and resample if IK fails; step 4: render the scene, object and robot separately and composite them by depth per pixel; step 5: retarget actions to the new base frame and co-train Diffusion Policy with source data at 0.25 and synthetic data at 0.75.
The MoMaSplat pipeline that turns source demonstrations into demonstrations from new base poses, drawn from Sections III and IV-A of arXiv 2611.08372 (authors' description). The alignment in step 2 relies on the object being static during navigation, so FastVGGT and SfM pose the same cameras; for articulated objects only the moving rigid part is treated as the object. Generated demonstrations replay the source object trajectory and do not model contact forces or dynamics. Arrows show only the order of the pipeline; numbers, circle sizes and colours indicate no ranking or measurement. 1.2 × 1.2 m and 0.25 / 0.75 are experimental settings from the paper, not performance results.

The preprint MoMaSplat (arXiv 2611.08372, 2026-11-05, RobIn Lab, The University of Texas at Austin) addresses a practical problem in mobile manipulation: when a robot navigates to a workstation, its base rarely stops at the exact pose used to collect the demonstrations, and visuomotor policies trained by imitation learning are sensitive to camera viewpoint. Existing novel-view synthesis (NVS) augmentation treats the scene as static, so it cannot correctly re-render an object once it has been picked up or opened. MoMaSplat reconstructs the static scene and the manipulated object separately, re-renders whole demonstrations from sampled new base poses, including the post-contact phase, and converts the actions into the new base frame. The authors report an average success rate of 37% at unseen base poses across five real-robot tasks, against 17% for the strongest baseline. All numbers below are author-reported [R]; this site has not reproduced them.

The problem: a shifted base means a shifted view

The paper attributes base-pose deviations to navigation uncertainty, localization drift, obstacle avoidance and imperfect planning. The camera is mounted on the robot’s head or torso, so when the base moves, the policy sees a different image from the one in its demonstrations. The direct fix is to collect demonstrations from many locations, but the authors note that real-robot teleoperation is expensive and hard to scale.

Generating new-view demonstrations with NVS is the other route. None of the methods the paper lists, VISTA, the 3D Gaussian Splatting method of Yang et al., One-Thousand Trajectories and EgoDemoGen, explicitly reconstructs the object as it moves. Because the object moves during the demonstration, these methods can only augment the pre-contact approach, or they produce geometrically inconsistent images. The authors regard the post-contact interaction phase as the most critical part of a demonstration.

The paper’s starting point is that standard mobile-manipulation data collection already provides the multi-view observations needed for reconstruction. Navigation trajectories see the static scene and objects from different angles; the manipulation demonstrations themselves provide object-centric observations and the object’s motion.

The method: scene, object and robot rendered as three streams

During collection, the robot first navigates from different starting points to N target base poses, then records M manipulation demonstrations at each pose with the base fixed. Actions are 7-D commands: end-effector translation and rotation deltas in the base frame, plus a gripper command. MoMaSplat uses these two kinds of data in the following steps:

  1. Static scene. COLMAP is run on the navigation frames to obtain camera poses and a sparse point cloud, which initializes a 3DGS model; the uncertainty-aware formulation of WildGaussians down-weights regions that are inconsistent across frames. The manipulated object and the robot are masked out during reconstruction so the object is not fixed into the scene, which would leave a static “ghost” when the object is composited in later. Structure-from-motion (SfM) leaves the scale unknown, but action retargeting needs metres, so the onboard depth camera sets the scale: measured and rendered depth are compared, taking the median of the per-pixel ratios (the appendix specifies a median of the per-frame medians).
  2. Manipulated object. Grounded-SAM and SAM 2 segment the object; with the background and robot masked out, the frames go into the feed-forward 3D foundation model FastVGGT, which gives a per-frame camera pose relative to the object. These poses are in the model’s own frame, with arbitrary scale, orientation and origin. Navigation frames are the key to alignment: the object is static during navigation, so the same cameras are posed both by FastVGGT and by SfM, and the two trajectories differ only by a Sim(3) transform, solved in closed form with the Umeyama algorithm. In the experiments, alignment uses 50 frames from one navigation demonstration. The object poses and the object’s 3DGS are then optimized jointly. For an articulated object, only the moving rigid part (a laptop lid, an air fryer’s basket) is treated as the object and the rest stays in the scene; articulation parameters are not estimated, and motion is not extrapolated beyond the demonstration.
  3. Sampling base poses. Planar positions are sampled uniformly within a region around each original base pose (1.2 × 1.2 m in the experiments), and the heading is set to centre the object in the camera’s field of view, then perturbed with uniform noise. With the end-effector pose relative to the object held fixed, inverse kinematics (IK) is checked at several probe timesteps across the demonstration; a pose without a solution is rejected and resampled.
  4. Rendering and compositing. A virtual camera is placed for the new base pose, with the real camera’s intrinsics. The scene and object are rendered with 3DGS; the robot’s joints are solved with IK and the robot is rendered in OmniGibson. The three images are sorted by depth at each pixel and alpha-composited front to back, so nearer surfaces occlude farther ones.
  5. Retargeting actions and proprioception. The end-effector pose is multiplied by the transform from the original base frame to the new one; translation deltas are rotated by that transform’s rotation part; for rotation deltas, the axis is rotated and the angle kept; the gripper command is unchanged.
  6. Training. K synthetic demonstrations are generated per source demonstration, and Diffusion Policy is trained on the union of the two, sampling source data at 0.25 and synthetic data at 0.75.

Step 5 can be understood with a simplified example (this site’s illustration, not data from the paper): suppose the new base is at the same position as the original but turned 90° to the left, with the base frame’s x axis pointing forward and y axis pointing left. A translation delta of “0.10 m forward” in the original base frame produces the same displacement in the world, but in the new base frame it is “0.10 m to the right” (y = −0.10 m). If only the image were changed and not the action, the policy would learn an action label that contradicts the image. Not converting actions is one of the paper’s explanations for VISTA’s poor results.

Setup: five real-robot tasks, 3 trials per unseen pose

Five tasks in three scenes: Pick-Place Bottle and Open Drawer in the desk scene, Pick-Place Remote and Close Laptop in the living room, and Open Air-fryer in the kitchen, covering rigid pick-and-place and single-degree-of-freedom articulated objects. Close Laptop has 75 demonstrations and the other tasks 90 each, spread over three base poses; 9 navigation trajectories were also collected. Each method generates 10 synthetic demonstrations per source demonstration, 750–900 per task (400 for One-Thousand Trajectories on Pick-Place Bottle; see below).

For evaluation, each task is run 3 times at each of 12 unseen base poses, which is 36 trials per task and 180 across the five tasks (this site’s calculation); the unseen poses are 5–50 cm from the original poses. Each of the three original poses is run 15 times.

All four baselines were implemented or configured by the authors and use the same source demonstrations:

  • Source Only: Diffusion Policy with no augmentation.
  • Action Augmentation: zero-mean Gaussian noise on proprioception and actions, images unchanged.
  • VISTA: new views generated frame by frame with ZeroNVS, actions unchanged; planar perturbations within ±10 cm per axis, beyond which the authors say image artifacts become large and imitation learning results get worse. The frame-by-frame images are not temporally consistent.
  • One-Thousand Trajectories: the static scene is reconstructed with the same pipeline as MoMaSplat but without masking out the object; the authors added MoMaSplat’s action retargeting and left out the original method’s additional trajectory augmentation. On Pick-Place Bottle, only 40 demonstrations have an initial object pose matching the reconstructed scene, so only they are used to generate 400; all 90 source demonstrations are still used for training.

Results: 37% on average at unseen poses, with the widest gap at large distances

MethodSuccess at unseen base poses (average of five tasks)
MoMaSplat37%
One-Thousand Trajectories17%
VISTA8%
Source Only6%
Action Augmentation6%

At the original base poses, all methods perform comparably. At unseen poses, Source Only and Action Augmentation fall close to zero beyond 10 cm on most tasks, which the authors take to mean that perturbing actions without changing observations gives little robustness to base shifts. One-Thousand Trajectories degrades more gradually but still falls off at large distances, which the authors explain by its augmenting only the free-space approach. The authors report that MoMaSplat has the highest overall unseen-pose success on every task and is at least as good as every baseline in every distance bin; for example, 33% on Pick-Place Bottle at 30 cm and 50% on Close Laptop at 40 cm, where every baseline scores 0. The text this site read and the captions of Figs. 4–5 give neither the number of poses per distance bin nor confidence intervals.

The effect of the number of source demonstrations is tested only on Pick-Place Bottle: with 10, 50 and 90 source demonstrations, each augmented tenfold, success at 4 test locations 10–20 cm from the original poses is 33%, 42% and 50%; 90 source demonstrations without augmentation give 8%. These percentages match exactly 4/12, 5/12, 6/12 and 1/12, as with 3 trials at each of 4 locations, 12 in total, but this paragraph of the paper does not state the trials per location, so this is this site’s inference. If the inference holds, adjacent settings differ by a single success.

Generation quality is first evaluated in simulation. The authors built Sim Pick-Place Bottle in OmniGibson, modelled after the desk scene (same task structure, without replicating its geometry), and obtained ground truth by executing each generated demonstration’s converted actions in the simulator and rendering the onboard camera. Every tenth frame of 75 generated demonstrations was used, 2,486 frames in total; these views are 0.4–27.3 cm from the views used for reconstruction.

RegionPSNR (dB) ↑SSIM ↑LPIPS ↓
Full image21.27 ± 1.120.902 ± 0.0190.253 ± 0.030
Scene26.40 ± 2.180.936 ± 0.0210.248 ± 0.037
Object18.18 ± 2.150.688 ± 0.0810.341 ± 0.106
Robot17.43 ± 1.300.812 ± 0.0570.287 ± 0.048

Scene PSNR uses one frame per demonstration (75 frames in total). The authors cite about 23 dB reported by the original 3DGS paper on Tanks & Temples as a reference; that is a different dataset and serves only as an order-of-magnitude comparison. The authors attribute the lower object score to limited viewpoint coverage of a moving object, and the robot error mainly to lighting and shadow differences introduced in compositing. On 30 source demonstrations, the estimated object pose has a translation error of 1.46 ± 0.71 cm and a rotation error of 4.01° ± 4.66° relative to simulator ground truth. On real data, with the robot and object masked out, the scene scores 24.00 dB on 110 held-out frames; the object, computed on its bounding-box crop with the background masked out, scores 23.63 dB at poses registered directly by VGGT and 22.59 dB at interpolated poses. That region differs from the object mask behind the simulated 18.18 dB in the table above, so the two should not be compared directly. Because the simulated robot looks different from the real one, the authors do not report full-image metrics on real data.

How this relates to content on this site

Physical AI Lab 1.6 showed how behaviour-cloning errors carry a robot outside the demonstrations’ coverage. A shifted base is another way out: before the policy moves at all, the image it sees is already outside the training distribution. MoMaSplat does not collect more real demonstrations; it fills in observation coverage along the base-pose dimension. The policy it uses is the Diffusion Policy discussed in 1.6.

Physical AI Lab 1.5 listed synthesis as the third source of data. According to the paper’s related-work section, MimicGen-style methods generate purely kinematic plans that typically still need open-loop rollouts on the real robot to verify feasibility and capture images; MoMaSplat’s trade-off is to reconstruct the real scene and object and use simulation only to render the known robot embodiment. The frame conversion in step 5 can be read alongside the coordinate frames and kinematics of Physical AI Lab 1.1.

Cracking Mobile Manipulation in Frontier Radar on 2026-10-16 adds interventions at the planning, base-pose and end-effector stages; MoMaSplat takes another route, training a policy that tolerates base shifts instead of requiring the base to stop precisely. When reading success rates binned by distance from 36 trials per task, the approach of Physical AI Lab 1.8 helps: first count how many trials stand behind each percentage.

Limitations

  • This is an arXiv preprint; all numbers are author-reported [R], and this site has not reproduced them. The paper says code will be released; the text this site read gives no code address.
  • Each task has only 12 unseen poses with 3 trials each; the text this site read and the captions of Figs. 4–5 give no confidence intervals and no number of poses per distance bin. Single-cell figures such as 33% at 30 cm and 50% at 40 cm may rest on very few trials.
  • The four baselines were implemented or configured by the authors: VISTA’s perturbations are limited to ±10 cm per axis, and One-Thousand Trajectories received MoMaSplat’s action retargeting without the original method’s trajectory augmentation. The comparison depends on these settings.
  • The simulated evaluation of generation quality uses a scene modelled after the desk without replicating its geometry, with view distances of 0.4–27.3 cm. That is not the same quantity as the base distances of up to 50 cm in the policy evaluation, and the text this site read reports no fidelity figures beyond 27.3 cm.
  • Limitations stated by the authors: the method assumes mostly static scenes and rigid, visible objects, so deformable objects, heavy occlusion and texture-poor surfaces are hard; regions hidden by the object in its initial state (the surface beneath a lifted bottle, the inside of an opened drawer or air-fryer basket) are never observed during scene reconstruction and are not modelled when revealed; it depends on consistent open-world segmentation; generated demonstrations replay the source object trajectory without modelling contact forces or dynamics, so they are not physical demonstrations; view quality depends on the coverage of the navigation data.
  • The tasks are only rigid pick-and-place and single-degree-of-freedom articulation, base-pose sampling is limited to a 1.2 × 1.2 m region around the original poses, and large-scale navigation is not involved.
  • Real-robot success rates measure task completion, not safety validation; fidelity in simulation does not replace real-robot testing.

Links