A cube on a conveyor belt, a moving cup: once the target moves, a VLA that does well on static tasks tends to miss. DSDyn-VLA, a preprint from the University of Sydney, Central South University, UC San Diego and Ace Robotics (arXiv 2609.39198, 2026-09-30, evidence level [R]), breaks the cause into three gaps (perception, latency and control) and pairs a slow planning stream at about 2–3 Hz with a fast residual-correction stream at about 30 Hz. The authors report that the average success rate on four dynamic real-robot tasks rises from 8.4% with π0.5 to 49.0%, the “about 6×” in the abstract. The more informative comparison is with the strongest baseline, DynamicVLA: 38.1% versus 49.0%.
Three gaps: where static VLAs get stuck on moving targets
The paper sorts the failures of current VLAs on dynamic manipulation into three kinds:
- Perception gap: a standard VLA takes static RGB snapshots, which keep the semantics but drop motion cues. Without velocity information it cannot estimate the target’s trajectory or time to contact.
- Latency gap: under asynchronous inference the VLA starts inferring when
T_latsteps remain in the action buffer, so actions are generated from the stateT_latsteps earlier but executed afterwards. This hardly matters in a static scene; with a moving target it causes misalignment. - Control gap: once an action chunk starts executing it runs open-loop, and until the next inference the robot is “blind” to changes in the environment.
Among related work, RTC, VLASH and DynamicVLA mainly address the latency gap. The authors argue that these methods still rely on static visual input and open-loop chunk execution, and so leave the other two gaps open.
The slow stream, Flow-Planner: optical flow in, planning for the execution time
The slow stream is built on the π0.5 architecture (a conditional flow matching policy) with two changes:
- Motion-aware perception: RAFT-Large extracts dense optical flow between consecutive frames. A polar mapping turns the two-channel flow into a three-channel RGB-like image, with direction as hue and speed as saturation, so that it aligns with CLIP’s pre-trained semantic space. The RGB image and the flow image go into two parallel Vision Transformer encoders, and their output tokens are concatenated along the sequence dimension.
- Future state awareness: while inference runs, the robot keeps executing the buffered actions of the previous chunk, and those are known. Flow-Planner rolls these buffered actions through forward kinematics to compute the proprioceptive state at execution time, and conditions on that state instead of the state at observation time. Training uses a randomized time shift: the image at time t is paired with the proprioceptive state and action chunk at t+Δ.
Training has two stages. Stage 1 fine-tunes Flow-Planner with supervision on dynamic demonstrations, with the time shift applied. Stage 2 freezes Flow-Planner and trains only the fast stream.
The fast stream, Res-Refiner: a bounded residual at 30 Hz
The fast stream is a lightweight residual policy: a ResNet-10 visual backbone with 8 learnable spatial blocks produces a 256-dimensional representation. This is concatenated with the proprioceptive state and Flow-Planner’s action for the current step, then passed to an MLP with two hidden layers of 256 units, which outputs a Tanh-squashed residual. The executed action is the current step of the chunk plus the residual (Eq. 2 in the paper), and the residual in each dimension is bounded to 10% of the full action range.
Training uses RLPD: sparse binary rewards, a replay buffer that is half offline demonstrations and half online experience, and an ensemble of critics to curb Q-value overestimation. On the real robot, the reward is given manually once the task is seen to succeed, and the RL stage takes about 30–60 minutes per task.
Results reported by the authors
The real-robot platform is a 6-DOF Agilex Piper arm with two Intel RealSense D435 cameras. The slow stream is based on π0.5 with a chunk size of 10, and the fast stream runs at 30 Hz. The four tasks are conveyor picking, dropping into a moving container, stacking onto a moving cube and pouring into a moving cup, each with 5 × 50 trials:
| Real-robot task (success rate) | π0.5 | RTC | VLASH | DynamicVLA | DSDyn-VLA |
|---|---|---|---|---|---|
| Conveyor picking | 8.8% | 12.4% | 20.0% | 33.2% | 44.8% |
| Dynamic dropping | 14.8% | 43.6% | 50.0% | 56.0% | 62.8% |
| Dynamic stacking | 1.2% | 2.4% | 6.8% | 18.0% | 32.0% |
| Mobile pouring | 8.8% | 16.0% | 30.4% | 45.2% | 56.4% |
| Average | 8.4% | 18.6% | 26.8% | 38.1% | 49.0% |
Against the strongest baseline, DynamicVLA, the average improves by 10.9 percentage points, and by 6.8 to 14.0 points per task. “About 6×” is the multiple relative to π0.5.
There are two simulation evaluations. DynBench is the authors’ own MuJoCo benchmark with a Franka Panda arm and 9 tasks (four ported from the real-robot tasks), each with 100 rule-generated demonstrations and 100 evaluation trials. Average success rates: π0.5 9.7%, RTC 19.4%, VLASH 27.1%, DynamicVLA 34.3%, DSDyn-VLA 48.4%. Kinetix is a state-based benchmark; the authors add an artificial delay of 0–4 steps on 12 tasks, with 2048 trials per setting. The paper says that in the high-latency setting the failure rate of SOTA methods falls from 13% to 3% (the abstract puts it as a reduction of over 76%), and that success stays above 97% at the largest delay (δ=4). The appendix also reports results no lower than π0.5 on the four static LIBERO suites (for example 93.30% versus 92.60% on LIBERO-10), and a success rate of 20.74% on the external dynamic benchmark DOMINO, above PUMA’s 17.18%.
Ablations and overhead
In the real-robot design ablation (four tasks, 50 trials) the full model scores 48.5%. Without optical flow it falls to 30.5%; without the polar mapping (a ResNet-34 encoding the flow directly) 33.0%; without future state awareness 36.5%; without Res-Refiner 25.0%; and with Res-Refiner trained by supervised fine-tuning instead of RL 37.5%. On DynBench, removing either the Refiner or the Future module gives 42.7%, and removing the flow branch 35.3%.
To address the objection that the gains merely come from more temporal context or extra RL, Appendix D gives RTC, VLASH and DynamicVLA two additional history frames and 1 hour of end-to-end PPO each. They rise from 18.0% to 24.0%, 26.5% to 32.5% and 37.5% to 39.0% respectively, still below DSDyn-VLA’s 48.5%. Replacing optical flow with simple frame stacking gives 42.5%.
Per-module timing on an RTX 4090: the VLA backbone takes 73 ms in total, system latency 13 ms, RAFT-Large 3 ms, the flow encoder 14 ms and the kinematics rollout under 1 ms, for a Flow-Planner total of 103 ms. Res-Refiner takes 3 ms in total, within one 30 Hz period (about 33 ms). On Dynamic Stacking and Mobile Pouring, the residual stays below 80% of its bound in every dimension in 95.8% of cases; the authors say the bound rarely saturates, suggesting that the 10% budget is sufficient. When optical-flow quality degrades, success declines gradually: 42.0% at 70% occlusion and 40.5% under blinking light (48.5% under clean conditions).
How this relates to other pages on this site
Physical AI Lab 0.3: See, Think, Act — Putting Each Rate Back in Its Own System uses the analogy that the cerebrum thinks slowly while the cerebellum acts quickly, and stresses that it is only an analogy. The DSDyn-VLA authors likewise say the dual-stream design is inspired by the division of labor between cerebrum and cerebellum. By that lesson’s rule, the roughly 2–3 Hz and 30 Hz here belong only to the DSDyn-VLA system and should not be combined with the rates of other models or interfaces. The full season is laid out in the course outline.
VLaRL also freezes a base policy and adds residual RL on top, but it trains the residual only in simulation, with no real-robot RL. DSDyn-VLA trains its residual on the real robot with manually given binary rewards, with the aim of keeping up with a moving object while a chunk is executing. The 103 ms latency breakdown can be read alongside Frontier Radar deep dive | Edge and real-time; the paper says the fast stream can be deployed on edge devices, but all the timings it gives were measured on an RTX 4090.
Limitations
- This is an arXiv preprint; all numbers are reported by the authors and this site has not reproduced any of them. The abstract says all code and weights will be open-sourced; the reproducibility statement says the code is in the supplementary materials and the weights will be released upon publication. The material this article relies on gives no public repository address.
- In the Kinetix experiments, Flow-Planner is replaced by an MLP policy that predicts action chunks (H=8), the optical-flow module is omitted, and the only baselines are RTC and VLASH. That experiment tests future state awareness and residual correction, not the full vision-based VLA.
- DynBench was built by the authors and its demonstrations are rule-generated. The real-robot evaluation uses a single platform and four tasks, rewards are judged manually, and RL is run separately for each task.
- The real-robot main table (49.0%, 5 × 50 trials per task), the design ablation (48.5%, 50 trials) and the baseline numbers in Appendix D (no trial count given) differ slightly from one another, and the paper does not explain the differences; take care when comparing across tables.
- In their conclusion, the authors state that the generalization of the residual policy and real-robot evaluation on more tasks and platforms still need to be extended, and that safety and failure risks in environments shared with people need dedicated handling.
Links
- arXiv 2609.39198: DSDyn-VLA: A Dual-Stream Dynamic Manipulation Framework with Motion Perception, Future Awareness, and Realtime Correction (authors: Wenhao Li, Xiu Su, Yu Han, Yichao Cao, Shan You, Chang Xu)
- Full paper in HTML
