After a pretrained robot foundation model is adapted to a target task with offline teleoperated demonstrations, deployment still leads it into states the demonstrations did not cover: contact-rich interactions, situations that require recovery, and precision-critical configurations. In the preprint TIDE (Training In DEployment, arXiv 2610.29264, 2026-10-21), authors from Fudan University, the Shanghai Innovation Institute, the Logos Robotics Group and the University of Surrey turn deployment itself into training. The robot executes the current policy, an operator takes over only when necessary, the system labels a 50-step action chunk by its source, “human-executed” or “policy-executed”, only when all 50 actions share that source, excluding mixed chunks, and a learner updates the policy online within the same session, swapping in new checkpoints at episode boundaries; no learned critic, reward model or action scorer is involved. The authors report that on four real-world bimanual tasks TIDE achieves higher task progress and more full completions than the warm-start policy and matched additional offline demonstrations, but ends up comparable to a baseline that collects data with a fixed policy and trains afterwards; its distinction is that new checkpoints become usable during deployment. All numbers below are author-reported [R]; this site has not reproduced them.
The problem: offline post-training waits for the next round, online RL has to estimate value
The paper groups existing approaches into two kinds and names the cost of each. The first is offline post-training, which follows a collect-train-deploy cycle: deployment experience is used only in the next offline training round, so adaptation under distribution shift is slow. Existing human-in-the-loop methods also tend to treat deployment mainly as a source of expert action labels: human-intervention segments are used as demonstrations, while policy-generated segments are discarded or treated as unlabeled data, even though most deployment experience is generated by the policy itself. The second is online reinforcement learning, which can use both interventions and policy executions but usually relies on value functions or critics trained from sparse episode-level success-or-failure feedback; the authors argue that such weak supervision makes it hard to evaluate high-dimensional action chunks stably, increasing training complexity and instability.
In their related-work section the authors explain how several systems each close only part of the loop (the following is the authors’ characterization). RoboCopilot’s workflow remains round-based, and the deployed policy still comes from batched retraining. RECAP scales full-model human-in-the-loop improvement with advantage conditioning but still operates through phase-based offline updates, leaving a fully concurrent online framework as future work. RL Token trains a lightweight actor-critic on top of a frozen pretrained policy instead of updating the full backbone. Hi-ORS avoids a learned critic through outcome-based rejection sampling but still relies on trajectory-level outcome supervision. ConRFT retains a Q-function and gates online fine-tuning with human interventions. TIDE positions itself where real-robot deployment stays in the data loop, the video backbone, cross-modal adapter and action expert are updated jointly, and supervision is organized by action-source provenance rather than learned values or trajectory-level reward labels.
The paper frames human-in-the-loop learning as a response to covariate shift and compounding errors: a human can intervene in states the current policy itself visits. Physical AI Lab 1.6: Imitation Learning showed how an error rolls out of the demonstrations’ coverage; Physical AI Lab 1.7: Reinforcement Learning covered how rewards are written and what a critic is trained on. TIDE sidesteps the latter: whether an episode succeeded only decides whether it enters the online dataset, and is never used as a reward, model condition or training target. This site’s Frontier Radar deep dive | Training and self-improvement divided VLA post-training into three stages, the last being learning from deployment experience; TIDE is one approach in that stage that does without value estimation.
Three parts in one session: the robot client, the inference-and-recording server, the online learner
TIDE adapts a pretrained world-action policy. A video-generation backbone turns the observation and language instruction into a compact “world prior”; a flow-matching action expert cross-attends to this prior and the robot state to generate a 50-step action chunk, while the backbone predicts future video frames over the same horizon. Before deployment, all trainable modules are warm-started on offline teleoperation data for the target task. The policy reported in the paper has 3B parameters and is trained on eight H100 GPUs.
- The robot client handles low-latency control: at each control step it collects observations, requests action chunks from the server and executes the policy’s actions unless the operator intervenes. Inspired by RTC, it uses asynchronous real-time chunking: committed actions keep executing while future actions are refreshed in the background, which reduces inference-induced stalls and smooths transitions between consecutive chunks. During takeover, the operator uses VR delta control to command incremental motions around the current state and can hand control back to the policy without resetting the episode; every handoff clears the queued policy chunks so that stale actions are not executed. Executed actions and intervention flags are streamed asynchronously, off the critical control path. Physical AI Lab 2.3: Fine-Tuning SmolVLA worked through the timing of an action queue for another system’s asynchronous inference and is a useful comparison.
- The inference-and-recording server centralizes GPU inference and writes online episodes to a shared dataset. At the end of an episode, the operator manually confirms whether the task was completed; only confirmed-successful episodes are committed, while failed, timed-out, aborted or stopped, reset-before-completion, re-recorded, stale and unrecovered episodes are discarded. The server adopts learner-produced checkpoints only at episode boundaries, so that each episode comes from a single policy generation.
- The online learner reads newly committed records and, together with offline task data, jointly updates the video diffusion backbone, the cross-modal adapter/Q-former, the action expert and the source embeddings; the pretrained text encoder and video VAE stay frozen. The learner allocates 0.2 optimizer steps per newly committed control frame and publishes a checkpoint every five committed episodes; the server prefetches the new weights and adopts them at the next episode boundary.
The paper states its time measure as follows: for each task, the continuous wall-clock interval from the start of online human-in-the-loop rollouts until the final checkpoint is deployable is less than one hour. This interval includes asynchronous data processing, training, checkpoint handling, robot execution, human takeover and episode resets, but excludes initial setup and the final autonomous evaluation. On this setup each online optimizer step takes about 0.4 s. Bit Insertion is given in detail: 41 committed episodes and 38,839 control frames (21.6 minutes at 30 Hz), of which 1,432 frames (3.69%) were human-controlled and 37,407 were policy-controlled.
A rough calculation from the paper’s stated rates (this site’s arithmetic; the paper does not report the actual number of optimizer steps or checkpoints): 0.2 × 38,839 ≈ 7,768 optimizer steps, which at about 0.4 s per step comes to about 3,107 s, or about 52 minutes; with a checkpoint every five of 41 episodes, about 8 checkpoints are published. If each action step corresponds to one 30 Hz control frame, a 50-step chunk covers about 1.67 s (also this site’s arithmetic).
Provenance labels: a 50-step chunk is either all human or all policy
Within an online episode, human takeover and policy execution alternate. Rather than treating a whole episode as one demonstration, TIDE labels at the level of action chunks: at every control step it records the action sent to the robot and whether it came from the operator or the policy. A chunk executed entirely by the human is labeled 0; entirely by the policy, 1; a chunk that mixes human and policy steps is labeled ∅. A 50-step target-action horizon receives a human or policy label only when all 50 actions share that source; ∅ horizons are excluded from provenance-conditioned training, with no additional exclusion window around a handoff.
The paper repeatedly stresses two boundaries. First, provenance records execution source only; it is not a reward, a correctness label or a quality score. Second, episode success is used only to decide whether an episode enters the online training dataset; it is not a model condition, reward, provenance label or training target. In training, each provenance label maps to a learnable embedding that is added to the flow-time embedding of both the video backbone and the action expert; the objective is the sum of two flow-matching losses, one for future video and one for actions. All reported online autonomous rollouts and final no-takeover evaluations fix the source condition to the human mode (ℓ = 0).
The data mix is also stated. Offline teleoperation data are treated as human-provenance chunks (ℓ = 0). The learner keeps separate offline and online buffers, and online samples receive a 5:1 weight relative to offline samples, i.e. selection probabilities of 5/6 and 1/6. The value α = 5 was chosen in preliminary tuning and is fixed across all reported tasks and variants. Within the online buffer, human-executed and policy-executed chunks are balanced so that sparse intervention data are not overwhelmed by autonomous execution. Two task-preserving augmentations are added: when the task remains valid under the transformation, observations, states and actions are mirrored across the workspace symmetry plane; and action chunks and the corresponding future observations are resampled at different execution rates. Augmented samples inherit the provenance label of the original chunk.
One consequence follows from the label definition (this site’s inference; the paper does not discuss it): because a horizon counts as “human-executed” only if 50 consecutive actions share that source, a takeover shorter than 50 control steps yields no human-labeled horizon at all, and the horizons containing it are labeled ∅ and excluded. In Bit Insertion, human-controlled frames make up only 3.69%; the paper does not report how many of them ended up in pure-human horizons or how many horizons were labeled ∅.
Evaluation: 20 no-takeover rollouts per pair, two metrics, three comparisons
All real-robot experiments use a bimanual UR5e platform with Robotiq 2F85 grippers, one third-person RealSense D455 camera and two wrist-mounted RealSense D405 cameras. The four tasks and their milestones are: Bit Insertion (insert a screwdriver bit into a bit holder; pick, insert), Towel Folding (flatten a spread towel and fold it into a compact square; flatten, first fold, second fold), Towel Hanging (place a towel onto a hanger; pick hanger, place towel, hang hanger) and Chain Packing (arrange a metal chain into a box; place chain, close lid, tuck flaps). The policy outputs a 50-step, 14-dimensional, joint-relative bimanual action chunk: each predicted offset is added to the current joint state before the controller receives the resulting absolute joint target. Before choosing this action representation, the authors compared eight action spaces in offline-only training with the same data, initialization and budget; joint-relative actions performed best in that controlled study.
Each method–task pair is evaluated with 20 autonomous rollouts from the same predefined set of initial states, with no human takeover. Task progress is milestone-normalized: the number of milestones completed in a rollout is divided by the task’s total, then averaged over the 20 rollouts; if a rollout ends because of a timeout, safety stop or task failure, only milestones already stably completed are counted. Full-task completion is reported separately as k/20 and counts only rollouts that complete every milestone. Task performance is measured only with these no-takeover evaluations because, during collection, human takeover can complete episodes that would otherwise fail.
All three comparisons start from the same task-specific warm-start checkpoint. Warm is the warm-start policy itself. Offline collects new teleoperated demonstrations matched to the amount of online data and trains for the same number of optimizer steps as TIDE. Fixed Policy keeps the warm-start policy fixed during human-in-the-loop collection matched to TIDE by duration, then trains after collection with the same objective, buffer configuration, filtering rules and total optimizer-update budget. The paper describes the results in words:
- On milestone-normalized progress, both Fixed Policy and TIDE numerically outperform Warm and Offline on all four tasks.
- TIDE matches Fixed Policy on Bit Insertion and Towel Hanging progress, and is higher on Towel Folding and Chain Packing.
- Full-task completion is tied on Bit Insertion; TIDE is higher by one rollout on both towel tasks and by two rollouts on Chain Packing (out of 20 in each case).
- On this basis the authors say the two have comparable final performance under this evaluation; TIDE’s distinction is that updated checkpoints become available during the deployment session rather than only after a separate collection, training and redeployment cycle.
The body text read does not list the per-method percentages shown in Fig. 6. With 20 rollouts per pair, one additional full completion is 5 percentage points (this site’s arithmetic). Physical AI Lab 1.8: Evaluation practised breaking such small differences down into paired outcomes, and this site’s Frontier Radar | Beyond Binary Success covered a statistical method for comparing policies with sequential tests when real-robot rollouts are limited. The same questions apply to the one- or two-rollout differences between TIDE and Fixed Policy.
Two further diagnostics are run on Bit Insertion only. In the ablation, all variants start from the same warm-start checkpoint, use the same online interaction budget and are evaluated with 20 no-takeover rollouts from the same initial states: TIDE reaches 80.0% normalized task progress, versus 65.0% without provenance conditioning and 67.5% when only human-executed online chunks are used. The authors say this single-task ablation supports both explicit source conditioning and the use of policy-executed chunks. The intervention ratio is the fraction of executed control steps under human takeover in each training episode: in the paper’s Fig. 8(a), the frame-weighted ratio over the most recent five committed episodes decreases after the early deployment phase. The authors state explicitly that this trace reflects supervision burden only, is not a measure of autonomous performance, and does not by itself isolate learning from temporal factors.
Limitations
- This is an arXiv preprint; all numbers are author-reported [R] and this site has not reproduced them. The text read gives a project page address but does not say whether code or model weights are released; this site did not open the project page.
- Limitations listed by the authors: TIDE requires a human supervisor; retaining only manually confirmed successful episodes may introduce selection bias; the evaluation covers one bimanual platform, four tasks and one world-action backbone, does not test long-term retention or sensitivity to the online-training schedule, and does not measure variation across repeated online post-training sessions. Generalization across platforms and architectures therefore remains unverified.
- Against Fixed Policy, the authors’ own conclusion is “comparable”, not “better”; the differences in full completions are only 0–2 rollouts (out of 20 per pair). The body text read does not give the values in Fig. 6, and this article only restates the paper’s verbal comparison.
- The provenance-conditioning ablation (80.0%, 65.0%, 67.5%) was run on Bit Insertion alone. The declining intervention ratio is not a measure of autonomous performance.
- “Less than one hour” is the wall-clock time from the start of online rollouts until the final checkpoint is deployable; it excludes initial setup, offline warm-start and final evaluation, and applies to training a 3B policy on eight H100 GPUs. The roughly 7,768 steps, 52 minutes, 8 checkpoints and 1.67 s in the second section are this site’s arithmetic from the paper’s rates; the point about short takeovers in the third section is this site’s inference from the label definition, and the paper reports no corresponding data.
- Human takeover in the paper is a data-collection mechanism; the text read does not present it as a safety evaluation, and these real-robot results are not a hardware safety validation.
Links
- Training in Deployment: Human-in-the-loop Online Post-Training for Robot Manipulation (arXiv 2610.29264)
- Full paper (HTML)
- Physical AI Lab 1.6: Imitation Learning — Working Out How an Error Rolls Out of the Demonstrations’ Coverage, Then Putting the ACT, Diffusion Policy and Data Scaling Numbers Back in Their Own Conditions
- Physical AI Lab 1.7: Reinforcement Learning — Working Out One Episode’s Return to See How a Reward Gets Gamed, Then Reading the Paper’s Terrain Curriculum and Time-out Resets
- Physical AI Lab 1.8: Evaluation — Splitting a “Net Gain of 2” into a Paired Table, Then Checking the Failure Labels and Random Seeds
- Physical AI Lab 2.3: Fine-Tuning SmolVLA — Planning Demonstrations for an Instruction, Checking What Each Number Measures, Then Timing the Asynchronous Action Queue
- Frontier Radar deep dive | Training and self-improvement: three stages of VLA post-training and task collapse in online RL
- Frontier Radar | Beyond Binary Success: sequential tests that separate robot policies with fewer real-world rollouts
- Physical AI Lab: a physical AI course where every number can be checked (course outline)
