A claim that a digital twin raises output once it closes the loop is only worth reading if closing the loop is the one thing that changed. The Factory Twin Lab in the noteflowai/robot-reel repository (this article follows the lab documentation and methods and limits at commit 7a1f08f, evidence level [O], i.e. the project’s own account) shows a review method you can copy. Every run is paired: in one the twin’s commands reach the plant, in the other they are only logged, and both share every disturbance and sensor-noise sample. The per-pair values for all 12 pairs are stored in seeds.json, including the 2 where the closed loop produced less.
What the pairing isolates
The “plant” is an entirely simulated factory and campus: a six-station production line, three AMRs, hall climate and campus energy, stepped at 1 s from 10:00 to 13:00. The digital twin receives telemetry from it, infers its state, compares candidate actions and sends commands back. Each seed runs in two modes:
- Closed: commands reach the plant 5 s after they are sent.
- Shadow: the same twin runs and logs the same kinds of decisions, but sends nothing.
Randomness comes from a counter-based generator keyed by seed, stream and index, so both runs of a pair see identical disturbances and noise samples. On that basis the documentation defines the difference between them as the effect of closing the loop in this model. That scope is narrow: it is not an estimate for a real factory, but a controlled comparison inside one model with one set of parameters.
This site’s “Physical AI Lab 0.1” describes physical AI as running the “see, think, act” loop over and over. Here the same loop is scaled up to a factory hall, and shadow mode supplies the control group that sees and thinks but does not act.
What the twin sees and how it decides
Reviewing a twin’s decisions starts with its information boundary. The documentation states that the twin never reads plant state directly; it works from telemetry alone:
| Step | What the twin does |
|---|---|
| Sense | Every 5 s it receives five packet groups: PLC, CNC, energy meter, hall climate and AMR poses. Each group is lost with 2–3% probability, all arrive 5 s late, and values carry noise and rounding |
| Sync | Mirrors station states, buffers and AMR poses from the last packet of each group and keeps the age of each group |
| Estimate | An extended Kalman filter tracks the hidden wear w and wear rate of the CNC 2 spindle from the vibration model v = 2.4 + 5.5 w² + noise. It starts from nominal parameters (initial wear 0.45, nominal rate, threshold 1.0) that differ from every seed’s hidden values |
| Predict | When the conservative (2σ) remaining life drops below 150 busy minutes, every 5 minutes it forward-simulates its own line model for 90 minutes under 14 plans; every minute it forecasts the current and next billing interval |
| Decide | Schedules service, derates feed, raises the hall setpoint and caps EV charging |
The 14 plans are seven service timings (now, in 10, 20, 30, 45 or 60 minutes, or not at all), each with and without derating. A plan’s value is expected good parts, minus the spindle life thrown away by early service (40 parts per unit of wear), minus 25 parts if the plan fails. The twin requests service only when “now” is the best plan; it switches derating only when the gain exceeds 1.5 parts; demand control moves through four levels (hall setpoint +1 / +2 °C, EV chargers at 50%) with hysteresis and a comfort guard below 27 °C.
These weights (40, 25, 1.5) were chosen for the demonstration. A review should record them as part of the decision: with different weights, the same telemetry could lead to a different service time.
The featured shift: list the gains and the costs
The featured shift in the documentation is seed 10 (10:00–13:00):
| Shadow (advice only) | Closed loop | |
|---|---|---|
| Good parts | 182 | 202 |
| CNC 2 spindle failures | 1 (45 min repair) | 0 (one 15 min planned service) |
| Scrap | 10 | 2 |
| Peak 15-min demand | 560.3 kW | 466.3 kW |
| Billing intervals over 520 kW | 2 | 0 |
| Max hall temperature | 24.2 °C | 24.8 °C |
The last row is one cost of the closed loop: its hall reached a maximum temperature 0.6 °C higher. Raising the hall setpoint is one of the twin’s demand-control actions, but the documentation does not break down the cause of the 0.6 °C. The closed-loop run also included one 15-minute planned stop. This is one recorded cost; the documentation does not give the full cost of avoiding the failure. Capping EV charging is also a cost of the closed loop; the Robot Reel documentation gives no seed-10 EV service metric, while the workbench maintainers report a seed-10 EV service metric of 74% for their imported v0.18.0 records, failing their default criteria, but the cited material does not establish that those records match the commit used here. A table that lists only gains hides these trade-offs from the reader; a review should ask for both sides.
Read all 12 pairs, including the two that lost
Across 12 paired seeds, the documentation reports:
- Spindle failures: 0 in closed loop, 11 in shadow.
- Billing intervals over the limit: 0 in closed loop, 8 in shadow.
- Good parts: up in 10 pairs, down in 2; mean +11.1, range −3 to +20.
The mean of +11.1 on its own hides those two pairs. The documentation and METHODS.md at commit 7a1f08f say both losing seeds, 3 and 11, had spindle failures in shadow mode, but they came late: only 13.6 and 14.5 minutes of the 45-minute repair fell inside the 3-hour window. Planned service of 15 minutes in closed mode therefore cost more output than those late failures. This trade-off belongs to these demonstration parameters and the 3-hour window; it does not establish maintenance gains in a real factory.
It is the same problem this site’s “EvalArc: the score went up, so where did the agent regress?” discusses: a higher average does not mean every item improved, so look item by item. This site’s introduction to Robot Reel covers the paired stress test of SmolVLA in the same repository; Factory Twin is a separate experiment that applies pairing to a decision loop.
Checking: replay instead of trust
Every pair can be re-executed. The documentation says only integer and IEEE-754 arithmetic influences the simulation, and that the Python standard library re-runs it bit for bit. --verify replays the twin from the recorded telemetry and requires identical estimates and decisions, which also tests the claim that the twin uses telemetry alone.
The steps below are the ones the upstream documentation gives; this site did not run them. Both routes require the robot-reel command to be available. The cited material does not specify installation or bind the executable version, so these commands are not a complete environment-reproduction guide. The first is the offline experiment: download and unzip experiment.zip from the project site or robot-reel-factory-twin-experiment.zip from the GitHub Release. In the METHODS.md example, --output points to the unpacked experiment folder (named factory-twin there); the upstream material does not say which directory to run the commands from:
robot-reel factory-twin --output factory-twin --verify # featured pair
robot-reel factory-twin --output factory-twin --verify --all-seeds # all 12 pairsThe second is to run from a checkout of the repository at commit 7a1f08f, following the Reproduce section of docs/factory-twin.md, with docs/factory-twin as the output directory:
robot-reel factory-twin --output docs/factory-twin --verify # featured pair
robot-reel factory-twin --output docs/factory-twin --verify --all-seeds # all 12 pairsAn installation of robot-reel from PyPI without a pinned version, the project site’s experiment.zip and the GitHub Release marked latest are not pinned to commit 7a1f08f, so their contents may differ from the version quoted here; the checkout in the second route pins the cited source files and recorded data, but does not pin the robot-reel executable actually invoked.
METHODS.md states that both commands need only Python 3.12 and the standard library.
Files to read during a review: seeds.json holds all 12 pairs; trace-closed.json and trace-shadow.json hold every sample of the featured seed, including plant truth, raw telemetry packets, twin snapshots, the decision log with evidence and prediction, and forecast checks against what the plant then produced. A service decision can be traced from the evidence in the log to the prediction made at the time, and then compared with the truth.
The 3D scene and the flythrough are made in Blender, but Blender only replays the recorded samples as keyframes and performs no physics. robot-reel factory-twin --verify checks the input hash of the Blender check receipt but does not rerun Blender. The film is presentation, not evidence.
Limitations
- The whole plant is simulated. Its parameters and the twin’s cost weights were chosen for the demonstration and are not calibrated to any real factory. The documentation says plainly that the numbers only show the loop works as described and are not an estimate of savings.
- That closing the loop works in this model does not show savings in a real factory, nor that these commands could safely control real equipment.
- Good parts fell in 2 of the 12 pairs, so the result cannot be reduced to “closing the loop always raises output”.
- This article did not rerun the commands above. The upstream checks, and whether the release files can be downloaded, are not independent reproduction; all numbers here are the project documentation’s own.
- The download channels (the PyPI package, the project site’s
experiment.zipand the latest Release) are not pinned to commit7a1f08fand may not match the documentation quoted here; a checkout at commit7a1f08fpins only the source files and recorded data; the invoked program version still needs separate confirmation. - The PAI Design Workbench project reports hosted acceptance of a read-only review: import pinned Robot Reel v0.18.0 records, freeze criteria, then compare every seed. Default criteria reject the plan: output falls at seeds 3 and 11, and seed 10 has an EV service metric of 74%, failing the default criteria. The cited workbench documents neither define how this metric is calculated nor give its threshold; 74% must not be read as a proportion of charging energy. This hosted acceptance did not rerun the factory simulation or validate real equipment control; it is the workbench maintainers’ acceptance record, not independent reproduction by this article’s author. The cited acceptance record does not establish that the v0.18.0 files are the same as commit
7a1f08fquoted in this article.
Links
- Factory Twin Lab documentation (robot-reel, commit 7a1f08f)
- Factory Twin Lab: methods and limits (METHODS.md, commit 7a1f08f)
- Factory Twin Lab, interactive page
- robot-reel (PyPI)
- This site: Physical AI Lab 0.1: What Is Physical AI? From Chatting to Doing
- This site: Robot Reel: turning robot policy simulations into frame-by-frame verifiable replays and 3D scenes
- This site: EvalArc: the score went up, so where did the agent regress?
