An explanation a model writes from sampled robot frames does not by itself show whether it matches the recording. The Robot Reel guide Your first model-claim review takes a more restrained approach: watch the original two-camera recordings first, take one model answer exactly as it was saved, and use robot-reel review-claims to check only the facts that can be checked mechanically, while explicitly marking the explanation itself as not assessed. This article walks through that workflow as documented at the pinned revision 7f71f58, with the focus on what it can and cannot establish.
Note: the commands, expected exit codes and experimental setup below all come from the project documentation (level O, the project’s own account). This site has not re-run them, and nothing here is an independent reproduction. For an overview of the Robot Reel project as a whole, see Robot Reel: turning robot policy simulations into frame-by-frame verifiable replays and 3D scenes; this article covers only the model-claim review step.
What the demonstration covers, and what it does not
According to model-review.md, the demonstration selects three episodes from the existing seed-09 SmolVLA stress experiment, one each for the reference, dim and camera conditions. Each episode has a main-camera and a wrist-camera recording; five frames are sampled from each and given to the model Qwen/Qwen3.8-27B-FP8 in two input modes: the sampled frames alone (Images only), and the frames plus the machine-readable recorded outcome (Images + record). One answer was generated per mode per episode, six answers in total.
The documentation is just as explicit about what this is not: not a new policy rollout, not a model ranking, not a population accuracy estimate, and not evidence of a physical failure mechanism. Three selected episodes are a demonstration, and six answers cannot support any statistical conclusion.
Setup: a pinned package and pinned evidence
The workflow needs no GPU, model account, repository checkout or new simulation. The guide asks for Python 3.12 or later and installs the published package in a fresh directory (the standard installation also pulls in Robot Reel’s simulation dependencies, but only the offline review command is used here):
mkdir robot-claim-first-review && cd robot-claim-first-review
python3 -m venv .venv
.venv/bin/python -m pip install robot-reel==0.16.0The evidence is a ZIP from the v0.16.0 release containing three SmolVLA episodes, both original camera recordings, traces, the exact sampled images, six model answers and the offline page. The guide’s script compares the SHA-256 first and refuses to extract on a mismatch:
.venv/bin/python - <<'PY'
import hashlib
import io
from pathlib import Path
from urllib.request import urlopen
from zipfile import ZipFile
url = "https://github.com/noteflowai/robot-reel/releases/download/v0.16.0/robot-reel-model-review.zip"
expected = "7691e06219d840dab3a19f3e411ad513ed69f00e3c69437e046b1f032cc601b3"
with urlopen(url, timeout=60) as response:
data = response.read()
if hashlib.sha256(data).hexdigest() != expected:
raise SystemExit("Archive checksum differs; do not extract it.")
Path("recorded-review").mkdir()
with Path("review.zip").open("xb") as output:
output.write(data)
with ZipFile(io.BytesIO(data)) as archive:
archive.extractall("recorded-review")
PYAfter extraction, open recorded-review/index.html, select dim, play the paired camera recordings, then compare the answers from the two modes. The order matters: watch the original recordings before reading the model’s explanation. The hosted version is the Robot Reel model review page.
Checking an answer exactly as it was saved
The claims file that review-claims accepts has exactly four fields:
{
"outcome": "step_limit",
"action_count": 160,
"cited_frames": [0, 160],
"explanation": "A visual interpretation to be reviewed separately."
}This JSON is a format example written by the documentation’s author, not model output. The real input is the text field taken from the recorded/ directory, with no editing or cleaning:
.venv/bin/python - <<'PY'
import json
from pathlib import Path
source = Path("recorded-review/recorded/episode-b-with-record-17.json")
record = json.loads(source.read_text())
with Path("claims.json").open("x") as output:
output.write(record["text"])
PY
.venv/bin/robot-reel review-claims \
recorded-review/media/episode-b/trace.json claims.json --output my-review.jsonThe checker reports each fact as matched, contradicted or unassessed, binds both input files by SHA-256, and refuses to overwrite an existing report. The exit codes mean:
| Exit code | Meaning |
|---|---|
| 0 | Both the outcome and the action count match, and at least one cited frame exists |
| 1 | Valid JSON, but with a contradicted or unassessed claim |
| 2 | Invalid input, or an output error |
For this Images + record answer the expected exit code is 0: the outcome step_limit, the action count of 160 and the cited frame labels all agree with the trace, and the report keeps "explanation_status": "not_assessed". unknown and null preserve uncertainty and are never treated as a pass.
The guide adds an important caution: this answer was given the outcome record as input, so repeating it is record reading, not independent discovery of the failure by the model. A frame number that exists does not show that the image supports the model’s explanation, let alone prove a physical cause.
Keeping a format failure visible
In the same directory, extract the Images only answer unchanged and run the same checker:
.venv/bin/python - <<'PY'
import json
from pathlib import Path
source = Path("recorded-review/recorded/episode-b-images-only-17.json")
record = json.loads(source.read_text())
with Path("images-only.json").open("x") as output:
output.write(record["text"])
PY
.venv/bin/robot-reel review-claims \
recorded-review/media/episode-b/trace.json images-only.jsonThe Images only answer for the same episode did not return plain JSON as requested; it wrapped its JSON in a Markdown code fence. Passing its exact text to the same checker is expected to exit with 2, because it is not a strict JSON document.
The guide’s rule: do not strip the fence and report the cleaned content as the model’s original result. The offline page extracts the fenced content only for readable display and keeps the format failure marked. For evaluation purposes, “the model did not answer in the requested format” is itself a result worth recording.
What the model received
The protocol was committed before generation. examples/model-review/sources.json fixes the source commit, the clip and trace hashes, the sampled frame indexes and the exact contact-sheet hashes. The contact sheets are decoded from the published lossy MP4 derivatives; they are not the original RGB buffers the policy received. The two prompt modes differ in one respect only: whether the machine-readable recorded facts are attached.
Generation settings: model revision 017b9c7af6b5689d5dd426a76e0bc077eb5ca20a, seed 17, temperature 0.6, top-p 0.9, top-k 20, thinking disabled, a 512-token budget. All completed outputs, including invalid JSON and wrong claims, are retained. The recording used an L40S; the documentation warns that fresh generation may differ across hardware or library versions, and that ordinary review should simply use the published outputs.
The documentation also records a loading problem: Transformers 5.17.0 interprets the checkpoint’s nonexistent dense-model mlp.gate exclusions as regex prefixes, wrongly excluding gate_proj as well and losing its FP8 scales. The recorder’s --fix-dense-fp8-skip removes those router exclusions in memory only, leaves the source weights unchanged, and rejects any missing, unexpected or mismatched checkpoint keys; notes from failed initializations are kept separately from completed outputs.
Limitations
- The automatic check covers only the outcome label, the action count and whether cited frames exist. The explanation and any causal judgement are always
not_assessedand need separate human review against the recordings. - The trace is supplied evidence, not independently authenticated physical truth; agreeing with the trace does not mean agreeing with the real world.
- Three selected episodes with one generation per condition cannot be used to compare models or estimate accuracy.
- The model saw contact sheets decoded from lossy MP4 files, which differ from the images the policy actually received.
- Keep the archive’s
NOTICE.txtwhen redistributing the robot media; Robot Reel’s code license does not relicense the upstream robot assets. - This site has only read the pinned documentation and has not run the commands above.
“Checkable” is not “confirmed”. This matches the evidence standard of the Physical AI Lab course: being able to find the original source does not mean the conclusion in that source holds.
