Checking an agent’s physics estimate against held-out solver results: the evaluation passed, but the height trend was backwards

pai-design-workbench uses a small evaluation to measure a model’s physics estimates: given a bracket and one measured reference point, the model states how the deflection at the motor axis scales, as a power law, with plate thickness, width and height, and those exponents are then compared with held-out solver results. At commit 70330e2 the repository keeps the record of one real-model run: median error 0.114, worst 0.321, Spearman 0.95. All three clear the bar, yet the height exponent has the opposite sign to the measured trend. This article covers how the evaluation keeps the model’s output as data rather than code, how to read these numbers and baselines, and why the solver remains the final authority even after a pass.

It is based on three files pinned at commit 70330e2: the evaluation record docs/evidence/physics-eval.json, the evaluation script scripts/eval-physics-solver.ts and the requirements table docs/REQUIREMENTS.md ([O], the project’s own account). The record describes one saved development run; this site did not run the script, call a model or reproduce any of its numbers. How the same repository carries CFD cases to AWS Batch is covered in Carrying a pinned OpenFOAM case to AWS Batch: hashed results, two meshes and a drag comparison with the local run; this article is about something else: the gap between a model’s estimate and the solver’s results.

What the evaluation asks: one reference point and three exponents

The script’s prompt asks the model, as a mechanical engineer, to predict how this bracket’s deflection at the motor axis scales with its parameters, and to return a single JSON object and nothing else:

{"thicknessExponent": a, "widthExponent": b, "heightExponent": c, "rationale": "<= 600 chars"}

The prompt defines the exponents: deflection = d_ref * (t/t_ref)^a * (W/W_ref)^b * (H/H_ref)^c, where d_ref is the measured reference deflection and t, W and H are plate thickness, width and plate height. It tells the model to reason from plate bending, the ribs at the side edges and where the load enters, and bounds the exponents to [-6, 6]. The material the model sees is the task workspace’s README.md (the bracket description) and reference.json (the reference point), both enclosed verbatim in the prompt.

The record names the task agentforge examples/pai-workbench/eval/tasks/estimate-bracket-deflection, noted as merged in 9594430. Requirement F17 lists this as using eval/ to measure the model’s physics estimates.

The model returns numbers, not code

The key design choice is that the model’s output is always data. The record describes the path as “scripts/eval-physics-solver.ts → NoteFlow bounded executor (shared ledger, 1 attempt, no retry); the model returns exponents as JSON, no code is executed”. The script does this as follows:

• It calls the model through the bounded executor using only the first enabled engine profile (enabledProfiles(config).slice(0, 1)), with a 60-second timeout. Requirement F3 describes real-model calls with a primary, a backup and a second backup; only the first is used here, so the result belongs to that one profile.
• If the executor does not return done, or returns no answer, the script throws an error. There is no retry.
• It cuts the answer from the first { to the last } and validates it with zod: the three exponents must be numbers in [-6, 6] and rationale a string. The prompt asks for a rationale of at most 600 characters; validation accepts up to 2,000.
• It fills the three numbers into a fixed template and writes estimate.mjs, which reads reference.json and returns the estimate by the power law above. Nothing the model writes becomes executable code.
• It also writes a record to eval/<id>.json under the state directory, holding the engine details, the three exponents and the profile and status of each attempt.

According to the script’s header comment, the base eval harness runs it in the task workspace:

node examples/pai-workbench/eval/run.mjs --command "node --env-file=<wb>/.state/demo.env --import tsx <wb>/scripts/eval-physics-solver.ts"

Here <wb> is the location of the workbench checkout. run.mjs is not among the files this article relies on, so how it reads the held-out data and computes the three metrics cannot be checked here; the script itself produces only estimate.mjs and the record above. The command also needs the AgentForge task directory and a working model profile. The supplied material gives no versions or installation steps for these, so pinning the workbench commit does not pin the whole runtime environment.

The saved result: three metrics and three baselines

The record was checked at 2026-10-03T17:35:00Z, with provider kiro, engineVersion 2.27.1 and model claude-opus-5.5. Its modelEvidence field is acp_advertised: the model name is the identity the engine advertised over ACP, not independently verified model provenance. The model’s exponents were t: -2.6, W: +1.2, H: +0.8.

Metric	Model	Threshold
Median error (median)	0.114	0.15
Worst error (worst)	0.321	0.35
Rank correlation (Spearman)	0.95	0.9
Verdict (passed)	true	—

The record lists the threshold values without stating the direction of comparison; requirement F17 reports the same result as “median error 11.4 %, worst 32.1 %, Spearman 0.95, passed”. Read with lower error and higher rank correlation as better, all three are within the thresholds. Spearman only measures whether estimates and measurements are in the same order, not how far apart the values are.

The record also lists three baselines, each of a different kind:

Baseline	t	W	H	Result in the record	How to read it
fixtureSolution	-2.5	0.7	0	median 0.011, worst 0.192, Spearman 1.0	The task’s fixed solution, not a fresh model attempt
pureThinPlate	-3	0	0	median 0.145, worst 0.354, failed	Estimates only from thin-plate stiffness scaling with t³; median within 0.15, worst above 0.35
leastSquaresOnHeldOut	-2.35	0.72	-0.37	No score given	Fitted to the held-out data itself: an upper bound, not a fair competitor

The thin-plate baseline shows what the bar does: the single textbook relation “plate stiffness scales with t³” already brings the median error close to the threshold, but the worst point exceeds it and the baseline fails. The least-squares row uses the very answers the evaluation asks for, and the record states plainly that it is only an upper bound. Its exponents can summarise the trends in the held-out data, but it cannot be scored against the model.

Passing the bar does not mean every trend is right

The record’s own reading: the model got the thickness and width trends right in direction (t^-2.6; W^+1.2, larger than the fitted measurement of +0.72), but the height trend wrong in sign (+0.8 against a fitted -0.37). Here, taller plates with deeper ribs are stiffer at the axis. The record concludes: “It passes the bar, and the solver still decides.”

The model’s rationale mentions a 50 mm lever turning 60 N into about 3 N·m at the face, the fixed 31 mm bolt pattern, plate stiffness D scaling with t³ and more. These are the model’s reasoning about the bracket; this article has not checked those numbers, and the record scores only the exponents, not whether the rationale is correct. Keeping an explanation apart from checkable results follows the same idea as marking explanations not_assessed in Checking a model’s explanation against the original robot recording: what Robot Reel’s review-claims can establish.

A hand calculation with the exponents makes the effect of the height sign visible. This is a hypothetical calculation from the formula by this site, not a measurement: one parameter at a time is raised to 1.2 times its reference value, with the others unchanged. The record does not give the parameter range of the held-out points, so 1.2 times may lie outside it.

Only this parameter × 1.2	Deflection change from the model’s exponents	Change from the exponents fitted to held-out data
Thickness t	1.2^-2.6 ≈ 0.62, about -38 %	1.2^-2.35 ≈ 0.65, about -35 %
Width W	1.2^1.2 ≈ 1.24, about +24 %	1.2^0.72 ≈ 1.14, about +14 %
Plate height H	1.2^0.8 ≈ 1.16, about +16 %	1.2^-0.37 ≈ 0.93, about -7 %

The two are close for thickness, agree in direction but differ in size for width, and point in opposite directions for height. The three aggregate metrics alone do not reveal this reversed trend. The record gives no per-point errors, and this article does not speculate on why the aggregate metrics stayed within the thresholds. If such an estimate were used to choose a design direction, for example raising the plate to reduce deflection, the model would give the wrong advice. This is consistent with the flow in requirement F7, where the AI only supplies seeds with estimates and the solver’s measurements and a formal review decide in the end.

Keep the answers out of the agent’s reach

An evaluation is only worth something if the party being evaluated cannot see the answers. What the supplied material establishes:

• The prompt contains only the bracket description and one reference point; the held-out solver results are not in the prompt.
• For this call the script creates a new eval/<id> directory under the state directory with mode 0o700 and hands that to the executor, rather than passing the task workspace. The supplied material does not say whether the engine can read other files during execution.
• The model’s output is only three bounded numbers and a rationale; the executable estimate.mjs is generated by the script from a template.
• The script itself computes no score; according to its comment, the base eval harness invokes it in the task workspace.

From this, this site suggests (an editorial recommendation, not a rule in the project documentation): keep the held-out data and the scoring code outside the agent’s workspace and readable paths, and have an independent harness read them; let the agent hand back only bounded, checkable data, and generate anything that runs from a trusted template; allow one attempt per run and keep the status of every attempt, failures included. The least-squares row makes the same point: once the estimator can see the held-out data, its “score” is only a goodness of fit and says nothing about reasoning.

Limitations

• This is one saved development run with a single attempt; this site did not rerun or independently reproduce it. Another time, another engine version or another answer from the same model may give a different result.
• The model name is the engine’s self-report (acp_advertised), not independent proof of model provenance.
• The scoring code run.mjs, the held-out data and its number of points, the parameter ranges and the exact error definition are not in the files this article relies on; the reading of the threshold directions here rests on the wording of the requirements table.
• fixtureSolution is the task’s fixed solution, not a new model attempt; leastSquaresOnHeldOut is fitted to the held-out data itself and is only an upper bound.
• Passing this evaluation does not show that the model has general physical understanding or can extrapolate reliably, and it says nothing about physical safety. The requirements table lists “treating simulation as physical validation” as out of scope. The record calls the held-out data “measured”, but the supplied material does not say which solver or settings produced it; this article does not treat it as evidence from physical measurement.
• The executor’s “shared ledger” appears in the supplied material by name only, without a description of its contents; this article does not speculate about it.

Links

• Evaluation record: docs/evidence/physics-eval.json (commit 70330e2)
• Evaluation script: scripts/eval-physics-solver.ts (commit 70330e2)
• Requirements and acceptance status: docs/REQUIREMENTS.md (commit 70330e2)
• Related on this site: Carrying a pinned OpenFOAM case to AWS Batch: hashed results, two meshes and a drag comparison with the local run
• Related on this site: Checking a model’s explanation against the original robot recording: what Robot Reel’s review-claims can establish