Handing off an EvalArc report folder: input copies, recorded options and the limits of evalarc verify

When EvalArc’s diff, eval-health, hillclimb-review and hillclimb-run are given --output, they write a self-contained report folder: a byte copy of every input file plus a params.json that records every option. Whoever receives the folder can run evalarc verify to redo the computation offline and confirm that the saved report really follows from those inputs and options. Based on the implementation, documentation and regression tests pinned at commit 8b134e2, this article covers what the folder stores, how verify recomputes it, which question each of the exit codes 0, 1 and 2 answers, and what the check cannot prove.

This site’s EvalArc: the score went up, so where did the agent regress? introduces the tool, and Reviewing a model upgrade: complete-plan matches rose from 15/24 to 19/24, so why does the EvalArc gate still reject it? shows how to recompute an upgrade verdict from published records. This article is about a different task: when you hand your own report to someone else, how they can check that it was not edited and still matches its inputs. Every behavior described below comes from the source and test files ([O], the project’s own code). Test assertions are the expectations written in the code; this site did not run the tests.

What the folder stores

According to the module docstring of evidence.py and docs/verification.md, the folder contains:

• Byte copies of the inputs: result files, split.json, cases.json, and the harness files that were scanned. Harness files are stored in order as harness/0000, harness/0001 and so on; for each one, params.json records the file name, its scan label and its SHA-256. If the harness copies add up to more than 16 MiB, writing fails with an error.
• params.json: schema evalarc.report-params.v1, with a kind (diff, eval-health or hillclimb-review) and every option of the run. Examples are max_cost_ratio, cost_metric, require_generalization and leak_min_chars for diff; saturation, min_effect, ordered and plan_attempts for eval-health; objective, stall_after, min_steps and labels for hillclimb-review.
• The saved report itself: diff.json, health.json or hillclimb.json.

While copying an input, the code checks that the bytes it read match the expected SHA-256 and otherwise reports that the file changed while it was being read. All paths in the folder are relative, so it can be moved or archived. The report records the original file names, so a copy still verifies after being renamed.

How verify recomputes

When evalarc verify receives a directory, or a diff.json, health.json or hillclimb.json inside one, it first looks for a recognizable params.json. If there is one, it treats the input as a report folder:

1. It checks the schema and kind in params.json and confirms that the report’s schema matches that kind.
2. It reads each result copy and compares its SHA-256 with the input identity recorded in the report; harness copies are compared with the entries in params.json. Absolute paths, paths containing .. and symbolic links are rejected.
3. It calls compute_diff, compute_health or compute_hillclimb with the recorded options. The module docstring states that the commands that write reports and verify call the same compute_* functions, so the two cannot drift apart.
4. It drops the created_at and evalarc_version fields at every level and compares the recomputed result with the saved report; any difference is a failure.

On success it prints a result with schema evalarc.verification.v1: verified, kind, passed, a summary for each report type (gate_passed and blocking_changes for diff, healthy and warnings for eval-health, recommendation and the per-step decisions for hillclimb-review), the SHA-256 and byte length of every file it read, and a scope statement.

Walking through the repository example

The regression test tests/test_report_verify.py builds three folders from the example data shipped in the repository. The test calls the command-line entry function main([...]) directly; the equivalent command lines are below, run from the root of the repository at the pinned commit:

git clone https://github.com/noteflowai/evalarc.git
cd evalarc
git checkout 8b134e2374333adb54ace0aa81f7068e1c3b6881

These files give no installation steps. Checking out the commit pins the example data and the source code, not the evalarc on your PATH. Confirm yourself that the evalarc you run comes from the same commit; otherwise the results may differ.

evalarc diff examples/results-diff/inspect/baseline.json \
  examples/results-diff/inspect/current.json \
  --held-out examples/results-diff/inspect/split.json \
  --harness examples/results-diff/inspect/harness \
  --max-cost-ratio 2 --cost-metric duration \
  --output out/diff

evalarc eval-health examples/results-diff/inspect/baseline.json \
  examples/results-diff/inspect/current.json \
  --cases examples/results-diff/inspect/cases.json \
  --min-effect 0.05 --ordered --plan-attempts 4 \
  --output out/health

# The test passes the JSON files in results/ that start with 0, in file-name order
evalarc hillclimb-review examples/hillclimb-review/results/0*.json \
  --held-out examples/hillclimb-review/split.json \
  --objective cost \
  --harness examples/results-diff/inspect/harness \
  --output out/climb

The test writes to a fresh temporary directory each time. It does not check the exit codes of these three commands; it only asserts the results of the verify calls that follow:

evalarc verify out/diff --json      # Test asserts: exit 0, kind=diff, passed=false
evalarc verify out/health --json    # Test asserts: exit 0, kind=eval-health, passed=false
evalarc verify out/climb --json     # Test asserts: exit 0, kind=hillclimb-review, passed=true

evalarc verify out/diff --json --require-accepted     # Test asserts: exit 1
evalarc verify out/health --json --require-accepted   # Test asserts: exit 1
evalarc verify out/climb --json --require-accepted    # Test asserts: exit 0

The tests also assert that params.json appears in files in all three results, that passing out/climb/hillclimb.json directly also returns 0, and that verify returns 0 on the output folder of the hillclimb-run example as well.

What exit codes 0, 1 and 2 answer

Exit code	Condition	Question answered
0 (default)	The recomputed result matches the saved report	Are the records consistent? Also 0 when a gate rejects or eval health reports warnings
1 (with --require-accepted)	The records match, but passed is false	Is the conclusion acceptable?
2	Inputs, options or report do not match, or the folder is invalid	The evidence itself is contradictory or incomplete; the JSON output has verified: false

verify.py sets both fully_resolved and accepted to the same passed value. The documentation says --require-resolved follows the same exit-code convention, but tests/test_report_verify.py, the test file used for this article, does not test its behavior on report folders.

passed means something different for each report type. For diff: the gate passed, and the generalization check passed if it was required, and the cost gate passed if a cost limit was set. For eval-health: healthy is true, which the documentation describes as having no warnings. For hillclimb-review: merge_recommended is true, that is, a merge is recommended. In the tests, the diff and eval-health folders verify with 0 by default and with 1 once --require-accepted is added. That is exactly the point: “the report was not edited” and “the report’s conclusion is a pass” are separate statements.

The tests also modify the folders one at a time and assert that verify notices:

Change	What the test does	Test assertion
Edited decision	Sets the decision of the second step in hillclimb.json to keep	Exit 2; the error contains “differs from the recomputed”
Edited input copy	Replaces the first "C" in the diff folder’s current.json with "I"	Exit 2; the error contains “differs from the input the report recorded”
Edited harness copy	Overwrites harness/0000 with clean prompt	Exit 2; the error contains “harness file that was scanned”
Changed option	Sets saturation to 0.5 in the eval-health params.json	Exit 2
Moved folder	Moves the climb folder elsewhere	Exit 0

The “Consistency, acceptance and resolution” section of docs/verification.md still says that --require-accepted applies only to suites, which is older wording. The section of the same document on report folders, the implementation in which verify maps passed to accepted, and the tests above all extend it to these three folder types.

Limitations

• The scope in verify’s output states the boundary: it reruns the computation from the preserved input bytes and recorded options and compares it with the saved report; it does not rerun the source tools’ grading, does not authenticate where the inputs came from, and does not check HTML or Markdown.
• The expected hashes of the result copies are recorded in the saved report, and those of the harness copies in params.json, both inside the same folder. Hash checks catch edits to the result and harness copies; changes to split, cases or options are detected only when they change the recomputed output. The check does not stop someone from rewriting the inputs, hashes and report together so that they agree. In the documentation’s words, a coordinated fabrication can be internally consistent; when execution evidence is required, rerun trusted grading.
• The comparison ignores evalarc_version, and the recomputation uses whatever implementation the verifier has installed. Whether producer and verifier ran the same version has to be confirmed separately; the preserved input bytes do not establish it.
• The exit codes and error messages above are test assertions, not results this site observed by running the tests. This article makes no claims about performance, real users or a complete security audit.
• This article is based on commit 8b134e2; later versions may differ in commands, fields or exit codes.

Links

• Writing and recomputing report folders: src/evalarc/evidence.py (commit 8b134e2)
• The verify entry point: src/evalarc/verify.py (commit 8b134e2)
• Regression tests: tests/test_report_verify.py (commit 8b134e2)
• Documentation: docs/verification.md (commit 8b134e2)
• Related articles on this site: EvalArc: the score went up, so where did the agent regress? · Reviewing a model upgrade: complete-plan matches rose from 15/24 to 19/24, so why does the EvalArc gate still reject it?