Reviewing a model upgrade: complete-plan matches rose from 15/24 to 19/24, so why does the EvalArc gate still reject it?

A model configuration upgrade raised complete-plan matches from 15/24 to 19/24, yet the upgrade gate rejected it: five checks regressed and five became less reliable. The EvalArc guide “Your first model-upgrade review” turns that decision into a review path anyone can reproduce: install a pinned release of the tool, download a pinned recording, and recompute the verdict yourself, with no repository checkout, GPU or model API. This article follows that guide, pinned at commit 6af26bf, and covers how to run the review, what each exit code and report field tells you, and where the conclusion stops.

This site’s EvalArc: the score went up, so where did the agent regress? already introduced the tool and the outcome of this comparison. This article skips the introduction and covers only the review steps and how to read the results. All figures come from the project’s own documentation and its preserved records ([O], a development example reported by the authors). None of this is an independent third-party evaluation.

What the upgrade compared

The baseline is Qwen/Qwen3-8B (BF16) and the current configuration is Qwen/Qwen3.8-27B-FP8. For each of eight public support-planning cases, three fresh answers were generated per configuration, and all 48 answers are kept. Plans are compared but not executed. Each answer gets five pytest checks: JSON shape, routing calls, note contents and retry keys, closure, and the complete ordered plan. That gives 5 × 24 = 120 check outcomes per side.

Paired by case and check, there are 40 items: 10 improved, 20 were unchanged, 5 regressed and 5 became less reliable. Those last 10 blocked the upgrade. They trace back to format or schema problems in just two cases, not to 10 independent causes:

• already-closed: all five checks fall from 3/3 to 0/3. At seed 17, for example, the baseline returns {"actions":[]} and the current model returns {}, leaving out the required actions key.
• resolved-bug: all five checks fall from 3/3 to 1/3 because two answers wrap the JSON in Markdown fences.

The checks are not independent: once the JSON is invalid, every check for that answer fails. The number of blocking checks is therefore not a measure of severity. Go back to the original answers to see what actually went wrong.

Reproducing the decision without a GPU

You need Python 3.11+ and a new directory. The guide’s commands use Bash on Linux/macOS. First install the released package:

mkdir evalarc-model-first-review && cd evalarc-model-first-review
python3 -m venv .venv
.venv/bin/python -m pip install evalarc==0.15.0

Next, download the v0.15.0 release asset evalarc-model-upgrade.zip. The guide’s download script checks the SHA-256 first, extracts only on a match and keeps the original file. On a mismatch it exits without extracting. The expected value is 6b1b9212d5b25da281c54f192489298af7a0b0f4524f807ea6e2821bfa6f515c, which also appears in the release’s SHA256SUMS. After extraction, open recorded-review/index.html to browse all 48 original answers offline.

.venv/bin/python - <<'PY'
import hashlib
import io
from pathlib import Path
from urllib.request import urlopen
from zipfile import ZipFile

url = "https://github.com/noteflowai/evalarc/releases/download/v0.15.0/evalarc-model-upgrade.zip"
expected = "6b1b9212d5b25da281c54f192489298af7a0b0f4524f807ea6e2821bfa6f515c"
with urlopen(url, timeout=60) as response:
    data = response.read()
if hashlib.sha256(data).hexdigest() != expected:
    raise SystemExit("Archive checksum differs; do not extract it.")
Path("recorded-review").mkdir()
with Path("review.zip").open("xb") as output:
    output.write(data)
with ZipFile(io.BytesIO(data)) as archive:
    archive.extractall("recorded-review")
PY

Then recompute the upgrade decision:

.venv/bin/evalarc diff recorded-review/reports/baseline.xml \
  recorded-review/reports/current.xml --output my-review

Read the result in my-review/index.html or my-review/diff.json. In the recorded browser review, select already-closed with seed 17 to see the pair of answers described above. This step only compares the preserved native test results (JUnit XML). It neither regenerates answers nor reruns the checks.

What the exit codes and gate_passed mean

The guide has two exit codes that are easy to misread:

Command	What it does	Expected exit code	Meaning
evalarc diff	Compares the preserved JUnit results	1	The comparison completed and the gate rejected the upgrade
regrade.py	Reruns the five checks on every preserved answer, writes fresh JUnit files and recomputes the comparison	0	The script finished; regraded/comparison.json still says "gate_passed": false

So exit code 1 from diff is a verdict, not a crash. Exit code 0 from regrade.py only means the script finished, not that the upgrade passed. This introductory guide documents only the expected exit codes 1 and 0; the gate rules and input formats are in examples/results-diff/README.md. After a regrade, read the gate_passed field instead of just checking whether the process reported an error.

The optional regrade needs the pytest version used for the recording, and a new output directory each time:

.venv/bin/python -m pip install pytest==8.4.2
.venv/bin/python recorded-review/regrade.py --output regraded

This step also calls no model. The grader redacts only two kinds of metadata: hostnames in JUnit and absolute report directories in stdout. Hashes before and after redaction are recorded. Model answers and check outcomes are never changed.

Applying it to your own upgrade review

For “bring your own change”, the guide asks for only a few things: keep case IDs, check names and the evaluated contract aligned between the two runs; retain every attempt; and pass the native results to evalarc diff. Input formats and gate rules are in the repository’s examples/results-diff/README.md. The experiment guide for this example adds a few practices worth copying:

• Fix the protocol before you have results. protocol.json was committed before any completed output, and the expected plans are kept independently of the generated answers.
• Give both sides the same conditions. Both models get the same policy, case text, sampling settings and seeds: seeds 17, 29 and 43, temperature 0.6, top-p 0.9, top-k 20, thinking disabled and a 512-token output cap. The recorded hardware is an NVIDIA L40S.
• Record every correction made at load time. Transformers 5.17.0’s regex-prefix matching mistakes the checkpoint’s nonexistent dense mlp.gate router exclusions for exclusions of gate_proj, and so discards its scale tensors. The FP8 recorder removes those router entries in memory and requires every checkpoint key to be accounted for. The exact adjustment and the loading report are in the identity file. Initialization failures that produced no completed answer are listed in runtime-failures.json.
• Keep the failed answers too. Every answer behind the 10 blocking checks is preserved in the recording, uncleaned.

Limitations

• This is a small development example, not a model leaderboard. The two configurations differ in architecture, parameter count and quantization, so the difference cannot be attributed to any single factor or used to estimate overall model capability.
• The cases are selected public development cases and may have appeared in training data. No decontamination was done, and no population accuracy is claimed.
• Full plan equality is a narrow contract check. It does not prove a real support task was completed, and the plans were not executed.
• The three generations per case are separate sampling calls, not three gradings of one answer. The authors do not present them as a general Pass^k estimate.
• The default review only shows that the verdict can be computed from the preserved results. regrade.py only reruns the checks on the preserved answers. Neither is an independent reproduction that regenerates answers. Different hardware, kernels or library versions may change sampled outputs. The recorded timings are diagnostic observations, not a throughput comparison.
• This article is based on the documentation at commit 6af26bf and evalarc==0.15.0. Later versions may differ in commands or output.

Links

• Review steps: docs/first-model-review.md (commit 6af26bf)
• Experiment protocol and reproduction notes: examples/model-upgrade/README.md (commit 6af26bf)
• Interactive comparison: noteflowai.github.io/evalarc/model-upgrade
• Related on this site: EvalArc: the score went up, so where did the agent regress? · Frontier Radar deep dive | Simulation and evaluation: what VLA evaluation is missing beyond initial-state success rate