EvalArc: the score went up, so where did the agent regress?

Compares two agent eval runs check by check under explicit acceptance rules to catch regressions hidden by a higher average, and audits graders on known bugs.

EvalArc is a Python command-line tool for reviewing agent evaluation results. Instead of looking at the overall score, it compares two runs case by case and check by check, decides pass or fail against acceptance rules written down in advance, and produces a report that others can recompute from the raw records. It targets a specific situation: the average score goes up while a check that used to pass starts failing. If you look only at the overall score, regressions like this slip through.

A concrete example

The demo that ships with the repository is a simulated customer-support ticket task (support-routing). The new version’s score rises from 90% to 93.75%: two checks improve, and one check that used to pass now fails. The cause is buried in the tool-call records: one tool call had in fact already written a note but returned an error; the agent retried with a new idempotency key, and the same note was written twice. This case scores 0.9375, but the strict note rule fails it on acceptance. EvalArc’s trajectory view lays out the error, the retry, and the resulting state change one by one.

Scoring is based only on the final ticket state on the host; a candidate program’s own claim of “success” does not count. A case counts as solved only if all of its checks pass.

Using it on your existing evaluation results

evalarc diff reads two Inspect AI logs, promptfoo --output files, or JUnit XML files, and compares each check of each case in pairs. Whenever a check that passed on the baseline starts failing, has a lower pass rate, is skipped, or disappears, the command exits with code 1 and writes an offline report that includes copies of both inputs. In the recorded Inspect example in the repository, accuracy rises from 0.625 to 0.8125 while 3 checks lose their passing status. A GitHub Action is also provided so you can block a PR directly:

- uses: noteflowai/evalarc@v0.16.0 # or a full commit SHA
  with:
    baseline: evals/baseline.json
    current: results/current.json

The repository also publishes a record of a real model configuration change: Qwen/Qwen3-8B (BF16) versus Qwen/Qwen3.8-27B-FP8, on 8 public customer-support planning cases, with 3 generations per configuration per case and all 48 raw responses kept, scored with native pytest checks (recorded on an NVIDIA L40S). Full-plan matches rose from 15/24 to 19/24, but 5 checks regressed and 5 became unstable, so the upgrade gate failed. The README also explains that these 10 blocking checks share two kinds of formatting problems: responses for already-closed omitted the actions field, and resolved-bug twice wrapped its JSON in a Markdown code block. So they are not 10 independent causes of failure. The author also states that this is only a small development example, not a model leaderboard: the two models differ in architecture, parameter count, and quantization, and no decontamination was done.

Auditing the grader itself

The other half of EvalArc’s job is testing whether a grader can catch known defects: you prepare a reference implementation and several versions with deliberately planted bugs, then check whether each bug is detected in the expected dimension. Three task packs are built in:

TaskContentDeclared defectsDetected by only one case
durable-kvCoding artifact: responses, transactions, persistence across restarts83
support-routingSimulated ticket tools: routing, notes, closing, unrelated state72
robot-evidence-reviewRecordings with provenance: coordinates, clocks, missing observations61

The saved audit detected all 21/21 declared defects; 6 of them were detected by only one case, and removing that case would miss them. The coding task has 15 cases per seed across six weighted dimensions (basic behavior 25%, validation 15%, transactions 20%, CAS 15%, persistence 15%, crash recovery 10%). The “boolean compared equal to number” defect earns a partial score of 0.925 but does not count as fully solved: partial scores and acceptance are computed separately. Expected values are computed outside the candidate container, and the grader never trusts a reward reported by the candidate program.

How to try it

The online demo replays saved Docker runs and needs no installation, account, or model key: noteflowai.github.io/evalarc. To install locally (Python 3.11+):

python3 -m venv .venv
. .venv/bin/activate
python -m pip install evalarc==0.16.0
evalarc --version

Offline review needs no Docker, Node, GPU, or model API; the steps are in docs/first-review.md. The consistency-check command exits with code 0, and the comparison command exits with code 1 because a regression is present. Actually running the reference implementation and the 8 coding defects requires Docker:

docker pull python:3.12-slim
evalarc audit --seeds 17 41 97 --output runs/audit

Exit code 0 means the reference implementation passed and every defect was detected in its expected dimension; 1 means the audit or a candidate failed; 2 means invalid input or an environment failure. To evaluate your own coding agent, generate a workspace with evalarc init workspace/durable-kv, give TASK.md to the agent, and run evalarc evaluate once it has made its changes.

Limitations

  • The status is Research preview. The current version is 0.16.0 (2026-09-28), MIT-licensed. The local workflow supports Linux only, and the Python core has no third-party runtime dependencies.
  • Detection results hold only for the declared defects, the selected cases, and the recorded execution conditions. They say nothing about coverage of unknown defects or about resistance to reward hacking. Changing only the random seed is not an independent held-out evaluation.
  • EvalArc itself does not call any LLM and does not configure model credentials. Trajectory import accepts only specific export formats, not arbitrary cloud exports; the README states explicitly that no online AgentCore evaluation has been done.
  • --backend local --trust-local runs candidate programs with the current user’s permissions; use Docker to isolate untrusted candidates.
  • The small published GPU experiments are mostly negative or inconclusive. For example, 36 attempts on three public SWE tasks produced no native acceptance at all, and 5 of them were recorded as inconclusive because of upstream infrastructure flags. The author keeps them as development experiments and claims no effect.

Links

Related Posts