Running a hill-climbing loop with EvalArc: keep useful changes, roll back an overfitted proposal

Cycle diagram: on the left, an oval ring with four filled circles that contain only the numbers 1 to 4, and clockwise arrowheads between them leading 1, 2, 3, 4 and back to 1. On the right, a matching numbered list: 1 Write the last kept result's tuning failures; no held-out cases. 2 propose may edit only allowed files; outside: exit 2. 3 Patch checked first: case text rolls back; empty: no_change. 4 evaluate runs, compares with last kept result: keep or roll back. The title, One iteration of hillclimb-run, sits above a horizontal line, and the source note sits below a second horizontal line. — The four stages of each evalarc hillclimb-run iteration, drawn from docs/hillclimb-run.md at commit c3ed02f. A keep becomes the next iteration’s last kept result; the loop ends at max_iterations, when stall triage stops it as stopped_below_noise, or on exit 2. These stop conditions are not shown. Step 5 in the article, stall triage after stall_after iterations without a keep, is not drawn as a stage of its own.

Letting a model revise a prompt or rule file and rerunning the evaluation after every edit is a form of hill climbing, and it drifts easily: pasting a failing case’s answer into the file, or fixing just one tuning case, will both raise the score. EvalArc’s evalarc hillclimb-run drives this loop for you. You supply two commands: one evaluates the workspace and writes an Inspect AI, promptfoo or JUnit result; the other proposes a change, typically by calling a model with your own key. EvalArc never calls a model itself. It decides what each command may see and change, and whether to keep each change. Based on the documentation and the scripted example pinned at commit c3ed02f, this article walks through the example’s five iterations to explain each step’s checks, outputs and exit codes, and what the allow guard cannot protect against.

This site’s EvalArc: the score went up, so where did the agent regress? covered comparing two evaluation runs check by check, and Handing off an EvalArc report folder: input copies, recorded options and the limits of evalarc verify showed how to recheck reports such as hillclimb-review offline. In the documentation’s terms, hillclimb-review audits a hill-climbing loop after the fact, while hillclimb-run drives that loop as it happens; this article is about the latter. All behaviour described below comes from the project’s documentation ([O], the project’s own account). This site did not run these commands; the example output shown is the result saved in the documentation.

Two commands and one configuration file

The example’s configuration file, hillclimb.toml, reads:

workspace = "workspace"            # relative to this file
allow = ["rules.md"]               # globs the propose command may edit
held_out = "split.json"            # evalarc.case-split.v1
evaluate = ["{python}", "evaluate.py", "{output}"]
propose = ["{python}", "propose.py", "{iteration}", "{failures}"]
objective = "quality"              # or "cost"
max_iterations = 5
stall_after = 2
timeout_seconds = 60

Both commands are argument arrays run without a shell, in the workspace, and they inherit your environment. The placeholders are {python} and {workspace}; {output} for evaluate, the result path it should write; and {failures}, {allowed} and {iteration} for propose. allow lists the file globs the propose command may edit, and held_out points to a case-split file with schema evalarc.case-split.v1. The documentation also lists the keys cost_metric, max_cost_ratio, min_effect, result_suffix (.json, .xml, .eval), format, threshold, leak_min_chars (listed with 12) and paste_check (listed with true), without explaining each one; this article does not go into them.

Five gates in every iteration

Only tuning failures are handed over. EvalArc writes steps/NN/tuning-failures.json: the failing tuning cases of the last kept result, with inputs, expected answers, outputs, grader evidence and a triage category. Held-out cases are never included.
propose may change only files inside allow. If it changes any workspace file outside allow (including .git; only __pycache__ and tool caches are ignored), the loop restores the allowed files, deletes files the command created, and stops with exit code 2, naming any pre-existing files it modified so you can restore them yourself. If the propose command itself fails, the step is recorded as propose_failed. Allowed paths that were turned into symlinks are replaced with their kept content.
The patch is checked before any evaluation. The change to allowed files is saved as patch.diff. If its added lines contain held-out case text, it is rolled back as rollback_leakage; if they contain tuning case text, it is rolled back as rollback_pasted_case, meaning the failing cases were pasted in instead of the cause being fixed. Only recorded strings of at least leak_min_chars characters are checked, so a short answer such as billing is not caught. An empty patch is recorded as no_change.
Evaluate and compare with the last kept result. Only after the first three gates does evaluate run. The result is then classified under the hillclimb-review rules as keep, rollback_regression, rollback_overfit or rollback_no_gain. Rolled-back files are restored.
Stall triage. After stall_after consecutive iterations without a keep, the remaining tuning failures are triaged; if the tuning headroom is below the resolvable change, the loop stops as stopped_below_noise.

Gates 1 and 3 work together. The propose command can see the expected answers of tuning cases, so the cheapest “fix” is to copy those answers into the rule file; gate 3 rolls such a patch back before any evaluation cost is spent. Held-out cases never enter the failure list and may not appear in a patch, which is why their movement can show whether a change merely fitted the tuning set.

The example: where each of the five iterations is decided

The repository’s example is a ticket-routing workspace. Its “agent”, workspace/evaluate.py, routes a ticket correctly when rules.md has a rule for it; workspace/propose.py stands in for a model and appends one scripted edit per iteration from workspace/proposals.json. No model is called. The following three lines are this site’s pinning steps; they do not come from either file:

git clone https://github.com/noteflowai/evalarc.git
cd evalarc
git checkout c3ed02f7fb2645c1b8651f16c11a47176c230736

The documentation’s own commands start at cp -r examples/hillclimb-run /tmp/climb-example; the loop edits the workspace, so it is copied first:

cp -r examples/hillclimb-run /tmp/climb-example
evalarc hillclimb-run /tmp/climb-example/hillclimb.toml --output runs/climb --trust-local

Neither file gives installation steps. Checking out the pinned commit fixes the example and the source code, not the evalarc on your PATH; confirm yourself that the version you run comes from the same commit. The output saved in the documentation is:

  1. keep (tuning +25.0 pp, held out +33.3 pp)
  2. rollback_pasted_case
  3. rollback_overfit (tuning +12.5 pp, held out +0.0 pp) — stalled
  4. no_change
  5. keep (tuning +25.0 pp, held out +50.0 pp)
Stopped: max_iterations | Kept: 05.json | Recommendation: merge

Iteration	Proposed edit	Outcome	Decided at
1	`route refund -> billing`	`keep`: tuning and held-out both improve	Gate 4
2	A failing tuning ticket pasted in with its answer	`rollback_pasted_case`, not evaluated	Gate 3
3	`case t06 -> accounts`, which fits one tuning case	`rollback_overfit`, then stall triage	Gate 4, then gate 5
4	No edit	`no_change`	Gate 3
5	`route outage -> oncall`	`keep`	Gate 4

Iteration 3 is the one to study: tuning rose by 12.5 percentage points, held-out by 0, and the change was rolled back. Iterations 2 and 3 were two consecutive non-keeps, reaching stall_after = 2, and the output marks the step as stalled; the loop then continued to iteration 5 and stopped on max_iterations, not stopped_below_noise. The kept result is 05.json, with a merge recommendation.

The example README adds a follow-up: package the baseline and the final result as a pairwise comparison and score it with a scripted judge. judge.py only compares each output with the expected queue; the README notes that a real judge command would call a model different from the one under evaluation.

evalarc judge-packet pairwise runs/climb/results/00-baseline.json runs/climb/results/05.json \
  --output runs/ab
evalarc judge-run runs/ab --config /tmp/climb-example/judge.toml --output runs/ab-verdicts.json --trust-local
evalarc judge-score runs/ab runs/ab-verdicts.json --require-current-preferred

What to look at when the run ends

The workspace holds the last kept version, original/ holds the starting files, and final.diff holds the net change.
hillclimb.json, summary.md and index.html are the hillclimb-review report over every evaluated result.
loop.json records every step, including on failure.
Exit codes: 0 means merge recommended, 1 not recommended, 2 a configuration, command or safety failure.

Exit code 0 is only the loop’s merge recommendation. The documentation advises reviewing final.diff before merging; the rolled-back steps are also recorded in loop.json, and this site suggests reviewing them alongside the kept ones.

The allow guard is not a sandbox

Both commands run on the host with your privileges and environment, which is why --trust-local is required. The allow guard compares the workspace before and after each propose command; it does not isolate the commands, so it cannot see changes outside the workspace or undo edits to files that already existed. The documentation advises running the loop on a copy or a clean git checkout, keeping held-out data out of the workspace, and reviewing final.diff before merging.

This site adds two points of its own; they are not in the documentation. First, both commands inherit your environment, so any cloud credentials in it are visible to the evaluate script as well; for model calls, use short-lived credentials scoped to the model invocation you need, and do not resolve a permission error by widening permissions. Second, files in the workspace may be read by the propose command and sent to a model along with its request, so keep keys and other sensitive files outside the workspace.

Limitations

The example is scripted: propose.py replays five declared edits, and the percentage points in the output are not a measured model improvement. This site did not run the example and has not tried the loop with a real model.
The documentation does not say whether the percentage points are relative to the last kept result or to the baseline, and this page does not give the thresholds behind decisions such as rollback_overfit or rollback_no_gain; it only says the hillclimb-review rules apply. This article makes no inference about either.
The leakage and paste checks look at whether added lines contain recorded case text, and only for strings of at least leak_min_chars characters. The documentation does not say they recognise a reworded case, so they should not be treated as complete leakage detection.
This article is based on commit c3ed02f; later versions may differ in configuration keys, decision names or exit codes.