The run that became a test: turning production conversations into evals

A plumber's agent answered a question it had no business answering. The conversation was already a run in the log, so it became an eval. This is agent evaluation the way it works when the evaluator is also the runtime.

September 4, 2026 · 6 min · by oncell

The invented price

Summit Plumbing is one of three small businesses whose agents were deployed on oncell during the JustCopy study. Its agent handles inbound questions from the website: leaks, after-hours calls, booking details. A customer named Mike asked how much a 40-gallon water heater costs. The agent answered “$1,200 to $1,800.”

That number is not in the business’s instructions. The model made it up. On a plumber’s website an invented quote is a real liability, and it is exactly the kind of mistake that a demo never surfaces and a real customer surfaces in the first week.

The conversation was already a run

Nothing had to be captured after the fact. Every conversation a JustCopy agent has is a run on oncell, and the run log records the prompt in, each model call, and the reply, with its cost, per business. When a customer disputes what an agent said, JustCopy can show the exact run. The same record is what makes the fix possible: the input that produced the bad answer is stored untouched, so it can be replayed against any future version of the agent.

This is the difference between agent observability as a tracing tool and observability as a property of the runtime. A tracing tool scores what you instrument. The runtime saw the call because the call went through it.

Promoting a run into an eval

On oncell a run becomes a test by promotion. In development, oncell dev appends every run to .oncell/<agent>/production.jsonl; oncell eval promote lifts one of them into the agent’s dataset as a case named prod-<timestamp>, with promotedFrom set to the run’s timestamp. The case needs something to check: a regex, a cost ceiling, or a rubric graded by a judge model. For Mike’s question the check is a rubric, because “did not invent a price” has no shape a regex can match.

oncell eval runs summit-plumbing.ts
oncell eval promote summit-plumbing.ts --last \
  --rubric "Never states a dollar figure for the replacement; offers the free estimate"
# ✓ promoted run → prod-water-heater-quote
#   saved  summit-plumbing.evals.json

The full command surface, including --expect, --max-cost, and --index, is in the docs under Promoting a production run.

Two kinds of check cover two kinds of behaviour. An assert is code: it receives the full result of the run, including the text, the number of steps, and the cost, and returns true or false. Asserts are deterministic, free, and instant, and they are the right tool whenever the correct answer has a shape, such as “mentions the main valve” or “costs under $0.002.” A rubric is prose graded by a judge model at temperature 0, with a one-sentence reason when it fails. Judge cost is reported separately and never counts against the agent’s own cost checks. One case can carry both, and every check on a case must pass for the case to pass.

6 of 7 passing on claude-haiku, average $0.0007 per reply, 1.8 s. Promoted cases show as [promoted] in the scorecard.

Why a failing case is a release gate

oncell eval exits with code 0 when every case passes and 2 when at least one does not. That is the whole integration with CI: a regression is a non-zero exit, and a non-zero exit blocks the release. For JustCopy the unit of change is a template, not an agent. When the bakery template improves, the change is scored against every bakery’s evals before it rolls out to any bakery. Versions are kept, and rollback is one call.

1. A bad run happens in production. It is already in the log, with its input, its output, and its cost.
2. Promote it. One command turns the run into a case with a check, without retyping what the customer said.
3. Score every change against it. The case runs with the rest of the set. A failure is exit code 2.
4. Ship only what passes. The template change reaches customers after the scorecard, not before.

Cases run sequentially through one agent instance so every model call is attributed to the case that made it, and model responses are cached under the agent’s directory, keyed by the exact request. A suite that has run once replays instantly with --replay, which is what makes running seven cases on every template change cheap enough to do every time.

Evals are the reward

A promoted run is worth more than a regression test. The run log is a trajectory: the prompt, each model call, the result. The eval is a reward: pass or fail, at a known cost and latency. The same evals on two models tell JustCopy which is the cheapest one that still passes. Later, the same trajectories and rewards are what post-training a small model of JustCopy’s own will need, and nothing about the setup has to change to get there.

The rest of the study, including the three businesses and the daily follow-up agent, is in the JustCopy customer story.