The question.
An agent can look better in a demo while becoming less reliable on unusual tools, long tasks, or small changes to the prompt.
The workbench treats traces, datasets, human judgments, latency, and cost as a connected evaluation record rather than unrelated dashboards.
The proposed approach.
- Create versioned tasks with explicit success criteria
- Replay candidate configurations against the same inputs
- Combine deterministic checks with calibrated model-assisted and human review
- Inspect failure clusters before aggregate score changes
Target architecture.
Versioned tasksCandidate runTrace storeEvaluatorsHuman calibrationRegression report
The difficult parts.
- Judge bias and unstable scoring rubrics
- Overfitting to a visible evaluation set
- Comparing runs with different tool and data snapshots
How to evaluate it.
- Agreement between automated scores and human labels
- Detection of seeded regressions
- Quality–latency–cost trade-offs across configurations
Planned deliverables.
- A versioned task and trace format
- An evaluator calibration workflow
- A release-comparison reporting design
Design principles.
An aggregate score should lead to inspectable examples.
Evaluate the complete task, not only a fluent final answer.