Evaluation Science Brief
How Close Is an Agent Evaluation to the Real Workflow?
Suppose an agent is evaluated on 300 document-analysis tasks. It receives a fixed document corpus and a fully specified question. Success is a correct final answer, and the agent achieves 82%. The intended use is an internal research workflow. There, the agent has to identify relevant sources, determine what information is missing, use company tools, ask staff for clarification when necessary, preserve sources and calculations, and leave a work product that another person can review. The 82% result may accurately describe the benchmark. The remaining issue is whether the conditions that produced it correspond to the workflow where the agent will operate.
Compare the evaluation with the workflow directly
Clinical research distinguishes efficacy under controlled study conditions from effectiveness in routine practice. PRECIS-2 was developed to make differences between controlled and routine conditions explicit [1]. Agent evaluation has a similar problem. The comparison can be organized across the following dimensions:
| Evaluation feature | Question |
|---|---|
| Tasks | Where did the tasks come from, and how closely do they match incoming work? |
| Task selection | Were tasks selected for representativeness, or because they were available and easy to evaluate? |
| Environment | How similar is the evaluation environment to the operating environment? |
| Tools and support | Are tools, permissions, integrations, human assistance, and information access comparable? |
| Agent instructions | Is the evaluation scaffold more prescriptive than routine use? |
| Interaction rules | Are retries, clarification, escalation, and human intervention handled similarly? |
| Follow-through | Is success measured at submission, or after downstream consequences become visible? |
| Outcome | Does the benchmark measure the outcome the operator actually cares about, or a proxy? |
| Analysis | Are crashes, timeouts, malformed outputs, and incomplete runs included? |
In the document-analysis example, several differences are immediate. Tasks: routine work begins before the information need has been fully specified. Environment: production uses internal systems and permissions absent from the benchmark. Interaction: the real workflow permits clarification and escalation. Outcome: the benchmark checks the final answer; the workflow requires a reviewable analysis package. Follow-through: the benchmark ends at submission; routine success depends on whether another person can review and use the work. These differences can be tested directly.
Task availability is not representativeness
Many benchmarks are assembled from tasks that can be obtained, specified, and graded reliably. That is reasonable for benchmark construction, but availability does not establish that the resulting tasks represent incoming work. An evaluation should distinguish: Target task population: the work that occurs in the target workflow. Evaluation task frame: the tasks that could actually enter the evaluation. Selection process: how tasks were sampled or selected from that frame. If important parts of the workflow cannot enter the evaluation, repeated runs on the available tasks do not address that gap.
The measured outcome may be a surrogate
Agent benchmarks usually measure something observable: a test passes, an answer matches, a rubric score improves. The workflow may care about something else: useful work completed, analyst time saved, errors avoided, safe actions taken, or decisions improved. Clinical research treats a surrogate endpoint as requiring evidence before it substitutes for the outcome of interest [2]. For an agent evaluation, record: Measured outcome: what the benchmark directly checks. Target outcome: what matters in the workflow. Evidence connecting them: what shows that improvement in the measured outcome predicts improvement in the target outcome. If the last field is empty, the relationship has not been tested.
Human review is part of the workflow
Many deployments assume that a human reviewer will catch remaining errors. That changes the system being studied. A benchmark measuring an autonomous agent does not measure an agent-plus-reviewer workflow unless the review process is also represented. Reviewer access to evidence, workload, error rate, and ability to recognize failures can all affect the final outcome. If human review is part of routine use, it belongs in the workflow comparison.
Benchmark-to-Workflow Profile
Record each dimension as: Matched: the evaluation closely reproduces the relevant workflow condition. Partially matched: the difference is known and there is some evidence about its effect. Unmatched: the workflow condition is absent or materially different. Unknown: the relationship has not been evaluated. For example:
| Dimension | Status |
|---|---|
| Tasks | Partially matched |
| Environment | Unmatched |
| Tools and permissions | Unmatched |
| Agent instructions | Matched |
| Interaction rules | Unknown |
| Follow-through | Unmatched |
| Outcome | Partially matched |
| Analysis of failures | Matched |
The profile should not be collapsed into one score. It shows where the evaluation reproduces the workflow and where further validation is needed.
References
- Loudon K, Treweek S, Sullivan F, Donnan P, Thorpe KE, Zwarenstein M. The PRECIS-2 tool: designing trials that are fit for purpose. BMJ. 2015;350:h2147.
- Prentice RL. Surrogate endpoints in clinical trials: definition and operational criteria. Statistics in Medicine. 1989;8(4):431–440.
The benchmark-to-workflow mapping in this note is an AES working adaptation. It has not been validated as a formal instrument.