What Should an Agent Evaluation Report So Someone Else Can Use It?

A table can say:

Agent A — 74%

and still leave most of the evaluation unrecoverable. Was this the base model or a particular scaffold? Which task version? Which tools and permissions? How were crashes and retries handled? What did the grader inspect? How many runs were performed? Which target workflow is the number being used to discuss? These details are often spread across a paper, repository, appendix, benchmark implementation, and private evaluation script. A reusable evaluation record should bring the decision-relevant parts together.

Identify the evaluated system

For an agent, the model name alone is usually incomplete. Record the model version together with the scaffold, prompts or policies that materially define behavior, available tools, permissions, environment, and date. If a tool, model, or scaffold changes, the identity of the evaluated system may have changed as well. This is the system side of the record. The evaluation side should be identified separately: benchmark or task-set version, environment, grader, tests, parser, run budget, retry policy, and other conditions that determine how behavior becomes a result.

State the task frame and run policy

A reader needs to know what work the result is intended to describe, which tasks could enter the evaluation, and how the evaluated sample was selected. For example, tasks drawn from publicly available repositories may represent only part of the work done in a company's internal codebase. Reporting the source and selection process lets the reader assess this limit.

The run policy also belongs in the record. Agent runs are interrupted by timeouts, budget limits, tool failures, harness crashes, human interventions, unanswered clarification requests, and partial completion. Clinical trial methodology treats analogous intercurrent events as part of the estimand because different handling rules change the reported result [1]. For agent evaluation, report in advance how each material event is counted, retried, excluded, or otherwise handled.

Preserve the evidence behind the score

The report should make it possible to inspect how observed work became the reported number. Useful fields include the number of tasks and runs, the artifacts or trajectories retained, the grader and reference standard, uncertainty estimates, repeated-run behavior, sensitivity to prompts or other evaluation choices, and material failure categories. A final-answer score can be useful, but decision-facing evaluation often requires the work product or trajectory that explains the score. This is especially important when protocol exploitation, grader error, or downstream reviewability is plausible. The evidence record does not require every artifact to be public. Private evaluations can still state what was retained, who reviewed it, and what evidence supports the reported conclusion.

Make the limits visible

The report should state which conclusion the result is intended to support and which assumptions remain untested. Argument-based validity provides a way to examine these inferences [2,3], and AIMS’s Measurement to Meaning applies this question directly to AI evaluation [8]. Our note on interpreting evaluation results discusses this in more detail.

For reporting, the practical requirement is simple: connect the conclusion to evidence a reader can inspect. If the evaluation covers one fixed configuration, say so. If its relevance to another workflow has not been tested, describe the missing comparison. The next person should be able to understand the result without reconstructing the entire experiment from scattered files.

References

1. International Council for Harmonisation. ICH E9(R1): Addendum on Estimands and Sensitivity Analysis in Clinical Trials. 2019.
2. Kane MT. Validating the Interpretations and Uses of Test Scores. Journal of Educational Measurement. 2013;50(1):1–73.
3. Cook DA, Brydges R, Ginsburg S, Hatala R. A contemporary approach to validity arguments: a practical guide to Kane's framework. Medical Education. 2015;49(6):560–575.
4. Bossuyt PM, Reitsma JB, Bruns DE, et al. STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies. BMJ. 2015;351:h5527.
5. Guyatt GH, Oxman AD, Vist GE, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008;336:924–926.
6. Hua Y, Na H, Ayubcha C, Lian L. Design and Report Benchmarks for Knowledge Work. arXiv:2605.23262. 2026.
7. Hua Y, Na H, Zhou Y, Kalose A, Ayubcha C, Lian L. StagedWorkspace: A Versioned Workspace for Knowledge-Work Agents. arXiv:2608.18050. 2026.
8. Salaudeen O, Reuel A, Ahmed A, et al. Measurement to Meaning: A Validity-Centered Framework for AI Evaluation. 2025.