Evaluation Science Brief
Is This Agent Result Reproducible?
Suppose the same agent scores 74% today, 69% tomorrow, 81% under another prompt, and 62% when another team implements the evaluation. Which result should another evaluator expect to reproduce? That depends on which conditions are supposed to remain fixed. Repeating one frozen setup asks a different question from testing stability across prompts, infrastructure, implementations, or time.
Repeatability and reproducibility are different
Analytical procedure validation distinguishes several levels of precision [1]. The corresponding levels in agent evaluation are: Repeatability: same system, same scaffold, same environment, short interval. Intermediate precision: same evaluation process, but vary conditions that can change within one organization, such as seed, day, operator, or infrastructure instance. Reproducibility: independent implementation, laboratory, or evaluation environment. Repeated runs of one frozen configuration provide repeatability evidence. They do not establish reproducibility.
Seeds are one source of variation
Agent evaluations have several plausible variance components:
| Factor | Why it may matter |
|---|---|
| Task sample | Different sampled tasks can change the estimated mean |
| Seed / stochastic run | The same task may produce different trajectories |
| Prompt template | Semantically similar instructions can change measured performance and rankings [2] |
| Scaffold | Orchestration and context management change the evaluated system |
| Tool version | APIs, search results, dependencies, or execution behavior can change |
| Environment state | Available files, caches, permissions, or external state can differ |
| Model version | The system itself may change over time |
| Retry policy | Re-running failures can change the reported quantity |
Running one configuration five times estimates only part of this variation.
Treat evaluation choices as experimental factors
List the evaluation choices that could plausibly change the result, then vary the important ones deliberately. For example:
Prompt: A / B
Tool version: 1 / 2
Retry policy: off / onEight planned combinations can provide more information about configuration robustness than eight repetitions of one setup. The design should distinguish factors that define the evaluated system from conditions under which the result is expected to remain stable.
Reproduce the task sample as well as the run
A second team can reproduce the execution protocol and still obtain a different result because it evaluated a different set of tasks. Task sampling and run-to-run stochasticity answer different questions. Repeating the same 100 tasks estimates variation conditional on those tasks. Drawing another 100 tasks tests stability across the task frame. The evaluation report should therefore distinguish the task population, task frame, and realized sample. When the benchmark is a fixed finite set, report that directly. When the evaluation is intended to describe a broader class of work, task-sampling variation belongs in the reproducibility analysis. The implementation boundary should also be explicit. A team that reuses the original harness tests a different source of variation from a team that independently implements the published specification.
Report results across defensible specifications
Prompt and harness choices create another source of variation. Several configurations may be defensible, and selecting one after seeing the results can make a fragile result look stable. Multiverse analysis and specification-curve analysis were developed to show how conclusions vary across defensible analytic choices [3,4]. The same idea can be applied to prompts, scaffolds, tool sets, parsers, or retry policies. An evaluation can therefore report:
Performance ranged from 63–78% across five pre-specified defensible configurations.
The range shows how much the result depends on implementation choices that were considered acceptable in advance.
Preserve version identity
Agent systems and their dependencies change quickly. A reproducibility package should record the model version, scaffold commit or release, benchmark version, tool and API versions, grader version, date, and run policy. Version stability can also be tested directly by rerunning a reference subset after a model, tool, grader, or benchmark change and comparing the result with a pre-specified tolerance.
Reproducibility Profile
| Level | Evidence to record |
|---|---|
| Repeatability | Same configuration, repeated runs |
| Intermediate precision | Variation across seeds, days, operators, or infrastructure |
| Configuration robustness | Prompts, scaffold, tools, policies, parsers |
| Reproducibility | Independent implementation, lab, or evaluation environment |
| Version stability | Repeated after system or dependency updates |
A compact result can read:
Reproducibility status: Repeatability established; configuration robustness partially tested; independent reproducibility unknown.
The status identifies which forms of stability have actually been tested.
References
- International Council for Harmonisation. ICH Q2(R2): Validation of Analytical Procedures. 2023.
- Mizrahi M, Kaplan G, Malkin D, et al. State of What Art? A Call for Multi-Prompt LLM Evaluation. Transactions of the Association for Computational Linguistics. 2024;12:933–949.
- Steegen S, Tuerlinckx F, Gelman A, Vanpaemel W. Increasing Transparency Through a Multiverse Analysis. Perspectives on Psychological Science. 2016;11(5):702–712.
- Simonsohn U, Simmons JP, Nelson LD. Specification curve analysis. Nature Human Behaviour. 2020;4:1208–1214.