Agents are systems
Evaluation concerns the model together with tools, memory, scaffolds, interfaces, permissions, and other components.
Follow AES events About AES
Building the science and community of agent evaluation
Agent Evaluation Science (AES) is an independent nonprofit scientific network bringing together researchers and practitioners working on how AI agents should be evaluated.
IWhy we exist+
Most AI evaluation has been developed around models. Agents change the object being evaluated.
As agents take on longer and more realistic work, evaluation becomes a systems problem. Results depend on tools, environments, information access, trajectories, graders, and the way work itself is represented. Hard tasks are difficult to create and validate, model-generated tasks and graders can share errors, and final scores often hide where failures occurred.
These are problems of measurement, experimental design, systems evaluation, reliability, inference, and representation of real work.
Evaluation also influences decisions across research, industry, policy, and everyday use. More researchers, practitioners, and domain experts should have a role in deciding what gets measured and what counts as useful evidence.
AES exists to build the scientific community and evidence base needed to address these problems.
IIWhat makes agent evaluation different+
Agents interact with environments
Environment design, state, access, and implementation can materially change measured performance.
Agents act through trajectories
Intermediate actions, recovery, cost, and risk can matter as much as the final outcome.
Agents perform real work
Real work involves context, judgment, dependencies, changing information, and multiple valid approaches.
Agent evaluations are systems too
Tasks, graders, environments, reference standards, and generated evaluation data can each introduce error.
Human-agent interaction matters
Agents increasingly work with people, so evaluation also needs to study delegation, reliance, correction, and oversight.
IIIScientific foundations+
Agent evaluation creates new technical problems, but many underlying scientific questions are familiar. AES brings methods from measurement science, statistics, epidemiology, experimental design, cognitive and behavioral science, human factors, systems engineering, reliability, and meta-science to the study of agentic systems.
| Agent-evaluation question | Relevant scientific foundations |
|---|---|
| What are we actually measuring? | Measurement theory / psychometrics |
| Does the evaluation represent the work and population of interest? | Epidemiology / sampling science / work studies |
| Are comparisons and interventions interpretable? | Experimental design / statistics / causal inference |
| How should behavior across interactions be studied? | Cognitive science / behavioral science / linguistics |
| Where and why did a system fail? | Systems / reliability / safety engineering |
| Which findings are robust across evaluations? | Replication / evidence synthesis / meta-science |
AES does not prescribe one benchmark, methodology, or school of thought. Researchers from different disciplines can bring their own questions and approaches.