Diagnose
Identify what current evaluations measure, where they fail, and which gaps limit reliable conclusions.
Fall 2026 · New York City
Agent Evaluation Science
A one-day symposium advancing the methods, measures, systems, and evidence used to evaluate AI agents.
- Date
- Friday, November 20, 2026
- Location
- New York City · Primarily in person
- Submissions
- Short papers and 2-page extended abstracts via OpenReview
Scientific evaluation◆Construct validity◆Measurement reliability◆Benchmark integrity◆Agent safety◆Grader design◆Failure analysis◆Evaluation infrastructure◆Real-world evidence◆Reproducible systems◆
Why this symposium
Evaluation is becoming a scientific discipline
AI agents act across tools, environments, and extended workflows. Evaluating them requires evidence beyond a single score or benchmark.
The symposium brings together researchers and practitioners working on measurement, benchmark design, evaluation infrastructure, reliability, safety, and real-world performance.
1focused day
2submission formats
4evaluation stages
Evaluation science agenda
Four stages organize the symposium
Measure
Develop frameworks, constructs, metrics, graders, and evidence for reliability and validity.
Operationalize
Build runnable benchmarks, environments, harnesses, and reproducible evaluation infrastructure.
Apply
Evaluate agents in realistic settings using failure evidence and real-world performance.
Symposium experience
Research, evidence, and discussion in one program
Each part of the day connects methodological questions with practical evaluation experience.
01
Invited perspectives
Academic and industry speakers connecting evaluation methodology with deployed agent systems.
View speakers →02Contributed research
Selected short papers and abstracts presented through focused oral and poster sessions.
View the call →03Evaluation in practice
Case studies, benchmarks, tools, failures, and evidence from real-world agent evaluation.
View program →04Shared evaluation agenda
Structured discussion spanning diagnosis, measurement, infrastructure, and real-world application.
Explore the agenda →Call for presentations
Share work that strengthens agent evaluation
Submit evaluation studies, audits, negative results, failure analyses, tools, datasets, production lessons, and real-world evaluation cases. Ongoing and previously published work can also be presented through the extended abstract track.
- Evaluation studies
- Audits and replications
- Negative results
- Failure analyses
- Tools and datasets
- Production lessons
- Real-world evaluation cases
Industry and real-world submissions are assessed on concrete evidence, production lessons, and generalizable insight. A new benchmark or algorithm is not required.
Submissions open
August 252026Abstract registration deadline
October 202026Submission deadline
October 252026Symposium
November 202026Affiliations represented
The committee brings together researchers and practitioners across universities, research institutes, and industry.
- Harvard University
- Massachusetts General Hospital
- Princeton University
- Cornell University · Cornell Tech
- The University of Texas at Austin
- University of Technology Sydney
- Australian Artificial Intelligence Institute
- University of Alberta
- Amii
- Meta Superintelligence Labs
- Sony AI
- Raycaster
- Google DeepMind
- Massachusetts Institute of Technology
- MIT Media Lab
Affiliations describe committee members’ institutional or organizational connections. They do not indicate sponsorship.
Agent Evaluation Science Fall 2026Friday, November 20 · New York City
Meet the committee