How Do You Know Your Agent Grader Works?

An agent submits an incorrect patch. The tests pass. Was that an agent success, or a grader failure? Agent evaluation often treats the grader as part of benchmark implementation. Other measurement-heavy fields treat the analogous object as an instrument that must itself be validated. Analytical procedure validation asks whether a method is fit for its intended purpose, including its accuracy, precision, specificity, and robustness [1]. Software testing asks whether a test suite actually detects faults [2]. Diagnostic-test research separates the performance of the test from the prevalence of the condition it is used to detect. The same separation is useful for agent evaluation.

Start with the target outcome

Before testing a grader, state what it is supposed to recognize. For a coding task, “the test suite passes” may be the implemented criterion, while the target outcome is closer to “the requested change is correctly implemented without introducing material regressions.” For a research task, the implemented criterion might be a rubric score, while the target outcome concerns whether the analysis is substantively correct and reviewable. The validation record should therefore identify: Target outcome: what genuine task success means. Reference standard: how challenge cases are classified independently of the grader being tested. A reference standard can itself be imperfect. Its identity and uncertainty should still be explicit so that the grader does not define success circularly.

Inject known mistakes

Mutation testing evaluates a test suite by injecting deliberate faults and measuring which faults the suite detects [2]. The same idea transfers directly to agent graders. Construct a challenge set with two types of artifacts. Wrong but plausible outputs. These should contain realistic faults the grader is supposed to catch: an incorrect calculation, missing required change, unsupported claim, corrupted file, superficially correct answer produced from the wrong state, or another task-specific failure. Correct but unusual outputs. These should satisfy the task while varying irrelevant features such as formatting, ordering, wording, implementation style, or file structure. Then report: False acceptance: incorrect work accepted as successful. False rejection: correct work rejected as unsuccessful. This tests the grader on cases with known status before it is used on unknown cases.

Treat grading as diagnostic classification

If genuine task success can be established on a challenge set, the grader can be described using the same quantities used for a diagnostic test: sensitivity to true success and specificity to true failure. The base rate matters as well. Suppose a grader is 95% sensitive and 90% specific, while only 20% of attempts on a hard benchmark are genuinely successful. Among 1,000 attempts, 190 genuine successes would be accepted, but 80 failures would also be accepted. The reported pass rate would be 27%, and about 30% of the reported successes would be false positives. These numbers are illustrative, not empirical findings about a current benchmark. They show why one agreement statistic cannot summarize the grader's error profile.

Agreement is not correlation

A common validation exercise compares an automated judge with human ratings and reports a high correlation. Bland and Altman's classic argument applies directly: correlation measures association, not agreement [3]. Two methods can rank outputs similarly while one is systematically more lenient or severe. For continuous scores, inspect bias and the distribution of differences using an agreement method appropriate to the design. For categorical decisions, report an appropriate agreement statistic together with the confusion matrix. For an LLM judge, repeatability should also be tested. Re-grade the same artifacts under the same configuration, then vary features that should not change the judgement. Analytical method validation calls the latter robustness [1]. Perturbations can include harmless changes in formatting, order, whitespace, file names, or wording. If these changes materially alter the judgement, that variation belongs in the evaluation uncertainty.

Version the grader

A grader can change even when the benchmark tasks do not. The test suite, parser, rubric prompt, LLM judge model, or a dependency affecting execution can all change. The grader identity should therefore include its version, prompt or rubric, parser, tests, relevant dependencies, and date. If any of these change materially, prior validation may need to be repeated.

Grader Validation Card

If these fields are unknown, grader error remains an unmeasured source of uncertainty.

References

1. International Council for Harmonisation. ICH Q2(R2): Validation of Analytical Procedures. 2023.
2. Papadakis M, Kintis M, Zhang J, Jia Y, Le Traon Y, Harman M. Mutation Testing Advances: An Analysis and Survey. Advances in Computers. 2019;112:275–378.
3. Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement90837-8). Lancet. 1986;1(8476):307–310.