Evals and Benchmarks: Turning “It Seems Better” Into Repeatable Tests

What an evaluation establishes

An evaluation systematically assesses behavior against an intended purpose. Before asking how well a system scored, state what system was evaluated, what work it was expected to do, what information and actions it could use, who could be affected, and what failures matter. A learned model—behavior fitted from examples rather than fully specified by code—adds variability, but it does not remove the need to test deterministic software around it. Machine Learning Fundamentals develops that learning process; here the concern is the evidence about the resulting system. [4][5]

Three different claims

Evidence for one claim does not automatically establish the next. A parser can confirm valid JSON without confirming that its fields are true; a database check can confirm a reservation without proving that the user authorized it. These are questions of construct validity: whether the chosen observations measure the property the claim names. A correct reservation can still impose enough review or delay to make the workflow worse. External validity is the separate question of how far a supported conclusion extends beyond the cases, conditions, people, and outcomes that were examined. [4][5]

Define worthwhile improvement

A baseline is the reference approach that the candidate must improve: the incumbent workflow, a simple rule, an earlier model, or sometimes no automation. The strongest baseline is not necessarily the most sophisticated model. It is the credible alternative for the same job. Acceptance criteria turn the intended benefit and its limits into observable conditions, while construct validity asks whether those observations really capture the property the decision concerns. [4][7]

Keep unlike outcomes separate

These dimensions can disagree. In METR’s bounded early-2025 study, 16 experienced developers completed 246 issues in familiar open-source repositories; allowing the tested AI tools was associated with 19% longer completion time even though participants later believed the tools had accelerated them. That historical result does not generalize to current tools or all developers. It demonstrates the narrower lesson: perceived speed and measured completion time are different constructs, so an adoption decision should measure the workflow outcome it actually values. [8]

Specify the case and its outcome

An evaluation case defines the input, information available at decision time, initial state, permitted resources and actions, stopping conditions, and assessment criteria. A trial is one execution of that case. An evaluation harness runs trials and records their artifacts. A grader inspects specified evidence and assigns an assessment. These roles should stay separate: a harness can execute correctly while a grader is wrong, and a grader can be correct about an output produced by a broken environment. [9]

The scored unit may be one answer, one extracted record, a conversation, or a complete task. For a reservation task, a fluent “Booked” message is evidence only about the response. The authoritative reservation record is evidence about the final state. If policy requires explicit confirmation, the recorded interaction must be checked separately. Agent Engineering’s outcome section explains how an acting system establishes what it accomplished. [10][11]

Define unsuccessful and unresolved runs

Sample the intended work

The task population is the work to which the conclusion should apply. A slice is a meaningful subset, such as a language, customer type, document source, task family, or risk category. Representative assessment samples the intended population or applies justified weights to recover its mix. A challenge set instead concentrates difficult or consequential conditions to discover weaknesses. Its failure rate is diagnostic; without population weights, it is not an estimate of how often those failures occur in deployment. [13][14][15]

Population weights describe the intended work, not the evaluation team’s choice to oversample difficult cases. In this example, both populations retain the same slice success rates; only their mix changes. [14]

Independence is about the generating unit, not the spreadsheet row. Paraphrases from one template, turns from one conversation, records from one customer, or passages from one source document can leak information across a random row split. Grouped splits keep related cases together; temporal splits keep later cases out of development when the intended claim concerns future work. Neither repair a sampling frame that excludes important users or conditions. Synthetic cases can broaden controlled coverage, but their generator can also impose unrealistic language, behavior, or shortcuts. [16][17]

What benchmarks standardize

A benchmark combines reusable tasks with an assessment protocol. Comparability depends on more than sharing inputs: the reference judgments, prompts, resource limits, allowed tools, attempt policy, aggregation, access rules, and benchmark version must also be compatible. A published ranking therefore describes performance under one protocol; it does not establish usefulness for a different application population or workflow. [15][18][19]

Must-know developments

TREC built on the earlier Cranfield tradition of using shared test collections for comparative retrieval evaluation. The broader development was not a replacement sequence: shared experiments expanded the properties and conditions that could be compared, while focused tests remained useful for narrower questions. Maintenance is part of the measurement work. Correcting underspecified tasks or faulty verifiers improves the benchmark rather than the evaluated model. The Art & Science of Benchmarking Agents develops practical choices around task quality, deliberate coverage, and benchmark usability. [20][23][24]

Choose checks that match the requirement

A test oracle is the procedure used to distinguish acceptable from unacceptable behavior. Creating inputs does not solve the separate problem of deciding whether their outputs are right. Complete executable specifications are rare, so most suites combine partial oracles whose guarantees must be stated narrowly. [25]

Requirement, evidence, and remaining gap

Ribeiro and colleagues’ 2020 CheckList crossed capabilities with minimum-functionality, invariance, and directional tests. For example, changing an irrelevant location name should preserve a sentiment judgment if the transformation truly preserves the task’s meaning. Such metamorphic relations are useful when a complete answer is unavailable, but the relation itself is still a human specification. The converse failure is specification gaming: an optimizer satisfies an implemented proxy while violating the intended task. DeepMind’s block-stacking example rewarded the bottom face’s height, allowing a block flip to collect reward without stacking. The optimization worked; the measurement did not express the goal. [26][28]

Use deterministic code for structure and precisely observable state, human assessment for criteria requiring expertise or preference, and model judges for scalable semantic assessment only after validation. A domain may require several fidelity layers; Document Understanding and OCR shows why transcription, structure, source location, and whole-document readiness are distinct properties. [29]

Keep denominators visible

A metric is a numerical summary of an observed property. Its numerator, denominator, positive class, aggregation unit, and handling of missing results are part of its meaning. Suppose “positive” means accepting an output. A true positive is a valid output accepted; a false positive is an invalid output accepted; a false negative is a valid output rejected; and a true negative is an invalid output rejected. [30][31]

Precision asks what fraction of accepted outputs were valid. Recall asks what fraction of valid outputs were accepted. False acceptance rate asks what fraction of invalid outputs slipped through; it is not generally $1-\mathrm{precision}$. If a denominator is zero, the corresponding metric is undefined, not perfect. Precision also depends on prevalence: when invalid cases become rarer, accepted outputs can have higher precision even if the conditional acceptance behavior is unchanged. [30][32]

Assessment coverage belongs beside performance

Aggregation changes the question. Pooled success weights slices by case count; equal-slice mean success gives every nonempty slice equal weight. Neither automatically represents deployment. Severity-specific outcomes should remain visible when a few consequential failures could be hidden by a large routine slice. Retry policy also changes the measured protocol: repeatedly retrying only cases that trigger a runner bug gives those inputs additional chances and can distort the aggregate. [33][12]

Build a human reference

Human assessment means people apply explicit criteria to observed system behavior. A rubric defines those criteria, decision boundaries, and representative examples. Separate criteria such as factual correctness, policy compliance, usefulness, and style instead of asking for an unexplained overall impression. Reference answers can illustrate valid outcomes without implying that only one wording is acceptable. [34][35]

A defensible procedure

Inter-rater agreement measures consistency under the protocol; it does not establish truth. Majority vote can conceal a shared misunderstanding, and authoritative adjudication can conceal a defective rubric. Pairwise assessment can be easier than assigning an absolute score, but it still needs ties and “not comparable” outcomes when appropriate. How Evals and Prompts Shape Agent Behavior describes how edge cases exposed disagreements within a team and motivated clearer examples for raters. [36][35]

Validate the model judge

A model judge is a learned model used as a measurement instrument. Its behavior is defined by the model version, rubric, reference material, candidate presentation, examples, decoding configuration, and result categories. A reference-based judge compares against supplied evidence or answers; a reference-free judge relies on its own learned knowledge and the rubric. Pairwise judging compares candidates; absolute judging assigns each candidate a category or score. [37][38]

Faithfulness asks whether claims follow from supplied material. It does not establish that the material is true or complete. Conversely, a true statement absent from the supplied context may fail a strict faithfulness check. Retrieval-Augmented Generation owns the full evidence-to-answer pipeline. Here the lesson is to give the judge the information required by the criterion and not silently substitute one property for another. [39][40]

Validate a frozen judge on independent human-assessed cases. Report its confusion matrix, ties or abstentions, uncertainty, and errors by consequential slice. Overall agreement can hide a judge that works on routine writing but falsely accepts policy violations. Review disagreements before changing either the application or the judge. [31][41][42]

Known nuisance variables

The 2023 MT-Bench and Chatbot Arena study compared model judgments with human preferences and tested order swaps, verbosity, and reference-guided judging. It showed that candidate order can change a verdict and that a judge can reproduce an error present in a candidate’s reasoning. A separate 2024 summarization study found tested models preferred their own generations more than human quality differences explained. These are scoped findings, not universal bias rates. They motivate controls in the actual harness: swap order, test style-only changes, use independent references where appropriate, and revalidate after changing the judge. A persuasive generated explanation is diagnostic text, not independent proof of why the verdict occurred. [43][44][45]

Labeled examples can sharpen a judge’s boundary, but they also become part of its input. Prompting and In-Context Learning explains how demonstrations affect behavior without changing weights. Keep those examples separate from the final validation set, and record their order and content in the evaluation manifest. [38][46]

Protect the independent assessment

Contamination occurs when evaluation information reaches training or another path that violates the intended test. Direct routes include duplicate tasks, paraphrased questions, published solutions, prompt demonstrations, retrieval sources, and evaluator examples. Whether access is improper depends on the task contract: web access may be the capability under test in one benchmark and prohibited leakage in another. [47][16]

Adaptive overfitting is different. Repeatedly choosing prompts, models, thresholds, or tools after inspecting the same holdout makes the selected system dependent on that holdout even if its examples never enter training. Dwork and colleagues’ 2015 work formalized this problem in adaptive data analysis and developed reusable-holdout methods that restrict validation feedback under stated assumptions. An ordinary hidden set with unrestricted score feedback receives no such guarantee. [48]

Practical separation

Freshness helps but does not prove independence when training corpora are unknown or runtime lookup is allowed. LiveBench’s 2024 design combined recent sources, automatically checkable answers, and planned updates, but the authors acknowledged that checkability excludes valuable open-ended tasks and that prompts still matter. Protection, coverage, and maintainability remain tradeoffs rather than a permanent “contamination-free” state. [49][47]

Preserve the run and the assessment

Operational reproducibility means preserving enough artifacts and conditions to test the reported computation again and interpret expected variation. It is not necessarily byte-identical output. The National Academies distinguishes computational reproducibility using the same data and methods from replication using newly collected data to address the same question. Generalization to another population is a further claim. [50]

Evaluation manifest

Inspect’s documented records connect task and model versions, execution settings, outputs, scores, errors, timing, and usage. Those records aid interpretation, but an identifier does not make a dependency immutable or recoverable. OpenAI’s archived 2023 seed documentation likewise described only best-effort consistency: matching parameters and a backend fingerprint still did not guarantee identical responses. A seed alone is not an environment snapshot. [46][51]

Rerun versus rescore

Keep harness, task, and grader failures separately labeled rather than deleting inconvenient samples. The raw per-task and per-attempt records will later determine the correct unit for uncertainty and paired comparison. [12]

Identify the sources of uncertainty

Measured performance can vary because different tasks are sampled, repeated executions take different paths, or assessors disagree or err. These are distinct sources. A confidence interval is produced by a procedure designed to cover a fixed population parameter at a stated rate over repeated samples under its assumptions. It is not the probability that one answer is correct, and a narrow interval does not remove sampling bias, grader bias, or distribution shift. [53][54]

A basic sample-size effect

For binary independent trials with a common failure probability, an exact one-sided 95% binomial bound after zero observed failures is 1 − 0.05^(1/n). With 20 independent trials and no failures, the upper bound is about 13.9%; with 200, it is about 1.5%. The larger sample narrows uncertainty, but neither result establishes zero future risk. This calculation assumes independent trials with one common probability. Heterogeneous or clustered tasks require a sampling model that preserves their structure; repeatedly executing one task estimates that task’s stochastic behavior instead of enlarging the task population. [54]

A bootstrap repeatedly resamples the independent observational units with replacement and recomputes the statistic. If cases are clustered by user, template, or scenario, resample appropriate clusters rather than pretending every row is independent. DigiWorld illustrates a nested target: apps contain scenarios, configurations, and stochastic rollouts. Its hierarchical bootstrap follows that structure to estimate variation within a fixed curated suite; it does not turn the chosen apps into a random sample of all deployment environments. [55][56]

More samples address random uncertainty only under the sampling model. They cannot fix systematically wrong reference labels, an incomplete oracle, omitted user groups, or a judge that shares the candidate’s error. Those defects require measurement repair. [4][43]

Compare changes on matched work

A paired comparison evaluates baseline and candidate on the same sampled tasks. For task $i$, first summarize each version’s prespecified repeated outcomes as $A_i$ and $B_i$. The task-level difference is $d_i=B_i-A_i$, and an equal-task-weight effect estimate is $\bar d=\frac{1}{N}\sum_i d_i$. Resampling task indices while carrying both versions together preserves the pairing. Related tasks still require a suitable cluster design. [56]

The effect size is the magnitude of the difference in the units that matter: percentage points of task success, minutes of human correction, or cost per completed valid task. Statistical detectability is not practical importance. Before comparison, define a smallest worthwhile improvement or an acceptable-loss margin from the intended use. For a higher-is-better outcome and $d=\mathrm{candidate}-\mathrm{baseline}$, a non-inferiority claim asks whether the data exclude $d\le -m$, where $m>0$ is the largest acceptable loss. A favorable point estimate with an interval crossing $-m$ remains inconclusive. [1][2]

Inspect individual gains and regressions as well as the aggregate. Improvements on routine cases can offset a critical slice regression numerically without making it acceptable. Also distinguish an equal-resource capability comparison from a whole-workflow comparison: changing models, retries, tool access, review effort, or latency may be the product configuration, but the resulting difference is not a model-only effect. [18][57]

Repeatedly testing many candidates and reporting the best one creates selection bias. Predeclare the main comparison and decision rule, control or disclose multiple comparisons, and confirm the selected change on protected evidence. Failure to detect a difference establishes neither equality nor acceptable equivalence. [34][2][48]

State the attempt policy

Success on one attempt, success at least once, and consistent repeated success are different operating claims. HumanEval introduced an executable code benchmark and pass@k, which asks whether at least one of $k$ sampled candidates passes its tests. It does not mean every attempt succeeds or that a user can identify the passing candidate without a checker. With $n$ sampled candidates and $c$ passing candidates, its estimator is $1-\binom{n-c}{k}/\binom{n}{k}$, averaged across tasks. [58]

By contrast, τ-bench’s pass^k asks whether all $k$ independent trials succeed. Mean success asks what fraction of attempts succeed. The same trial record can therefore support three different summaries. Adaptive retries, repairs informed by previous failures, correlated sampling, and variable budgets define yet another policy. Record the checker, attempt budget, selection rule, latency, and cost. Reasoning and Test-Time Compute owns the mechanisms for sampling and searching candidate solutions. [11][59]

One record, different questions

Interpret probability forecasts

Calibration means predicted probabilities for a specified event match observed event frequencies. If a system assigns probability 0.8 to “this automatically accepted answer is valid,” then among comparable cases receiving about 0.8, roughly 80% should be valid. Calibration differs from accuracy, the ability to rank easier cases ahead of harder ones, and uncertainty in an aggregate performance estimate. [3]

A reliability diagram groups predictions and plots each group’s average forecast against its observed event frequency. Counts must accompany the bins: a point based on three cases is not as informative as one based on three hundred, and different binning can change the picture. A diagonal pattern indicates empirical calibration for the tested population, not proof that the probabilities transfer under changed conditions. [3][60]

A probability scoring rule

The customary binary Brier loss ranges from 0 to 1, with lower values better. It reflects more than calibration, so a lower Brier value does not alone prove a better reliability diagram. Record the normalization when comparing libraries: an unscaled sum across two complementary binary classes is twice this customary form. [61][3]

Token likelihood, verbal certainty, a judge’s rating, and probability of task correctness are not interchangeable. Verbal-confidence studies have found that calibration depends on the model, task, and elicitation method. The vocabulary probabilities described in Transformers and Attention concern next-token prediction; a separate validated construction is needed to interpret them as probabilities of complete-task success. Fit any calibrator on data independent of the underlying predictor’s training cases, and reassess after distribution shift. [62][3][60]

Evaluate deferral as a policy

Abstention means declining automatic completion. Coverage is the fraction of eligible cases accepted automatically, while selective risk is expected error conditional on acceptance. A threshold can use a ranking score even when that score is not a calibrated probability, but the selected policy still needs independent evaluation. [63]

Choose the acceptance threshold as an operating policy: it decides which eligible cases finish automatically and which enter review. Lowering coverage often reduces accepted-case error because the system retains easier cases, but can increase review burden. Report capacity, reviewer accuracy, waiting time, unresolved cases, and failures after escalation. At zero coverage, accepted-case risk is undefined and the system delivers no automatic work. [63]

Threshold decisions need both sides

Understand simulation and replay

Offline evaluation assesses prepared cases outside the active workflow. A simulation generates interactions or environment changes under a model of users and state. Replay feeds captured artifacts into some part of the system again. Transfer can fail when task mix, available information, user behavior, external state, or human intervention differs from production—a form of distribution shift developed more fully in Machine Learning Fundamentals. [5][17]

A user simulator is part of the measurement instrument. In Lyft’s evaluation account, an initial simulated user patiently supplied complete explanations unlike frustrated production users. Incorporating production examples and a user model closer to customer language made the evaluation harder and lowered the reported score. The decrease alone did not prove predictive validity; it exposed how simulator behavior changes the task presented to the agent. A separate 2025 user-model study likewise found materially different assistant success under different simulators, while cautioning that lower scores do not by themselves establish realism. [17][64]

Replay has a causal boundary. Re-executing a parser or grader on captured input can test that component. Replacing an action in a stateful workflow changes the next state and therefore later observations. Feeding the original future observations to the changed policy generally does not simulate its outcome. Counterfactual replay needs a valid environment model and controlled external factors to recompute the divergent branch. RL Environments and Simulators owns environment construction and fidelity. [65]

Account for incomplete feedback

Production labels are often selected by the workflow. Selective labels arise when existing decisions determine which outcomes become observable. If only escalated cases receive expert review, automatic cases may lack a comparable correctness label. If users complain only about visible failures, silence is not proof of success. A click reflects exposure and presentation as well as usefulness; an accepted suggestion may still require later correction. [66][67]

Delayed feedback creates another distinction: an event may not have occurred, or it may not yet have been observed. Chapelle’s 2014 advertising study modeled eventual conversion separately from conversion delay because a short labeling window misclassified later purchases as negatives, while waiting longer made the data older. Its 30-day attribution rule was specific to that study, not a universal evaluation window. Every application must define follow-up and how unresolved outcomes enter denominators. [68]

Routing decisions can change apparent quality without changing underlying behavior. Nubank’s support report separates post-interaction satisfaction from self-service without human escalation and notes that sending difficult cases to people can improve satisfaction while reducing self-service. Report the full workload: eligible cases, automatic cases, escalations, reviewed cases, observed outcomes, and outcomes still unavailable. Production feedback should become a review candidate, not automatic ground truth. [67][69]

Choose the live experiment

Live designs answer different residual questions. A shadow evaluation copies current inputs to a candidate while production still supplies the user-visible result. It exposes compatibility and candidate outputs under current traffic, but it does not measure the consequences of applying those outputs. Shadow tools must have isolated state or denied production mutations; discarding a response does not undo side effects. A canary exposes a bounded live population to the candidate and checks prespecified operating and quality criteria. A randomized comparison assigns eligible units to control or candidate and can estimate an intervention effect under its assumptions. [70][71]

What each design supports

Choose the assignment unit that matches dependence. Request-level assignment can contaminate a user’s experience across turns; user-level assignment can still suffer interference through shared marketplaces, teams, or infrastructure; workflow-level assignment may reduce interference but reduce effective sample size. Cluster randomization and switchbacks are design options, not universal cures. Define primary outcomes, constraints, exposure, follow-up, and stop conditions before looking at results. Repeatedly applying an ordinary fixed-horizon test while watching for significance can invalidate its false-positive guarantee; valid sequential methods require their own protocol. [57][72]

METR’s later 2026 update documents how selection before randomization, changed task choices, differential completion, and difficulty attributing time during concurrent agent use weakened inference in a follow-up productivity experiment. Random assignment does not repair who declines to enter the experiment or which tasks are withheld. Measure completed work and total human effort, not only generated output or satisfaction. Forward Deployed Engineering develops how customer benefit is established in field delivery. [73]

Rollback stops further candidate exposure; it cannot undo effects already delivered. In irreversible or safety-critical workflows, stronger pre-exposure evidence and narrower authority may be required because the normal “ship, observe, and roll back” assumption does not hold. [70][74]

Test the failure explanation

Error analysis begins with inspected cases, not a universal taxonomy. First decide what failed: the task, the reference label, the rubric, the automated judge, the harness, or the environment. Then group cases by observed mechanisms and meaningful slices. Prioritization should consider exposure, frequency, severity, and uncertainty separately; multiplying them into a single unsupported score can hide the material driver. [75][76]

A trace is a recorded sequence of operations. It can reveal what information, tool calls, and transitions preceded an outcome, but collection is not judgment and temporal order is not causal proof. Observability owns instrumentation and operational diagnosis. Evaluation supplies the required outcome and the criteria by which the trace or final state is interpreted. [77][6]

Sometimes the measurement needs repair before the application. In the 2024 EvalGen study, participants examining entity-extraction outputs disagreed about names embedded in hashtags: one wanted exclusion, while another wanted the name retained without the hash. The same assertion could not satisfy both intended criteria. More labels cannot rescue an implementation that encodes the wrong requirement; clarify the criterion and reconsider affected annotations first. [78]

Test the feedback hypothesis

Treat model-generated explanations as hypotheses. In controlled 2023 experiments, models often changed answers in response to introduced position patterns or suggested answers and then supplied plausible rationales without acknowledging those influences. Explanations can guide interventions, but only independent controls test the proposed cause. [45]

Make and maintain the decision

A release decision combines the intended improvement, paired evidence, uncertainty, critical regressions, review burden, live outcomes, and consequences of being wrong. More evaluation is useful when uncertainty is the obstacle and additional representative evidence can narrow it. More trials do not repair a wrong metric, biased sampling frame, broken grader, or invalid environment. [80][4]

Evidence pattern to justified action

Capability suites explore behavior that is still difficult; regression suites protect behavior that already works. Incident-derived cases are valuable regression and diagnostic evidence, but adding many rare failures changes the interpretation of an aggregate unless representative estimates remain separately weighted. A local repair also needs broader regression testing: the illustrative stop-sign example in From Self-driving to Autonomous Voice Agents asks whether a fix that stops at the target sign accidentally makes the system stop everywhere. [81][82]

An evaluation gate ties a criterion to an action and a responsible owner. A score that changes no decision and routes no regression to an owner does not close the loop. Lyft’s account recommends offline launch criteria, production regression detection, and clear ownership; it does not establish a universal threshold or completed automated gate. [83]

Decision record

Preserve old manifests, outputs, and decisions when changing the system or measurement. Rescoring historical outputs can answer how a new rubric views old behavior; only new execution assesses the changed system. Continuous evaluation is therefore not one endlessly growing score. It is a maintained set of scoped claims whose evidence, conditions, and expiration triggers remain visible. [84][52][80]

Open questions

1. 01

   How can evaluation suites remain independent when developers, automated optimizers, and model providers repeatedly receive score feedback? Progress would require practical information-limited protocols that preserve useful debugging while making adaptive reuse auditable. [48]

2. 02

   How should teams estimate performance when tasks are nested within users, templates, environments, and repeated stochastic trials? The hard part is identifying the actual independent sampling units; progress would look like evaluation tools that encode the sampling design and produce matched, hierarchical uncertainty by default. [55]

3. 03

   How can model judges be validated for unfamiliar domains, adversarial candidate text, and changing model versions without making human review as expensive as the work being automated? Useful progress would combine targeted expert labels, abstention, slice-specific error monitoring, and cheap revalidation after judge changes. [91][41]

4. 04

   How can simulators predict live behavior when users, tools, and external state respond to the system’s actions? Better progress would be measured by prospective prediction of live outcomes across held-out deployments, not simply lower simulated scores or more realistic-looking conversations. [64][65]

5. 05

   How should delayed, selectively observed, and human-mediated outcomes enter continuous evaluation? Progress would require explicit observation windows, missing-outcome accounting, and designs that separate improved system behavior from changes in routing, exposure, or review. [66][68]