LLM Evaluation: Test Suites That Predict Production Quality

An LLM evaluation is useful only if its result changes an engineering decision and predicts an outcome that matters after deployment. A collection of clever prompts can demonstrate model capability, but it cannot tell a team whether a new prompt, model, or context policy is safe to release. A single average score is even weaker: it can rise while the candidate becomes worse for a rare language, a high-risk action, or the exact long documents that dominate production cost.

Good evaluation is measurement design. It specifies the task population, samples cases from that population, records what a successful response means, applies graders whose errors are understood, compares a candidate with the current system, and preserves uncertainty instead of hiding it behind decimals. The suite is a maintained prediction instrument, not a one-time leaderboard.

This article develops that methodology around a document-intake system that extracts obligations from service agreements. The example is deliberately narrow so the evaluation can be precise. The same reasoning applies to assistants, classifiers, code tools, summarizers, and multimodal systems: begin with the deployment decision, then build evidence strong enough to support it.

Define the Decision Before the Dataset #

“Evaluate the new model” is not a testable objective. Define the change, the traffic it may receive, the decision the result will inform, and the unacceptable outcomes. For example:

Decide whether prompt version 18 and model B may replace prompt version 17 and model A for English service-agreement intake, without reducing clause recall in high-risk agreements or exceeding the review-time budget.

That statement fixes a baseline, candidate, population, task, risk slice, and operational constraint. It also prevents scope drift. A benchmark about trivia knowledge or generic summarization fluency is irrelevant unless those capabilities causally support agreement intake.

Next choose the evaluation unit. It might be one user turn, a complete conversation, a tool trajectory, one document, or an entire business workflow. Score at the unit where failure becomes meaningful. If an extraction system processes a document in chunks, perfect chunk-level accuracy can still assemble duplicate or contradictory obligations. The agreement, not the chunk, should therefore be the primary unit, with clause-level diagnostics underneath it.

Represent quality as several named dimensions rather than one synthetic score:

Do not average dimensions that express hard constraints. A candidate cannot compensate for leaking confidential text by using fewer tokens. Mark non-negotiable criteria as gates and use softer dimensions for tradeoff analysis. Write this policy before seeing candidate results; changing it afterward invites motivated reasoning.

Model the Production Population, Then Sample It #

An evaluation predicts production only to the extent that its cases represent production. Start with a taxonomy of inputs and conditions, ideally from privacy-reviewed production traces, product analytics, support incidents, and domain-owner interviews. For agreement intake, relevant axes might include document length, scan quality, template family, jurisdiction, clause type, table density, amendment structure, language, and customer risk tier.

Random sampling estimates common-case performance, but rare consequential cases may barely appear. Use a stratified suite with three complementary parts:

1. A probability sample that approximates the expected traffic distribution.
2. Oversampled critical slices for rare but severe failures.
3. Challenge cases that isolate known mechanisms, boundary conditions, and past incidents.

Keep their interpretations separate. The probability sample can estimate expected production quality after applying appropriate weights. The critical and challenge sets answer “can this release handle these obligations?” but should not be blended into an unweighted global rate. Otherwise adding more adversarial cases appears to make the same system regress.

Each case needs provenance and a reason for inclusion. Store a stable case ID, input or reconstructable fixture, population slice, collection time band, expected behavior, label source, sensitivity class, and allowed uses. Remove customer identifiers and preserve semantics during redaction. Synthetic cases can fill controlled gaps, but mark them as synthetic and do not let their neat phrasing displace messy real inputs.

Guard against leakage. A case used repeatedly to tune prompts becomes part of the development set, even if nobody calls it training data. Maintain at least these partitions:

• Development: visible examples used during implementation.
• Regression: versioned failures that every candidate must pass.
• Holdout: access-controlled cases used for release estimates.
• Freshness sample: recently collected cases that detect population drift.

Deduplicate semantically, not only by exact text. Template agreements can differ only in company name and date, inflating apparent sample size. Group near-duplicates by source template or document family and keep groups in one partition so variants cannot leak across development and holdout sets.

Write Labels and Rubrics That Another Reviewer Can Apply #

Many LLM tasks permit several good outputs, so a canonical answer string is often the wrong target. Specify observable requirements and allowed variation. For each service agreement, the reference can contain obligation records with actor, action, condition, deadline, and supporting source span. It can also identify ambiguous clauses for which the correct behavior is escalation.

A rubric should define dimensions, ordinal levels, disqualifying errors, and examples near decision boundaries. “Good summary” is not reproducible. A stronger completeness rubric might read:

• 3: Includes every material obligation and preserves conditions and deadlines.
• 2: Omits or weakens one non-critical detail without changing the practical obligation.
• 1: Omits a material obligation or changes an important condition.
• 0: The output is unusable, unsupported, or assigned to the wrong party.

Rubrics need reviewer instructions about evidence. A grader should compare claims with the supplied document, not reward plausible legal language from prior knowledge. When the task has an explicit schema, deterministic validation runs before subjective review. Invalid output does not receive a generous semantic score because a human can infer what it meant; the downstream system cannot.

Create labels through a documented process. Domain experts should adjudicate high-risk cases. For nuanced tasks, have at least two reviewers independently label a calibration subset, discuss disagreements, and revise the rubric where wording caused inconsistent decisions. Agreement is diagnostic, not a target to game: high agreement on an underspecified but consistently wrong rule is still bad.

Record uncertainty in the reference. Some clauses genuinely admit multiple interpretations. A case can list acceptable alternatives or require abstention instead of forcing a false gold answer. Exclude a case from a metric only for a stated reason, and keep exclusion counts visible. Quietly discarding difficult disagreements makes the suite easier while making it less predictive.

Combine Deterministic Checks with Graded Judgments #

Use the least subjective grader that captures the requirement. A layered evaluator can stop early on structural failures and reserve expensive judgment for semantic questions.

Deterministic checks are appropriate for JSON validity, required fields, enum membership, exact routing, source-span boundaries, forbidden data, tool names, and arithmetic derived from structured inputs. Reference-based functions can calculate set precision and recall when expected items are enumerable. Semantic equivalence, instruction following, writing quality, and partial support usually need human or model judgment.

For extracted obligations, let $R$ be the reference set and $P$ the predicted set after a documented matching procedure. Then

The intersection is not necessarily string equality. Define matching fields and tolerance before evaluation: the same actor and action, compatible condition, deadline normalization, and a source span that entails the record. If a model splits one obligation into two records, the matching algorithm must avoid awarding both as independent discoveries. Review unmatched pairs to distinguish model failures from matcher failures.

Metrics should follow the cost of errors. Missing a termination obligation may be more consequential than producing one duplicate low-risk notice. Report critical-clause recall separately and make unsupported obligations a hard or near-hard gate. Do not bury those outcomes inside token-overlap scores such as BLEU or ROUGE, which reward surface similarity without establishing factual support.

Pair every baseline and candidate response on the same case, context, tool fixtures, and execution policy. Paired comparison removes case difficulty as a confounder. It also enables direct outcomes such as candidate win, tie, or loss, plus the per-case change in a numeric metric. Randomize response order and mask system identity from subjective graders to reduce position and brand bias.

Work a Release Comparison End to End #

Suppose the team wants to test a candidate that uses a revised extraction prompt and a different model. The suite contains a weighted traffic sample, a critical-clause set, and a challenge set. The following figures are illustrative, not observed measurements.

First freeze the run manifest:

evaluation: agreement-intake-2026-07
baseline:
  model: model-a-2026-05
  prompt: extract-v17
candidate:
  model: model-b-2026-06
  prompt: extract-v18
execution:
  temperature: 0
  max_output_tokens: 2400
  attempts_per_case: 3
dataset:
  traffic_sample: traffic-holdout-07
  critical_set: material-obligations-v4
  challenge_set: extraction-regressions-v9

Three attempts do not make a stochastic model deterministic. They estimate run-to-run variation and expose brittle cases. The evaluator stores every raw response, parsed record, grader version, and trace rather than retaining only aggregates.

An illustrative result table might be:

The candidate’s global recall is higher, but its critical-clause recall is lower. The correct response is not to average the table and declare a win. Inspect paired losses in the critical slice. Assume they cluster around obligations defined in amendments that override a base agreement. That pattern creates a falsifiable hypothesis: the revised prompt is prematurely finalizing base-document clauses before processing overrides.

Add no new holdout policy after seeing the pattern. Repair on development cases representing amendment precedence, then rerun the frozen suite as a new candidate. Promote the discovered mechanism to a regression family, but keep the original holdout cases protected. The evaluation has done its job even though it blocked release: it identified a production-relevant failure hidden by an aggregate improvement.

For each run, produce a decision record containing configuration hashes, dataset versions, grader versions, exclusions, aggregate and sliced results, uncertainty, investigated clusters, and the named approver. A result that cannot be reconstructed is not a release artifact.

Calibrate Model Judges Instead of Trusting Fluency #

An LLM judge can scale semantic grading, but it is another fallible model with position bias, verbosity preference, self-preference, and domain limitations. Treat it as a measurement instrument that must be calibrated against the reviewers whose judgments matter.

Build a calibration set sampled across quality levels and important slices. Have qualified humans grade it independently under the same rubric, adjudicate a reference, then compare judge decisions with that reference. Report confusion matrices by score and slice, not only overall agreement. A judge that reliably separates excellent from unusable answers but confuses adjacent middle levels may still support a binary release gate; it should not produce precise four-level claims.

Reduce avoidable bias by hiding model identity, randomizing pair order, evaluating one dimension at a time, requiring evidence from the input, and using structured judge output. Ask for cited reasons before the final score, then validate those citations where possible. Repeating a judge call can estimate instability, but majority vote does not correct a systematic bias.

Calibrate thresholds for the use case. If the judge’s “supported” label has unacceptable false positives on critical clauses, route those cases to humans or deterministic entailment checks. Sample judge passes and failures for ongoing human audit. Recalibrate whenever the judge model, rubric, task distribution, or prompt changes; a frozen judge prompt alone does not freeze behavior when its provider silently changes a model.

Avoid circularity. A candidate and judge from the same model family may share blind spots or stylistic preferences. Human adjudication, deterministic checks, and independent judge models provide useful diversity, but none removes the need to inspect disagreement. The goal is not to eliminate human work. It is to spend it where grader uncertainty and business risk are highest.

Analyze Segments, Variance, and Paired Uncertainty #

Evaluation results are estimates from finite cases and stochastic executions. Report sample counts and uncertainty. For a paired numeric metric, calculate each case’s candidate-minus-baseline difference, then estimate the mean or median of those differences. Bootstrap the cases, keeping all repeated attempts for a case together, to produce an interval without pretending individual generations are independent documents.

Segmentation is where many actionable failures appear. Pre-register slices tied to plausible mechanisms: long context, OCR noise, tables, amendments, clause type, language, customer tier, and model route. Report denominators. A 100% pass rate on two examples is a prompt for more data, not strong evidence.

Do not search hundreds of slices and present only the worst one as a proven regression. Exploratory slices are valuable for discovering hypotheses, but label them exploratory and confirm them on fresh cases. Correcting for multiple comparisons or requiring replication prevents normal noise from turning into endless release blockers.

Aggregate results with production weights only when those weights are known and stable. Also publish unweighted critical-slice outcomes. If traffic composition changes, reweighting old cases may provide a quick estimate, but it cannot represent a new document family absent from the dataset.

Inspect distributions rather than averages alone. A candidate may improve most agreements while catastrophically failing a few long ones. Per-case scatter plots, loss clusters, and worst-regression review reveal that shape. Preserve qualitative error codes such as missed_override, wrong_actor, unsupported_deadline, and duplicate_obligation; they connect metric movement to repairs.

Turn Results into Release Gates and Online Validation #

A release gate combines fixed safety floors, non-inferiority constraints, and tradeoff budgets. For example: zero severe disclosure failures; critical recall not lower than baseline beyond a predefined tolerance; unsupported-record rate below a ceiling; weighted quality improvement with an interval excluding a harmful regression; and review effort plus cost within budget. Assign an owner for exceptions and require a written rationale with an expiry.

Offline evaluation cannot capture every production effect. Users change their behavior, upstream documents drift, queues alter latency, and human reviewers may react differently to candidate wording. Validate a passing candidate through shadow execution or a limited experiment. Define online outcomes before launch: correction rate, escalation, task completion, reviewer time, abandonment, latency, and cost. Instrument configuration versions so outcomes can be attributed.

Online signals also have confounders. A lower correction rate may mean users trusted bad output, not that output improved. Passive thumbs-up data is sparse and selection-biased. Pair product metrics with sampled expert review and explicit feedback reasons. Never optimize a proxy without checking the user outcome it represents.

Feed confirmed production failures into the regression set after privacy review. Also refresh the probability sample on a schedule so the suite does not become a museum of old incidents. Track performance by collection time to detect drift. Retire obsolete cases deliberately, retaining history and the reason, rather than silently rewriting fixtures until the current model passes.

Evaluation code deserves normal software discipline: schema validation, unit tests for matchers, versioned prompts and rubrics, immutable run artifacts, deterministic dataset assembly, and access controls for sensitive holdouts. Monitor grader cost and latency, but never cache a score across changed inputs, judge versions, or rubric versions.

The durable standard is straightforward. Every release claim should identify the target population, dataset provenance, quality dimensions, grader validity, baseline, uncertainty, critical slices, and decision rule. If any link is missing, a polished score may still be useful for exploration, but it does not justify production change. A test suite predicts production quality when it continuously earns that claim through representative cases, calibrated measurement, paired evidence, and honest follow-up after deployment.