Skip to content
RES/EVAResearchActive Investigation~14 min read

Evaluation: eval pipelines as an engineering discipline, not a checkbox.

Treating evaluation like CI: traces, fixtures, regressions, budgets, and ownership. The unit of progress in non-deterministic systems. This research track documents what is currently understood, what remains contested, and what is not yet known.

Area · EvaluationLayer · QualityFocus · Regression DetectionStatus · Active Research

Evals are not a leaderboard. They are a regression net. The field has not yet internalized the difference.

The dominant mental model for AI evaluation is borrowed from benchmarking: run the system against a test set, measure a score, compare to prior scores and competitor scores, publish the result. This model produces numbers that are easy to report and nearly useless for production engineering. Benchmark scores tell you how a system performs on a test set that was designed for comparison. They tell you nothing about whether the system regressed on the specific behaviors your users depend on between last week’s deploy and this week’s.

The engineering mental model for evaluation is different: evals are a regression detection system. Their job is not to produce a score that can be compared across systems. Their job is to catch, before production, any change in system behavior that degrades quality on the behaviors the system was built to guarantee. This is the same job that automated tests do in traditional software engineering — the difference is that the non-determinism of language model outputs makes the implementation significantly harder.

This research track investigates four areas where that harder implementation remains an open problem: trace contracts as early-warning signals, eval pipelines as CI infrastructure, scorer composition as a coverage strategy, and evaluation economics at production scale. Each area has findings at different confidence levels. This document states those confidence levels explicitly rather than presenting contested positions as settled ones.

§ IWhy Evaluation Is Hard

Non-determinism breaks every assumption that makes automated testing tractable.

Traditional automated testing is tractable because software functions are deterministic: given the same input, the same output is produced every time. This property makes tests straightforward to write and unambiguous to interpret. A test either passes or fails. A failure is a signal that something changed. The signal is reliable.

Language model systems break this property at the most fundamental level. The same input produces different outputs across runs, across model versions, across temperature settings, and across context window states. This non-determinism means that a test failure is not a reliable signal: it might indicate a genuine regression, or it might indicate that the system produced a valid output that the test did not anticipate. Both look identical from outside the system.

Four specific difficulties compound this fundamental one:

The ground truth problem. For most language model tasks, there is no single correct output. There is a distribution of acceptable outputs and a distribution of unacceptable ones. Writing a test requires either enumerating the acceptable outputs (impossible for open-ended tasks) or defining a scoring function that can classify outputs correctly (hard to specify, harder to validate). Most evaluation work is really work on the scoring function, not on the system under evaluation.

The coverage problem. The input space for a language model system is effectively infinite. Any finite eval set covers a vanishingly small fraction of it. The eval set that was written to cover the system’s behavior at launch may not cover the behavior that matters six months later when the user distribution has shifted. Eval sets that are not actively maintained become less representative over time even if the system does not change.

The silent failure problem. A language model system can degrade in ways that no infrastructure alert will catch. The system is still running. Latency is normal. Error rates are normal. The outputs are simply worse — more hallucinations under certain query types, degraded coherence on edge cases, subtly incorrect information in a specific domain. This degradation is invisible without an eval system that is specifically designed to detect it. Most organizations discover it when a user reports it.

The model update problem. Upstream model providers update their models on schedules that are not always announced in advance. A model update can change system behavior without any code changing. An eval system that only runs on code deploys will not catch regressions introduced by model updates. An eval system that runs continuously will catch them — and will also need to distinguish between regressions and legitimate behavioral improvements.

§ IIResearch Territory

Four investigation areas. Different maturity levels. Some settled, most not.

The research territory map below plots the four investigation areas against two axes: how conceptually clear the problem is (what we are trying to solve) and how mature the tooling is (what exists to solve it with). Areas with high conceptual clarity and mature tooling are settled. Areas with low conceptual clarity and immature tooling are where the most important open problems live.

Figure 1 · Evaluation Research Territory Map
TOOLING MATURITY →LOWHIGHCONCEPTUAL CLARITY →LOWHIGHCLEAR PROBLEMPOOR TOOLINGSETTLEDOPEN RESEARCHTOOLING AHEADOF THEORYSETTLEDEVAL-AS-CIKnown. Under-adopted.ACTIVESCORER COMPOSITIONPatterns emerging.ACTIVETRACE CONTRACTSNo standard yet.OPENEVAL ECONOMICSContested. Context-dependent.SETTLEDOWNERSHIP
Fig. 1 · Evaluation research territory · Settled = known + tooled · Active = understood, tools emerging · Open = contested or unclear
AreaCurrent findingStatusConfidenceBlocker
Trace contractsSchema-level contracts on traces catch silent regressions before they surface in user-facing quality metrics.ActiveMediumNo standard trace schema for LLM systems. Contracts are system-specific.
Eval-as-CIEvals that run automatically on every deploy and block on failure produce meaningfully different deployment behavior than advisory evals.SettledHighCultural — teams override failing evals under deadline pressure.
Scorer compositionLayered scorer sets (deterministic → reference-based → model-graded) outperform single-scorer approaches on coverage-to-cost ratio.ActiveMediumModel-graded scorer reliability is task-dependent and unstable across model updates.
Eval economicsAt production scale, sample-based evaluation over stratified query distributions produces better signal per dollar than exhaustive evaluation over full query logs.ActiveMedium-lowStratification strategy significantly affects result. No consensus on stratification criteria.
OwnershipAn eval system without a named owner decays inside a quarter regardless of initial quality.SettledHighOrganizational — ownership is treated as optional overhead rather than load-bearing infrastructure.

Status legend: Settled — finding holds across contexts with high reliability. Active — finding holds in most contexts, edge cases remain. Open — contested or highly context-dependent.

§ IIITrace Contracts

Schema-level contracts on traces catch silent regressions before quality metrics degrade.

A trace contract is a schema-level assertion about what a system’s execution trace should look like under normal operation: which spans should be present, what their latency bounds should be, what token counts fall within acceptable ranges, which tool calls should produce outputs matching a specific schema. A trace that violates a contract is a signal that something structural has changed in the system’s behavior — before that change has had time to affect output quality metrics.

The value proposition of trace contracts is timing. Quality metrics measure outputs. Outputs are downstream of execution. By the time a quality metric degrades, the root cause may be several steps removed from the symptom. Trace contracts operate upstream: they catch structural changes at the execution level, where the causal chain is shorter and the diagnosis is faster.

The practical challenge is that trace contracts require a stable notion of what “normal” looks like for a given system, and that notion must be maintained as the system evolves. A trace contract written for v1 of a pipeline may be violated legitimately by v2 if the pipeline architecture changes. Managing contract evolution without losing the early-warning signal is the current open problem in this area.

The connection to Technical Readiness’s trace literacy rung is direct: trace contracts are only possible for practitioners who have developed fluency with what a normal trace looks like. The contract is a formalization of that fluency — the practitioner’s operational knowledge encoded as an automated assertion.

Current research question: can trace contracts be generated automatically from a corpus of production traces, rather than specified manually? Automatic generation would solve the cold-start problem for systems without an established normal baseline. The risk is that automatically generated contracts encode whatever was normal in the training corpus, including behaviors that were normal but not correct.

§ IVEval-as-CI

The engineering pattern is settled. The adoption problem is cultural, not technical.

The technical pattern for eval-as-CI is well-understood and directly operationalized in the Technical Readiness framework’s first rung: an eval suite runs automatically on every deploy, gates the deploy on pass/fail, and is treated as a blocking condition rather than an advisory one. The implementation is not simple, but it is not contested. Teams that have implemented it consistently report the expected outcomes: regressions are caught before production, deployment confidence increases, the quality of the eval suite itself improves because the team is motivated to maintain something that blocks their work.

The reason this is classified as “settled but under-adopted” rather than simply “solved” is the gap between knowing the pattern and maintaining it under organizational pressure. The pattern requires a cultural commitment that most teams make once and then erode: the first time a failing eval is overridden to meet a deadline, the gate becomes a suggestion. The first time the eval suite is not updated after a system change, the suite begins to drift from the behavior it is meant to test. Neither event is a technical failure. Both are predictable failure modes of any infrastructure that requires active maintenance.

The research question that remains open is not “how do you build eval-as-CI” but “how do you maintain the cultural commitment to treating eval failures as blocking conditions over time, under sustained delivery pressure?” This is an organizational design question as much as a technical one, and the technical community has not produced satisfactory answers to it. The closest analogy is the maintenance of test suites in traditional software engineering — a problem that is also not solved, just better understood.

§ VScorer Composition

Layered scorer sets outperform single-scorer approaches. The composition strategy is still being worked out.

A scorer is a function that takes a system output and produces a quality signal. The three categories of scorers that currently matter in production eval systems are:

  • Deterministic scorers operate on structural properties of the output: does it match a regex, does it parse as valid JSON, does it contain a required field, does it fall within a length bound. These scorers are fast, cheap, perfectly reliable, and limited to surface-level properties. They catch a specific class of failures completely and miss everything else.
  • Reference-based scorers compare the output against a gold-standard reference: exact match, BLEU, ROUGE, BERTScore. These scorers can evaluate semantic similarity without requiring a model call. They are more expensive than deterministic scorers and less reliable — especially for tasks where there are multiple valid outputs that differ from the reference but are equally correct.
  • Model-graded scorers use a language model to evaluate the output: is this factually accurate, is this response helpful, does this follow the instructions. These scorers can evaluate properties that are impossible to formalize deterministically. They are the most expensive, the least reliable, and the most powerful for catching the failures that matter most.

The current best practice is to layer these scorers in order of cost and apply them in sequence: run deterministic scorers first and fail fast on structural violations, then reference-based scorers on the passing subset, then model-graded scorers only on what passes both prior layers. This composition reduces the cost of model-graded scoring without sacrificing its coverage of semantic failures.

The active research question is the reliability of model-graded scorers across model updates. A model-graded scorer is itself a language model system, which means it is subject to the same instability as the system it is evaluating. When the grader model is updated, the scoring distribution may shift even if the system under evaluation has not changed. This produces false regressions and false improvements that are difficult to distinguish from genuine ones without a stable reference scorer to calibrate against.

The reliability of a model-graded scorer is bounded by the reliability of the model doing the grading. That bound moves every time the grader model is updated. This is the most significant unresolved problem in production eval system design.

§ VIEval Economics

At production scale, evaluation is a resource allocation problem. The tradeoffs are not yet well-understood.

At small scale, evaluation economics are not a concern: run all the evals on all the cases and the cost is acceptable. At production scale, with thousands of queries per day and model-graded scorers that cost money to run, evaluation becomes a sampling and prioritization problem. You cannot evaluate everything. The question is what to evaluate, how often, and with which scorers.

Three tradeoffs govern eval economics at scale:

Sample size vs. signal confidence.Larger eval sets produce more reliable quality estimates. They also cost more to run. The confidence interval on a quality estimate from a hundred examples is wide enough that meaningful regressions may not be detectable. The confidence interval from a thousand examples is tighter but the cost may be prohibitive for continuous evaluation. The right sample size depends on the expected magnitude of regressions you need to detect — which is unknown in advance.

Coverage vs. cost. Comprehensive coverage of the query distribution requires a large, diverse eval set. Maintaining a large eval set is expensive in both creation and execution. Focused eval sets that target known high-risk behaviors are cheaper to maintain and faster to run, but miss regressions in uncovered areas. The appropriate balance between comprehensive and focused evaluation is highly system-specific and not yet well-theorized.

Continuous vs. triggered evaluation.Continuous evaluation catches model update regressions that code-change-triggered evaluation misses. It also produces ongoing cost regardless of change rate. For systems with low change rates and stable upstream models, continuous evaluation may be spending money to confirm that nothing has changed. For systems with frequent upstream model updates, continuous evaluation may be the only mechanism that catches regressions before users do. The correct policy is system-specific.

This is the most open area in the research territory. The difficulty is that the right answers depend on empirical properties of specific systems — query distributions, regression frequencies, failure costs — that are not known in advance and vary significantly across use cases. General principles are possible; general recommendations are not.

§ VIIOpen Questions

What this research track does not yet know.

Research pages in this publication explicitly name what is not yet understood. The following questions are active and unresolved as of this writing.

  • Automatic trace contract generation.Can trace contracts be generated reliably from production trace corpora without encoding incorrect behaviors as normal? What validation mechanism distinguishes a contract that reflects correct operation from one that reflects operation that happened to be correct in the training window?
  • Grader model stability. How should eval systems be designed to remain stable across grader model updates? Is the right approach to pin grader model versions, to calibrate grader outputs against a stable reference set, or to treat grader model updates as breaking changes that require re-baselining the entire eval suite?
  • Eval set maintenance at velocity.As system behavior changes, eval sets must be updated to remain representative. What is the right maintenance cadence? Who owns it? How should deprecated eval cases be handled — archived, deleted, or retained to detect regressions to previously fixed behaviors?
  • Regression vs. improvement classification. When a system’s behavior changes on a specific eval case, how should that change be classified? A change in the direction of the intended behavior is an improvement. A change away from it is a regression. For evaluations that rely on model graders, the classification itself is uncertain. Is there a reliable way to distinguish genuine improvements from regressions that the grader is misclassifying?
  • Eval economics at the long tail.The long tail of query types in production systems is precisely where the most consequential failures occur and where coverage is lowest. How should sampling strategies account for low-frequency, high-impact query types? Is importance sampling practical for LLM eval systems?

These questions are actively investigated in the Evaluation Systems build log and operationalized in the Evaluation Pipeline system. The research track feeds the system design. The system design generates new research questions. That loop is the point.

Closing

Evaluation is the unit of progress. Everything else is speculation.

The claim that evaluation is the unit of progress in non-deterministic systems is not rhetorical. Without an eval system, the only evidence that a change improved the system is the developer’s judgment at the time of the change. That judgment is subject to confirmation bias, to incomplete testing, and to the silent failure problem that makes AI system degradation invisible to anyone not looking for it. With an eval system, the evidence is structural: either the system performs better on the measured behaviors, or it does not.

The current state of the field is that the pattern is understood and the adoption is poor. Most AI systems in production today do not have eval-as-CI. Most do not have trace contracts. Most do not have explicit failure budgets. Most discover regressions when users report them. The gap between what is known and what is practiced is the most significant quality problem in AI systems engineering, and it is not a research problem. It is an adoption problem.

This research track is not primarily investigating whether evaluation matters. That question is settled. It is investigating the specific open problems that prevent evaluation from scaling gracefully: trace contract generation, grader stability, eval set maintenance, and economics at scale. Those are the problems where the field is genuinely uncertain, and where better answers would change how production AI systems are built and operated.

RES/EVA · Active Research · Updated as findings develop