Skip to content

BLD/EVA · Build Log

Evaluation Systems — eval-as-CI for everything that ships.

Building a trace-backed regression net behind every shipped agent and retrieval surface, run on every change like a test suite.

Iteration 009Phase StabilizingSurface Quality planeStatus v1.4
§ 01Opening

At iteration seven, a schema change in upstream trace output silently dropped 11% of eval runs for two weeks. The fixtures expected a field that had been renamed. The fixture matcher logged non-matches as skips, not failures. The eval suite reported green. Nothing in the output indicated that over a tenth of the regression surface had stopped running. The discovery came from noticing that fixture coverage numbers had drifted — not from any alarm the system fired.

That incident is the clearest statement of what the evaluation system is for and what it is still failing to guarantee. The system exists to make regressions loud and immediate. When a component of the system itself degrades silently, it has the same failure mode as the systems it monitors. The gap between what the eval suite covers and what it reports covering is the most important operational metric this system tracks — and the one with the most unresolved work remaining.

Nine iterations in, the system runs on every commit, blocks deploys on quality delta, and operates across trace steps rather than final outputs. The fixture coverage gap, the contract test incomplete coverage, and the model-graded scorer variance are open. This log documents how the system arrived at its current state, what each iteration broke, and what has not been resolved.

§ 02Why This

Spot checks catch known regressions — the ones you thought to test for. A regression net catches unknown ones — the behavior changes that would not have been on the test list because the failure mode was not anticipated. The distinction matters most for LLM-backed systems, where the output space is effectively unbounded and the failure modes are often subtle degradations rather than hard errors. A hard error is loud. A retrieval system that returns results 12% less relevant than it did last week is silent until someone notices the output quality has changed.

The second motivating constraint is operational velocity. A system with no automated eval surface cannot change quickly without accumulating undetected regressions. Every prompt change, every retrieval parameter adjustment, every model swap is a potential regression. Without a gate that runs on every change, the cost of change is paid later — during incident review, not during development. The eval-as-CI model moves that cost to where it is cheapest: at the point of change, before the change ships.

The architecture decisions that followed from this framing are documented in SYS-03. The research questions that the system operationalizes — and the ones it cannot yet answer — are in RES/EVA.

§ 03Iteration History

Each iteration was forced by a failure in the previous one. The system did not grow by addition of planned features — it grew by encountering the failure mode that the current architecture could not handle, and building the layer that addressed it.

I01Manual spot checks

No automated eval. Output reviewed manually after each significant prompt change. No baseline, no fixture, no signal across runs.

Broke →

Regressions invisible. A change that degraded retrieval quality by 15% shipped undetected because the manual review sampled the wrong queries.

I02First automated scorer

Single format scorer applied to a fixed 20-query set. Checked output structure, not semantic quality. Ran manually before deploys.

Broke →

Coverage theater. Format passing while semantic quality drifted. Manual trigger meant it was skipped under time pressure.

I03Fixture anchoring

Query set replaced with versioned fixture files. Each fixture: input, expected output shape, ground truth label. Run on every commit.

Broke →

Fixtures drifted from production distribution over six weeks as prompts evolved. Tests passed against stale reality.

I04Semantic scorer layer

Added embedding-similarity scorer alongside format scorer. Two-layer check: structure + semantic alignment. First signal on content regression.

Broke →

Scorer threshold set by intuition. Too tight — false failures blocked valid changes. Too loose — real regressions passed. Calibration had no principled basis.

I05Delta gating

Shifted from threshold gates to delta gates. Block if current run scores more than N% below the trailing 5-run average. Self-calibrating baseline.

Broke →

Delta gate sensitive to fixture set composition. Adding new hard fixtures shifted the baseline and triggered false blocks. Fixture changes required gate recalibration.

I06Model-graded scorer

Third scorer layer: LLM judge evaluating semantic correctness against reference answers. Caught reasoning failures the embedding scorer missed.

Broke →

Model-graded scorer introduced non-determinism into the eval itself. Same fixture, same output, different scores across runs. Eval reliability became a question about the eval.

I07Trace ingestion

Eval moved from scoring final outputs to scoring intermediate trace steps. Step-level signal for multi-step pipelines. Fixture schema extended to capture full trace.

Broke →

Schema drift in upstream trace format silently dropped 11% of eval runs. Fixture matcher couldn't match traces with the new schema version. Failures logged as skips, not errors. Two weeks before detection.

I08Contract tests added

Trace ingestion contract test added to CI. Schema version checked before eval runs. Mismatch fails the build before eval fires, not after.

Broke →

Contract test covered the schema version field but not field-level type changes. A field narrowed from string to enum — contract passed, eval skipped affected fixtures silently.

I09Stabilizing

Contract test expanded to field-level type validation. Coverage reporting added — active fixture count, scorer hits per fixture, skip rate by fixture class. Current operational state.

Broke →

Coverage reporting surfaces gaps but does not close them. Fixture set covers 68% of production query distribution. The remaining 32% is known but unaddressed.

§ 04Trace Fixtures

A fixture is a versioned test case: a specific input, the expected output shape, and optionally a ground truth label against which scorers evaluate the actual output. Fixtures are the anchor that makes regression detection possible — without a stable baseline to compare against, there is no signal for whether quality has changed, only whether the current output is above some absolute threshold.

Fixture quality has two dimensions. Coverage: does the fixture set represent the production query distribution? Freshness: does each fixture reflect the current expected behavior of the system, not the expected behavior from three prompt versions ago? Both degrade silently. Coverage gaps mean regressions in the uncovered space go undetected. Freshness failures mean fixtures block valid changes by enforcing stale expectations.

The current fixture management process: fixtures are stored as versioned JSON files co-located with the test runner. New fixtures are added when a new query class enters production. Fixtures are reviewed — but not automatically invalidated — when prompts change. The review dependency is manual: a prompt change does not trigger a fixture audit automatically. This is the primary mechanism by which fixture staleness accumulates undetected.

The schema contract failure at I07 exposed a second fixture fragility: fixtures are written against the trace schema at the time of fixture creation. Upstream schema changes that rename or restructure fields break fixture matching without surfacing as test failures — they surface as skips, which are counted but not alarmed on. The contract tests added at I08 catch schema version changes and field type changes. They do not catch field renaming. The non-determinism of upstream trace formats, and what it means for eval reliability, is an open question in RES/REL.

§ 05Scorer Set Composition

Three scorer layers operate in sequence. Each layer is designed to catch a failure class the previous layer misses. Sequential execution — not parallel — means the most expensive layer only fires on outputs that passed the cheaper checks. Signal density without proportional cost.

The format scorer checks output structure: required fields present, types correct, length within bounds. It is cheap, deterministic, and catches the failure class that makes all downstream scoring unreliable — malformed output that looks like a valid result. A semantic scorer run against a structurally broken output produces a false low score against a valid schema expectation, not a useful signal.

The semantic scorer applies embedding similarity between the actual output and the ground truth label. It catches content drift — outputs that are structurally valid but semantically distant from the expected answer. The calibration problem at I04 (threshold set by intuition) was resolved by shifting from absolute threshold to delta gating at I05. The semantic scorer now has no fixed pass threshold — it contributes to the delta calculation against the trailing baseline.

The model-graded scorer evaluates reasoning quality — correctness, coherence, and completeness against a reference answer — using an LLM judge. It catches the failure class that embedding similarity misses: an output that is semantically adjacent to the ground truth but logically wrong. The non-determinism this layer introduces into the eval system is the most structurally uncomfortable open problem. An eval that is itself non-deterministic creates a measurement floor below which regression detection is noise rather than signal.

SILENT DRIFTI01I02I03I04I05I06I07I08I09Fixture anchoringSemantic scorerModel-gradedContract testsActivePartialInactive

Fig. BLD/EVA — Iteration Health Matrix · Component activation across nine iterations · shaded region = silent schema drift window

§ 06Regression Gates

The gate that blocks a deploy is a delta gate, not a threshold gate. A threshold gate blocks when the current score falls below a fixed value. A delta gate blocks when the current score falls more than N% below the trailing 5-run average. The distinction matters because threshold gates require recalibration whenever the fixture set changes — adding harder fixtures lowers the absolute score without representing a regression. A delta gate self-calibrates: the baseline adjusts to the current fixture set composition, and a genuine regression registers as a delta, not as a score level.

The tradeoff is sensitivity to fixture set changes used as a regression signal. If a fixture is added that the current system reliably fails, the trailing baseline incorporates that failure over five runs, and the gate stops blocking on it. The delta gate treats the system's current performance as the baseline, not a quality standard. A system that has been consistently failing new fixtures for five runs will not be blocked by the gate — the delta is zero. This is a known limitation. The gate blocks regressions — quality decreases from a recent baseline. It does not block inadequate quality — performance below an absolute standard.

The two gate conditions are complementary, not substitutes. Absolute threshold gates and delta gates answer different questions: is the system above the minimum acceptable quality, and has quality declined from recent baseline. Both are needed. Currently only the delta gate is implemented. The minimum quality floor is enforced by the format scorer — the gate only fires if the output passes structural checks — but the semantic quality floor is not separately enforced.

§ 07Architecture Decisions

Decisions that shaped the current system, with current status and rationale.

Stable
In review
In design
Contested
AreaDecisionStatusRationale
Fixture versioningFixtures stored as versioned JSON, co-located with test runner, reviewed on prompt changesStableFixtures must evolve with the system they test. Co-location makes the review path explicit — a prompt change without a fixture review is a visible gap, not an invisible one.
Scorer compositionThree layers in order: format → semantic → model-graded. Each layer required to pass before the next fires.StableSequential layering reduces model-graded scorer invocations — the most expensive layer only fires on outputs that passed structural and semantic checks. Signal density without cost overhead.
Gate trigger conditionDelta gate against 5-run trailing average, not static thresholdStableStatic thresholds require recalibration when fixture composition changes. Delta gates self-calibrate against recent baseline. The tradeoff is sensitivity to fixture set changes — managed by treating fixture changes as gate recalibration events.
Contract test placementSchema contract tests run before eval, failing the build on mismatchIn reviewThe I07 incident established that schema drift is silent under the current skip-on-mismatch behavior. Build-blocking contract tests make the failure loud at the point of ingestion, not two weeks later.
Model-graded scorerSingle judge model with fixed temperature, results averaged across 3 runsContestedAveraging across 3 runs reduces variance but triples cost. Whether 3 runs produces meaningfully lower variance than 1 run has not been measured for this specific scorer and fixture distribution. The open question in RES/EVA on scoring methodology applies directly.
Trace ingestion contractField-level type validation on all required fields, version field checked separatelyIn reviewThe I08 failure showed schema version alone is insufficient. Field-level validation catches type narrowing and field removal. Still does not cover semantic changes — a field renamed with the same type passes the contract and breaks fixture matching.
Coverage reportingCoverage reported as: active fixture count, scorer hit rate per fixture class, skip rateIn designCurrent coverage reporting shows what is covered. It does not show what is not covered. The gap between fixture distribution and production query distribution requires a different measurement: sampling production traffic and comparing against fixture representation.
§ 08Open Questions

What remains unresolved after nine iterations. These are not aspirational improvements — they are gaps that limit the system's reliability as a regression detector.

Unresolved

How should semantic field changes in upstream traces be caught before they break fixture matching?

Contract tests catch schema version drift and type changes. They do not catch semantic changes — a field renamed from 'response_text' to 'output_text' with the same type passes all contract checks and silently breaks every fixture that references the old field name. The fix requires either semantic contract tests (which require knowing the field's purpose, not just its type) or fixture-level validation that fails loudly on zero-match rates rather than silently incrementing a skip counter.

Contested

What is the correct sample size for model-graded scorer averaging to reduce variance without proportional cost growth?

The current 3-run average is a judgment call, not a measured decision. The variance reduction from 1 run to 3 runs has not been characterized for this scorer and fixture set. If variance is low — if the same judge consistently scores the same output similarly — 3 runs adds cost without meaningful signal improvement. If variance is high, 3 runs may be insufficient. The measurement required to answer this is straightforward but has not been run.

Active research

How should the eval system handle fixture obsolescence as production prompts evolve?

Fixtures written against an earlier prompt version may test behavior that the current system no longer exhibits — not because the system regressed, but because the expected output shape has legitimately changed. A fixture that was ground truth at I03 may be measuring a stale expectation at I09. No mechanism currently exists for marking fixtures as obsolete versus marking them as regression signals. The distinction matters: an obsolete fixture failing is expected; a regression fixture failing is a block.

Unresolved

At what fixture coverage percentage does the eval system provide reliable regression signal?

The current 68% coverage of production query distribution is a measured gap. Whether 68% is sufficient to catch the majority of meaningful regressions, or whether the uncovered 32% contains the failure modes that matter most, is unknown. Coverage percentage is the wrong metric — coverage of failure-prone query classes is the right one. Characterizing which query classes are failure-prone requires running the system through more regression events and tracking which ones the eval caught versus missed.

BLD/EVA · ITERATION 009 · STABILIZING

The regression net runs on every commit and blocks on delta. The silent drop problem has a partial fix. The fixture coverage gap is measured but not closed. The model-graded scorer variance is known and unresolved. Stabilizing means the system is reliable enough to trust for the cases it covers — not that the open cases are closed.