The deploy gate is the system. The dashboard is a side effect.
Most evaluation infrastructure in AI systems is built to answer the question: how is the system performing? The dashboards, metric charts, and score summaries that result are operationally inert. They inform without requiring. A practitioner can observe a declining score, note it, and ship anyway. The information exists; the consequence does not.
This pipeline is built to answer a different question: should this change ship? The architectural consequence of that question is a deploy gate — a structural block that prevents a deploy when evaluation signal indicates regression against the established baseline. Without the gate, the evaluation pipeline is documentation of quality rather than enforcement of it. An eval system without a deploy gate is a dashboard with extra steps.
The system covers every shipped agent and retrieval surface in the lab — the distillation agents documented in the Second Brain build log, the retrieval pipeline, the publishing surface. Each surface has its own fixture set, its own scorer configuration, and its own gate threshold. A regression in one surface does not block deploys to unrelated surfaces.
The conceptual foundation for the pipeline — what "regression" means for a non-deterministic system, why delta-based scoring is necessary, and what remains unresolved in the field — is documented in the Reliability research page. This document describes the operational system, not the research questions behind it.
Three parallel scoring lanes converging at a single gate
The pipeline has four layers. The baseline layer holds versioned trace fixtures — the ground truth every run is measured against. The capture layer records live traces from the runtime. The scoring layer runs three parallel scorer types against each trace, each with different cost, variance, and signal characteristics. The gate layer aggregates scorer outputs into a delta signal and makes the ship/block decision.
The three-lane scoring architecture is the central design decision. Each lane addresses a different failure mode: deterministic scorers catch structural failures at zero cost; reference scorers catch semantic drift against the baseline; model-graded scorers catch quality failures that neither structural nor reference comparison can surface. Running all three on every trace would be prohibitively expensive. The sequencing — deterministic first, reference second, model-graded only on candidates that clear the first two — manages cost without sacrificing coverage.
The diagram reflects the current v1.4 state. The model-graded scorer lane is operational but classified as stabilizing — the variance in LLM-as-judge scoring across equivalent inputs is high enough that its weight in the aggregator is currently reduced relative to the deterministic and reference lanes. The weight will be increased as the scorer's consistency improves. The build log documents the calibration history in detail when it reaches publication — the Evaluation Systems build log is currently a stub.
The baseline layer — what regression is measured against
A fixture is a versioned trace with an associated expected output range. It is the ground truth for a specific input: given this query, this context, and this system configuration, the output should fall within this range. Regression is defined as departure from that range, not departure from some ideal. The fixtures define what the system was, not what it should theoretically be.
Fixtures are promoted from real traces rather than synthetically constructed. The trace recorder captures every span in production with full fidelity — LLM inputs, outputs, tool call arguments, latency at each node. A captured trace becomes a fixture candidate when its output is manually reviewed and accepted as representative of correct behavior. The promotion step is the quality gate for the baseline itself.
Fixture versioning is intentional. When the underlying system changes in a way that legitimately changes expected behavior — a retrieval architecture change, a prompt revision, a model update — the fixture set is versioned to reflect the new baseline. The gate does not block a deploy because the system changed; it blocks a deploy because the system changed in a direction that was not expected. The version boundary is explicit and documented.
The trace recorder operates at zero sampling rate for fixtures — every trace that is a fixture candidate is recorded in full. For non-fixture traces, the recorder uses anomaly-triggered sampling: structurally abnormal traces are retained, structurally normal traces are sampled at a low rate. The sampling strategy is borrowed from the approach described in the Reliability research page, which also documents why uniform sampling fails for LLM systems.
Three lanes — cost, variance, and failure mode coverage
The scorer set is designed around one constraint: no single scorer covers all failure modes, and running all scorers on all traces is not economically viable. The three-lane architecture allocates scoring cost proportional to the signal value of each lane for a given trace type.
Runs on every trace without exception. Schema validation confirms that outputs conform to expected structure — a required field is present, a JSON envelope is valid, a numeric output is within range. Null checks confirm that the system did not silently fail. Latency bounds confirm that the system did not time out without surfacing an error. Deterministic scorer failures are treated as non-negotiable blocks regardless of what the other scorers report — a system that is structurally broken is not partially correct.
Runs on traces that clear the deterministic gate. Compares the live trace output against the corresponding fixture using three methods: string matching for exact-match assertions, embedding similarity for semantic drift detection, and structured diff for outputs with a defined schema. Each method produces a score. The reference scorer is the primary regression signal for retrieval quality — a change in which documents surface for a given query is detectable via embedding similarity against the fixture output even if the final generated text differs.
Applied only to traces that clear both the deterministic and reference gates, and only to surface types where semantic quality is the primary regression risk — the distillation agents and the publishing surface, not the retrieval layer. An LLM evaluator reads the trace output and scores it against a rubric. The rubric is explicit: faithfulness to source material, internal consistency, completeness relative to the query. The scorer is currently stabilizing — inter-run variance on identical inputs is the primary calibration problem. The research questions underlying this approach are documented in the Reliability research page; the field-level question of what constitutes a valid quality unit for LLM output remains unresolved.
The evaluation research questions — what constitutes a valid quality unit, how scorer confidence should be weighted, whether model-graded scoring can achieve acceptable inter-run consistency — are tracked in the research layer. The Evaluation research page is currently a stub; these questions will be documented there as the investigation matures.
Delta, not pass/fail — why the distinction matters operationally
A pass/fail gate blocks deploys when the absolute score falls below a threshold. The threshold is set once, early, against a system that may not yet be representative. As the system improves, the threshold becomes easy to clear. As the system drifts, the threshold may still be cleared by a system that is meaningfully worse than it was at a prior version. The absolute threshold measures current state against a fixed expectation. It does not measure change.
The regression gate measures delta: the difference between the current score and the score recorded at the last accepted baseline. A system that has always been mediocre does not fail the gate. A system that was good and became worse does. The gate enforces the direction of change, not the absolute position. This makes it sensitive to regressions at any quality level and insensitive to quality levels that are stable — which is the correct behavior for a deploy gate.
The gate threshold is per-surface and per-scorer-lane. A one-point delta in deterministic scorer output triggers a block unconditionally — structural failures have no acceptable regression. A delta in reference scorer output triggers a block at a configurable threshold, currently set at five percentage points of embedding similarity. The model-graded scorer contributes to the aggregate but does not independently trigger a block while it remains in stabilizing status.
The gate was added after a two-week silent regression in the distillation agent — a change to the prompt structure degraded output quality without producing any structural errors or latency changes that would have been visible in the pre-gate monitoring. The regression was discovered through user observation, not instrumentation. Contract tests covering that prompt structure are now part of the deterministic scorer set. The gate is the structural consequence of that incident.
Component reference — SYS-03 v1.4
Current operational state of all system components. Model-graded scorer (EVL-MGS) is active but classified as stabilizing — inter-run variance on equivalent inputs has not yet reached the consistency threshold required for full gate weight. All other components are stable.
| Code | Name | Layer | Status | Description |
|---|---|---|---|---|
| EVL-FIX | Fixture Store | Baseline | Active | Versioned trace fixtures with expected output ranges. Each fixture is a recorded real trace promoted to canonical reference — the regression baseline everything else measures against. |
| EVL-REC | Trace Recorder | Capture | Active | Lossless span capture across the runtime. Records full LLM inputs, outputs, tool calls, and latency at every node. The raw material for fixture promotion and live regression. |
| EVL-DET | Deterministic Scorer | Scoring | Active | Schema validation, null checks, format assertions, latency bounds. Runs on every trace at zero LLM cost. Catches structural failures before they reach the higher-cost scorers. |
| EVL-REF | Reference Scorer | Scoring | Active | Output comparison against fixture reference via string matching, embedding similarity, and structured diff. Medium cost, medium variance — the primary signal for retrieval regression. |
| EVL-MGS | Model-Graded Scorer | Scoring | Stabilizing | LLM-as-judge semantic quality scoring. Highest signal, highest cost, highest variance. Applied selectively — not on every trace, only on candidates that clear the deterministic and reference gates. |
| EVL-AGG | Score Aggregator | Aggregation | Active | Merges scorer outputs into a single delta signal per component. Weighted by scorer reliability tier. Deterministic scorer failures are non-negotiable blocks regardless of other scores. |
| EVL-GATE | Regression Gate | Gate | Active | Blocks deploy when score delta exceeds threshold against the versioned baseline. Driven by delta, not absolute score — a system that was always mediocre does not fail the gate; a system that regressed does. |
| EVL-DASH | Dashboard | Surface | Active | Minimal surface — regression visibility only. Not a metrics browser. One view: which components regressed, by how much, against which fixture version. |
v1.4 status. The gate, aggregator, deterministic scorer, and reference scorer are stable and operating without manual intervention. The model-graded scorer is active but underweighted pending consistency calibration. Fixture coverage currently spans the distillation agent and retrieval pipeline surfaces. The publishing surface fixture set is in progress — three of seven planned fixture families are complete.
Known gap. Contract test coverage is incomplete. The prompt structure regression that triggered the gate's addition was the first contract test added. Coverage is expanding incrementally but the deterministic scorer set does not yet fully cover the system's behavioral surface. This is the primary operational risk at v1.4 — a regression in an uncovered prompt structure would still require user observation to surface.
Technical Readiness posture. The Technical Readiness framework treats evaluation infrastructure as a production ownership condition. At v1.4, this system meets the minimum bar — a gate exists, it is connected to the deploy path, and it has caught regressions in use. It does not yet meet the full bar: coverage is incomplete, the model-graded scorer is not at full weight, and the fixture set has gaps. The system is operational. It is not complete.
Continue Reading