Skip to content

Research / Reliability

Reliability — observability for non-deterministic systems.

What "p99" even means for an LLM call graph, and the trace shapes, sampling strategies, and budgets that make non-deterministic systems operable. This page documents what is currently understood, where that understanding breaks down, and what remains genuinely unresolved — as of this writing.

Code RES/RELArea ReliabilityComplexity HighLayer Operations
§ 01The Problem

The instrumentation exists. The assumptions behind it do not hold.

The reliability problem in AI systems is not the absence of tooling. It is the presence of tooling calibrated for a fundamentally different class of system, applied without adaptation. Traces are collected. Dashboards exist. Alerts fire. None of it captures the actual failure mode — outputs that are structurally correct, semantically plausible, and factually wrong in ways that only a domain expert or a scoring model can detect.

Classical observability was built on four assumptions that hold for deterministic systems: a request either succeeds or fails, latency follows a continuous but bounded distribution, errors are countable, and the system is reproducible given identical inputs. LLM systems break each of these by construction. Outputs are probabilistic. Failure is a distribution, not a binary. Latency is bimodal — prompt evaluation and generation have different drivers, different capacity constraints, different optimization paths. The system is non-reproducible by design.

The consequence is a specific kind of instrumentation failure: metrics that are technically present but operationally meaningless. p99 latency that conflates two incompatible distributions. Error rates that count HTTP failures while missing the more prevalent and damaging failure mode. Alerts calibrated to thresholds derived from a baseline that shifts with every model update. The dashboard looks populated. The system is not observed.

Reliability without authority boundaries is incomplete. An agent can produce a technically correct result and still be unsafe to operate if it can act beyond the permissions the task requires. Production reliability therefore includes a bounded execution envelope: explicit permissions, constrained tools, isolated execution, controlled network access, auditable actions, and escalation for consequential decisions. Prompt injection and hostile context have to be treated as operating conditions, not exceptional cases.

This page investigates which classical SRE concepts transfer to LLM systems, which require significant adaptation, and which need to be replaced entirely. The investigation is not complete — several of the questions raised here are genuinely unresolved in the field. That is the honest state of this domain. What follows is what is currently understood and where the understanding runs out.

§ 02What Transfers

State of understanding — current assessment

The following table documents the current state of understanding for each classical SRE concept applied to LLM systems. Transfer status is assessed conservatively: "Partial" means the concept transfers with significant modification, not minor tuning. Confidence reflects how settled this assessment is — "Low" means active disagreement exists in the field, and the assessment here is one position among several contested ones.

ConceptTransfers?Adaptation RequiredConfidence
Distributed tracingPartialSpan context must survive async LLM calls, tool fan-out, and agent re-entry loops. Standard trace propagation headers break at model boundaries.High
p99 latencyPartialLLM latency is bimodal — prompt evaluation and generation are distinct distributions. Aggregate p99 conflates them and misleads capacity planning.Medium
Error ratesNoFailure is an output quality distribution, not a binary HTTP status. A 200 response containing a hallucination is not a success.Low
Sampling strategiesPartialUniform sampling produces a false baseline — rare failure modes are systematically under-represented. Tail-biased and anomaly-triggered sampling are required.Medium
Error budgetsPartialThe budget concept transfers. Defining 'spend' requires a quality scoring function that maps probabilistic output to a scalar — an unsolved problem.Medium
Alerting thresholdsNoStatic thresholds fail against non-deterministic output variance. Threshold derivation requires a baseline distribution, which shifts with every model version.Low

The pattern is consistent across the table: structural concepts — tracing, budgets — survive with modification because their shape transfers even when their details do not. Measurement concepts — error rates, alerting thresholds — do not transfer because they depend on a binary success/failure definition that LLM outputs cannot satisfy. The blocking problem is definitional, not implementational. The field cannot agree on what counts as failure, which means every downstream metric is load-bearing on an unsettled foundation.

§ 03Trace Shapes

Designing span topologies that survive LLM call graphs

The trace topology produced by an LLM call graph is wide, branching, and variable in depth across runs. The same user query may produce three spans or thirty, depending on which tools the model selects, how many retries occur, and whether the orchestrator decides to re-plan. Instrumentation that assumes a fixed topology — as most standard libraries do — produces traces that are structurally present but operationally uninterpretable.

The root span is typically an orchestrator: a planning step that issues a prompt, receives a structured response, and routes to downstream tools based on that response. Each tool call may fan out to parallel sub-calls. Some sub-calls fail and retry — producing sibling spans representing the same logical operation at different points in time. The model itself may be called multiple times within a single logical request as the orchestrator refines its plan. None of this maps cleanly onto the shallow, predictable trees that distributed tracing was designed to capture.

The diagram below shows a representative trace — not a pathological case. This is the topology of a single orchestrated query: plan, fan-out to two parallel tools, synthesize, and write with one retry. In production, the shape varies per request.

TIME →orchestrateROOT · plan + routellm.callmodel: gpt-4o · plantool.searchquery: "..."tool.retrievedoc_id: 44⎧⎩parallelllm.callmodel: · synthesizetool.writeERR · retryattempt 2 · okretryspan (ok)span (error · retried)⎧⎩ parallel fan-out
Fig. RES/REL — LLM call graph trace waterfall. Time flows left to right. Depth increases downward. Fan-out and retry produce variable-width, variable-depth traces across runs.

Three structural properties of this topology require explicit instrumentation decisions. First, span context must propagate through the model boundary. When an orchestrator LLM decides to call a tool, the trace context — trace ID, parent span ID — must flow into that tool call's instrumentation. Standard HTTP middleware handles this automatically for web services; for LLM tool calls it requires explicit context injection at the point of dispatch.

Second, parallel fan-out creates sibling spans with the same parent. This is structurally correct but produces traces where the critical path is not deterministic. Critical path analysis — the primary diagnostic for latency attribution in deterministic systems — does not have a clean analog for parallel fan-out with variable branch counts. This is not a tooling gap. It is a definitional one: the concept of a single critical path does not apply when branching decisions are made at runtime by the model.

Third, retries must be represented as distinct sibling spans, not as a single span with a counter. A failed tool call followed by a successful retry should produce two spans at the same parent position. The distinction matters for latency attribution: the retry latency belongs to the failure event, not to the operation. Collapsing retries into a single span makes the failure invisible to latency analysis.

§ 04Sampling

When uniform sampling produces a false baseline

Sampling is necessary at production scale — retaining 100% of traces is prohibitively expensive. The standard approach is uniform head-based sampling: a fixed percentage of requests are selected for full trace retention at the point of entry. At 1% sampling, one in a hundred requests is retained. The sample is statistically representative of the request distribution.

In deterministic systems, a representative sample of requests is also a representative sample of failures — failures are roughly uniformly distributed across request types. In LLM systems, failure distribution is non-uniform. Certain input patterns, prompt structures, and context lengths produce failure modes that are rare in the overall distribution but critical to detect. Uniform sampling systematically under-represents them. The sample is representative of normal operation and blind to the tail.

Tail-based sampling defers the retention decision until the full trace is available, then samples based on trace characteristics — latency outliers, error spans, output quality score. High-signal traces are retained at higher rates; routine traces are sampled aggressively. The result is a sample biased toward the failure distribution rather than the request distribution. It is not representative in the statistical sense. It is representative in the operational sense — it captures what matters.

The operational complication: quality-based tail sampling requires assessing output quality before the retention decision, which may require a secondary LLM call acting as scorer. The observability layer becomes non-trivially expensive and introduces its own reliability surface. An LLM-based scorer can fail, hallucinate a quality score, or produce inconsistent ratings across equivalent outputs. The cost model for this approach in production LLM systems is not well characterized.

A practical intermediate: anomaly-triggered sampling. Retain traces that exhibit structural anomalies — unexpected span counts, latency outliers, HTTP error codes — at high rates, and sample structurally normal traces uniformly at low rates. This captures mechanical failures without requiring quality assessment at trace time. It remains blind to the hardest class of LLM failure: responses that are structurally unremarkable and semantically wrong. That class requires a quality scorer, and the cost model for a quality scorer remains the open problem.

§ 05Failure Budgets

Adapting error budgets when failure is a distribution

The error budget model from SRE is structurally sound and worth preserving: define a target reliability level, and the gap between that target and perfect reliability is the budget available for change velocity. When the budget is exhausted, release velocity drops until it recovers. The concept creates a forcing function that makes reliability tangible and tradeable rather than abstract and aspirational.

The concept transfers to LLM systems. The mechanism that makes it operational does not. In a web service, budget spend is binary: an error either occurs or it does not, and the budget consumed is proportional to the error rate against the SLO. In an LLM system, spend requires a scalar quality score — a representation of how far a given response was from acceptable — and the budget is consumed proportionally to the quality deficit. That scoring function does not exist in standard form.

Different teams use different scorers: human preference labels, automated metrics (BLEU, ROUGE, BERTScore), LLM-as-judge pipelines, task-specific completion checks. None generalize across task types. None produce a scalar that is comparable across different query categories. A quality score on a summarization task is incommensurable with a quality score on a code generation task, which makes aggregate budget accounting meaningless.

The practical consequence is that most teams running LLM systems today operate on implicit failure budgets — informal tolerance for model regressions that is only made explicit after a high-visibility incident. The budget exists. It is not instrumented. This is structurally identical to running a web service without SLOs and calling it operational maturity.

The Technical Readiness framework treats reliability instrumentation as a precondition for production ownership — not a phase added after the system works, but a condition of the system being considered working. The implication for budgets: an imprecise quality scorer is more useful than no scorer, because it makes the implicit explicit. The precision of the scorer matters less than the discipline of having one.

§ 06Open Questions

The current research frontier

The following questions are unresolved as of this writing. They are not gaps in implementation — they are gaps in conceptual clarity that block implementation. Each represents a point where the field lacks a settled answer and where different practitioners are making different bets without shared empirical grounding.

What is the correct unit of reliability for an LLM response?

Unresolved

HTTP systems use binary success/failure. LLM outputs are a quality distribution. The industry has not converged on a scalar unit — different teams use BLEU scores, human preference rates, task completion rates, or refusal rates. None generalize across task types. Until this is resolved, every downstream reliability metric is built on an undefined foundation.

How do you attribute latency in a multi-hop agent call graph?

Contested

In a synchronous call graph, latency attribution is straightforward. In an agent graph with parallel tool calls, conditional branching, and LLM re-entry, the critical path is not deterministic across runs. Standard critical path analysis does not apply. The implication is that SLO targets for agent systems cannot be set from first principles — they can only be derived empirically, which requires stable enough behavior to sample from.

When does a model version change constitute a reliability event?

Unresolved

Model providers update base models without version bumps. Behavior shifts that would register as incidents in a deterministic system pass undetected. The detection problem requires a continuous behavioral baseline that is expensive to maintain and itself subject to model drift. The field has not agreed on what threshold of behavioral change triggers incident classification.

Can tail-biased sampling be implemented without prohibitive storage cost?

Active research

Tail-biased sampling requires retaining spans until the full trace is complete and quality can be assessed — which may require a scoring LLM call. This makes the observability layer itself non-trivially expensive and introduces a secondary reliability surface. The cost model for tail-based sampling in LLM systems is not well characterized. Practitioners are making different bets on this without shared empirical grounding.

How do you set an error budget when the acceptable failure rate varies by task?

Contested

In a system that handles both low-stakes summarization and high-stakes decision support, a single error budget conflates incompatible failure tolerances. Task-segmented budgets are theoretically correct but operationally complex — they require a task taxonomy, per-segment baselines, and aggregation logic that does not yet exist in standard tooling. The gap between theory and implementation is wide.

The unresolved questions here share a common structure: they cannot be answered by building. They can only be answered by the field converging on definitions — of failure, of spend, of incident — that are stable enough to measure against. That convergence requires empirical accumulation across enough production systems that patterns become visible. The contested questions would resolve immediately if the definitional ones did. The active research question about sampling cost would resolve if the definition of quality were settled enough to make a scorer worth building at scale. The dependency order is fixed: definitions first, metrics second, tooling third. The field is still at definitions. That is the honest state of this domain, and it is what makes this research rather than engineering.

Continue Reading