The assumption built into every general-purpose orchestrator is that workloads are general enough to benefit from shared scheduling primitives. That assumption holds for data pipelines with predictable step shapes and bounded output cardinality. It breaks when steps are non-deterministic, output sizes vary by orders of magnitude depending on generated content, and the correct next step is a function of what the previous step produced. General orchestrators do not fail loudly under these conditions — they fail by making the system harder to debug, harder to reason about, and harder to change. Control flow choice is architectural, not operational. Making it late means making it under pressure.
Three distinct approaches dominate the space: graph runtimes that compile pipelines into execution DAGs and manage scheduling automatically, explicit state machines that enumerate every reachable state and transition, and ad-hoc orchestration that expresses control flow directly in application code. Each makes a different tradeoff between expressiveness, observability, and operational cost. None is strictly dominant. What is poorly understood is exactly where each approach's failure mode becomes load-bearing — where the abstraction stops paying for itself and starts costing more than it saves.
Layered on top of the control flow question is durable execution: the class of systems where step state is checkpointed, process death does not lose progress, and execution resumes from the last committed boundary. Durable execution removes a real class of failure. It also changes what "correct" means — a step must be idempotent by construction, or the replay is wrong. That constraint propagates upward through every design decision in the pipeline.
Distributed systems research has built a deep body of knowledge about workflow execution, failure recovery, and queue topology. The question is which of those primitives transfer directly to LLM orchestration, which require adaptation, and which are misleading because the domain assumptions don't hold.
| Concept | Transfers? | Adaptation Required | Confidence |
|---|---|---|---|
| Saga pattern (compensating transactions) | Partially | LLM steps produce probabilistic outputs — compensation cannot reliably invert a generation. Sagas assume binary commit/rollback; LLM steps have no rollback primitive. | Medium |
| At-least-once delivery | Yes | Applies cleanly. Idempotency must be enforced at step boundaries — replay is a property of the executor, not the LLM. | High |
| DAG-based parallelism | Yes | Fan-out widths are effectively unbounded in LLM pipelines without explicit throttling. Static fan-out graphs assume predictable downstream capacity. | High |
| Backpressure propagation | Partially | Standard backpressure assumes roughly uniform step latency. LLM steps vary 10–100× depending on model, prompt length, and provider load. Upstream queue depth is a poor proxy for downstream pressure. | Low |
| State machine formalism | Partially | Explicit state machines are more observable than graph runtimes but require upfront enumeration of all reachable states — which LLM pipelines resist due to conditional branching on generated content. | Medium |
| Circuit breaking | Yes | Applies cleanly to LLM provider failures. The threshold calibration problem is the same as any high-latency upstream dependency. | High |
| Step-level idempotency | Yes | Required for durable execution replay correctness. The constraint is that LLM calls are not naturally idempotent — the same prompt can produce different outputs. Idempotency must be enforced by caching at the executor layer, not assumed from the step. | High |
| Graph runtime observability | Contested | Graph runtimes abstract control flow behind framework primitives, which reduces observable intent. Handcrafted state machines surface every transition explicitly. The observability tradeoff is not resolved — it depends on whether the primary debugging surface is the framework's trace or the application's state log. | Low |
State of Understanding — RES/ORC · Orchestration Primitives
Three models compete. A graph runtime compiles the pipeline into a DAG at definition time, then schedules execution against that graph — providing parallelism, retry, and observability through framework instrumentation. An explicit state machine enumerates every state and transition upfront, making control flow a first-class data structure that can be inspected, tested, and reasoned about independently of execution. Ad-hoc orchestration expresses control flow directly in application code — no framework, no graph, just conditional logic and function calls.
Graph runtimes win when the pipeline structure is stable and the scheduling problem is non-trivial — when fan-out is wide, when step dependencies are complex, when the same steps appear in multiple pipelines. The framework amortizes scheduling complexity across the system. The cost is indirection: when something breaks, the debugging path runs through framework internals before it reaches application logic. Observability becomes a function of what the framework chooses to expose, not what the application needs to surface.
Explicit state machines win when the pipeline is complex but bounded — when the number of reachable states is large enough to matter but small enough to enumerate. The machine is the specification. Any state it reaches was anticipated. Any transition it fires was deliberate. The operational cost is upfront: every state must be named, every transition must be handled. When generated content drives branching, that enumeration becomes expensive and brittle — a model that produces a new category of output forces a state machine change.
Ad-hoc orchestration wins when the pipeline is simple, when iteration speed matters more than operational rigor, and when the team can hold the entire control flow in their heads. It fails silently — there is no framework to enforce structure, no state machine to reject an unhandled transition. Failures accumulate as implicit state in variables and conditional chains that were never designed to be the control plane. The refactoring cost is paid later, usually during an incident.
The architectural decision is which failure mode is acceptable given the pipeline's stability and the team's debugging preferences. None of the three is wrong in the right context. All three are wrong when chosen by default rather than by deliberate analysis. The relevant reference for evaluating production readiness against this dimension is FRM/04 Technical Readiness.
Durable execution systems checkpoint step state to persistent storage. When a process dies — whether by crash, deployment, or infrastructure failure — execution resumes from the last committed checkpoint rather than from the beginning. This removes an entire class of failure: the in-flight work that disappears when a process exits. The cost is a constraint that propagates through the entire system: every step must be idempotent by construction. Replay is not an edge case — it is the recovery mechanism.
For LLM steps, idempotency is not a natural property. The same prompt sent twice produces different outputs. Durable execution imposes idempotency at the executor layer: the result of a completed LLM call is cached, and replayed execution reads from the cache rather than re-calling the model. This works correctly under the assumption that a cached result is a valid substitute for a fresh generation in the context of a resumed pipeline. That assumption holds for most steps. It fails when the step's correctness depends on freshness — when the generated output was conditional on information that has changed between the original execution and the replay.
The diagram below shows the structural difference between imperative execution and durable execution under a process death event. In imperative execution, process death loses all in-flight state — the pipeline restarts from the beginning or is lost entirely. In durable execution, steps committed before the failure are preserved; execution resumes from the last checkpoint with cached results for completed steps.
Fig. RES/ORC — Durable Execution Checkpoint Map · Imperative vs. durable execution under process death
The hidden cost of durable execution is what it hides about intent. When every step is checkpointable, the system's durability boundary is the framework's checkpoint granularity, not the application's logical transaction boundary. A pipeline author who has not thought carefully about which steps should be atomic gets a system where correctness is a function of checkpoint placement — and checkpoint placement was chosen by the framework, not by the application designer.
Standard backpressure mechanisms — queue depth thresholds, token buckets, semaphore-bounded concurrency — assume that step latency is roughly uniform and that the number of downstream tasks produced by an upstream step is bounded and predictable. LLM pipelines violate both assumptions. Step latency varies 10–100× depending on model, prompt length, and provider conditions. A single step that produces a list of items to process downstream can produce 1 item or 40 depending on the content it generated.
The consequence is that queue depth upstream of a fan-out step is a poor signal for actual downstream pressure. A queue of depth 10 waiting for a list-generation step could be hiding 400 downstream tasks. Backpressure that triggers at queue depth 10 is either too late (the downstream burst has already been committed) or incorrectly calibrated against an average that doesn't represent the actual distribution.
The correct backpressure signal in a variable-width fan-out topology should originate downstream of the fan-out, not upstream of it. The fan-out step should observe available capacity in the downstream queue before committing its output. This requires that fan-out steps be aware of downstream topology — which violates the encapsulation that graph runtimes provide. It is possible in ad-hoc orchestration where control flow is explicit, harder in graph runtimes where fan-out is a framework primitive, and formally incompatible with durable execution if the fan-out step's output has already been checkpointed before the downstream pressure is observable.
The non-determinism compounds here. As documented in RES/REL, reliability measurement in non-deterministic pipelines requires accounting for output distribution, not just pass/fail rates. Backpressure configuration faces the same problem: threshold calibration requires understanding the output cardinality distribution of fan-out steps, which must be measured empirically under realistic load.
Shared queues couple domains at the infrastructure layer. A burst of work in one domain — a large document batch, a spike in agent-initiated sub-tasks — raises queue depth for all consumers. If those consumers share a worker pool, the burst displaces work from other domains regardless of priority configuration. Priority queues address this partially: they can enforce ordering within a shared queue but cannot prevent resource starvation when high-priority work is uniformly high volume.
Per-domain isolation — separate queues, separate consumer pools, separate concurrency limits — eliminates cross-domain coupling at the cost of operational surface area. More queues means more monitoring, more consumer configuration, more places for deployment errors to introduce mismatches between producer and consumer versions. The isolation is real. So is the complexity.
The question that remains open for this system is whether the coupling cost of a shared queue materializes at the throughput levels in operation — tens to low hundreds of pipeline runs per day across a small number of domains. At high throughput, the case for isolation is established. At low throughput, the operational overhead of isolation may exceed the isolation benefit. The break-even point has not been measured for this specific workload profile.
The evaluation pipeline documented in SYS-03 currently uses a single queue across evaluation job types. The domain isolation question will become load-bearing when /systems/automation is operational and automation-triggered evaluations compete with manually-triggered ones for consumer capacity.
These are the questions where current understanding runs out. They are first-class content, not an appendix.
When does a graph runtime's abstraction cost exceed its scheduling benefit?
Graph runtimes provide scheduling, retry, and parallelism out of the box. The cost is indirection — debugging paths through framework internals rather than application logic. The crossover point appears to depend on pipeline stability: volatile pipelines (frequent structural change) favor ad-hoc orchestration; stable pipelines favor the framework's amortized investment. The exact threshold is not established.
Can durable execution semantics be applied selectively within a pipeline?
Durable execution treats all steps as checkpointable by default. In practice, some steps are idempotent and cheap to replay; others are expensive and stateful. Selective durability — checkpointing only at designated boundaries — would reduce replay overhead but requires explicit annotation of durability domains. No production pattern has emerged for this in LLM pipelines.
What is the correct backpressure primitive for a pipeline with variable-width fan-out?
Standard token-bucket and queue-depth backpressure primitives assume bounded downstream capacity. When a single LLM step can produce 1 or 40 downstream tasks depending on generated content, queue depth becomes unrelated to actual load. The backpressure signal needs to originate downstream of fan-out, not upstream of it. No general solution exists.
Does per-domain queue isolation outperform shared queues at the workload sizes relevant to this system?
The coupling cost of shared queues is well-established in high-throughput systems. Whether that coupling materializes at lower throughput — tens to hundreds of pipeline runs per day — is unknown. Isolation adds operational complexity (more queues, more consumers, more monitoring surface). The break-even point for isolation has not been measured.
How should orchestration layer errors compose with evaluation layer signals?
SYS-03 treats pipeline failures as a distinct signal class from evaluation failures — a step that times out is logged differently from a step that produces a low-quality output. Whether orchestration-layer errors should trigger re-evaluation, retry, or circuit-break depends on the failure mode. The decision tree has not been formally specified.
The resolution order is not arbitrary. The backpressure question cannot be answered without first establishing the output cardinality distribution of fan-out steps under realistic load — which requires the automation infrastructure to exist and run long enough to produce a measured distribution. The queue isolation question depends on the same data. The durable execution question about selective checkpoint granularity depends on first knowing which steps are idempotency-critical, which can only be determined empirically after operating the system through multiple failure modes. The control flow architecture question — graph runtime vs. state machine — is the only one resolvable by design decision rather than measurement, and it gates the others: the backpressure primitives available, the checkpoint granularity achievable, and the isolation model expressible are all functions of which control flow model the system adopts. That is what makes this research rather than engineering.