The first version used a single queue for everything. One pool of workers consumed ingestion, summarization, and dispatch jobs from the same depth-bounded queue. This worked until a large document batch entered the ingestion pipeline and filled the queue. Summarization and dispatch jobs waited behind ingestion items for hours. Priority classes were added. They changed the order of displacement without eliminating it — high-volume low-priority ingestion refilled the queue faster than high-priority items could drain. The queue was the problem. Sharing it was the problem. Priority configuration was not the fix.
Eight iterations later, the system runs per-domain queues for summarization, separate queues for ingestion and dispatch, a structured dead-letter queue with full failure context, and a contract validator at every input boundary. The displacement problem is solved. The backpressure problem under concurrent fan-out is not — the fix requires coordination primitives the architecture does not currently support. This log documents the path from one queue to eight, what broke at each step, and what remains open.
Background workers are infrastructure. Their failure mode is silent — a worker that stops processing does not raise an alarm unless something is watching. A dispatch job that sits in a queue for six hours because ingestion filled it looks identical to a dispatch job that has not been enqueued yet. Both are invisible until the output fails to appear.
The design constraint was isolation, not throughput. Maximum throughput is achievable with a single shared queue and a large worker pool — but it means every domain competes for the same resource, and any domain's burst affects every other domain's latency. Isolation means a burst in one domain is bounded to that domain. The operational cost is more queues, more consumer configuration, more monitoring surface. That cost is predictable. Cross-domain displacement is unpredictable and invisible until it affects output.
The orchestration primitives that underlie these design choices — particularly the backpressure question and the per-domain isolation tradeoff — are investigated in RES/ORC. This log documents what was built. RES/ORC documents what the remaining open problems are and what resolving them requires.
Each iteration added the layer that the previous architecture could not provide. The sequence is not planned — it is the order in which failures became load-bearing enough to force a structural change.
All work types — ingestion, summarization, dispatch — consumed from one queue. One worker pool with no domain awareness. Simple to operate, simple to monitor.
A large ingestion batch filled the queue depth ceiling. Summarization and dispatch jobs waited behind ingestion items for several hours. No mechanism existed to give priority to time-sensitive dispatch work. The queue was a FIFO with no isolation primitive.
Added three priority levels to the shared queue. Dispatch marked high priority, ingestion marked low. Expected to resolve displacement under burst conditions.
Priority inversion. High-volume low-priority ingestion kept re-entering the queue faster than the high-priority items could drain. The queue was bounded; ingestion refilled it before dispatch items were consumed. Priority classes do not prevent volume displacement — they change the order of displacement, not the fact of it.
Pulled dispatch into its own queue with its own consumer pool. Ingestion and summarization still shared. Dispatch latency stabilized immediately.
Summarization displacement from ingestion bursts persisted. The root problem — two work types competing for the same consumer pool — was only partially addressed. The fix for dispatch revealed the same problem in the remaining shared queue.
Pulled ingestion into its own queue. Three queues total: ingestion, summarization, dispatch. First fully isolated topology. Each work type had dedicated consumers.
Summarization within a domain could still displace summarization in another domain. A large content batch filled the summarization queue and slowed research summarization. Domain isolation within the summarization layer was the remaining coupling point.
Split the summarization queue by domain. One queue per domain (research, content, system), each with a bounded consumer pool. Cross-domain displacement eliminated.
Operational surface increased sharply. Five queues required five sets of depth alarms, five consumer health checks, five consumer version pins. Configuration drift between domains became the new failure mode — a consumer version mismatch in one domain silently degraded that domain's throughput.
Added CTR-01 schema validation at the ingestion boundary. Malformed items rejected to a holding area rather than propagating to domain queues. First explicit contract enforcement.
The holding area had no triage process. Rejected items accumulated without any classification of why they were rejected. The holding area became a silent accumulation point — items entered but nothing monitored the rejection rate or failure types.
Replaced the holding area with a structured DLQ. Each item retained full payload, failure reason, pipeline stage, and retry count. Retention window set to 14 days. DLQ depth monitored as an operational metric.
DLQ triage was manual and undisciplined. Failure type distribution was not tracked. A spike in classification failures went unnoticed for a week because no one was checking the DLQ failure type breakdown. The DLQ surfaced failures but did not classify them automatically.
Added SUM-02 fan-out controller to manage parallel sub-task expansion from summarization steps. Backpressure check added against downstream queue depth before fan-out commit. Current operational state.
Backpressure check is point-in-time — it reads queue depth but does not account for in-flight items already committed by concurrent workers. Under parallel fan-out from multiple domains, the check can pass simultaneously for all workers while the downstream queue fills faster than it drains. Not data loss in practice — queue depth limits are conservative — but the correct primitive is a capacity reservation, not a depth read.
The migration from one queue to per-domain isolation happened in three steps: dispatch separated first (I03), then ingestion (I04), then summarization split by domain (I05). Each step was forced by the displacement failure that the previous step left unresolved. The end state — eight queues across three worker layers — was not the plan at I01. It was the minimum isolation required to contain domain bursts within domain boundaries.
The diagram below shows the queue topology at three inflection points: I01 (single shared queue), I04 (ingestion and dispatch isolated, summarization shared), and I08 (current full per-domain isolation with dead-letter lane).
Fig. BLD/AUT — Queue Architecture Evolution · I01 shared → I04 partial isolation → I08 full per-domain isolation
The operational cost of the final topology is real: eight queues require eight consumer health checks, eight depth alarms, and eight consumer version pins. Configuration drift between domains emerged as the new failure mode at I05 — a consumer version mismatch in one domain silently degraded that domain's throughput without affecting the others. Consumer version pinning and deployment coordination across domains is now the primary operational discipline the architecture requires.
The dead-letter queue started as a holding area for rejected items — a place to put failures that could not be processed rather than discard them. It had no structure, no retention policy, and no triage process. Items entered it and stayed there until someone manually inspected it, which happened irregularly.
The failure that forced formalizing the DLQ at I07 was a week of undetected classification failures. A classifier model update changed the output schema for a class of inputs. The contract validator rejected those items. They accumulated in the holding area. No alarm fired. The failure surfaced when the content domain's summarization queue went dry — items that should have been classified and routed were sitting in the holding area unprocessed.
The formalized DLQ records the full payload, the failure reason, the pipeline stage at which failure occurred, and the retry count. DLQ depth is monitored as a primary operational metric. Spikes in DLQ depth are the most reliable signal that something upstream has changed in a way the contract did not anticipate. The evaluation system's schema drift incident documented in BLD/EVA is the same failure class — upstream schema change, silent drop, delayed detection. The DLQ is the mechanism that makes that failure class loud rather than invisible.
The triage process remains manual. Automated failure classification — reading the DLQ failure reason fields and computing rate by type over a rolling window — is in design. Until it ships, DLQ triage depends on someone looking at it on a weekly cadence. The weeks when no one looks are the weeks where a failure pattern accumulates undetected.
A worker should own no persistent state. State lives in queue items. A worker that accumulates state — caching results between runs, maintaining a local record of previous items, building an in-memory index — has a lifecycle that the queue architecture does not manage. When it restarts, its state is gone. When two instances run concurrently, their states diverge. The worker has become a stateful service that happens to be deployed as a worker.
The constraint has been navigated by a simple test: if a worker can be killed and restarted mid-run without losing any information that isn't in the queue item, it is a worker. If it cannot, it has accumulated state that belongs somewhere else. This test has prevented worker scope creep in the ingestion and dispatch layers. It has been harder to enforce in the summarization layer, where caching distillation results for re-use across related items is operationally attractive.
The boundary question becomes structural when a worker begins containing branching logic. A worker that makes decisions about which of two downstream queues to route an item to based on the item's content is doing classification work. That work belongs in ING-02, not in the worker. A worker that retries a step with different parameters based on the failure reason is doing orchestration work. That work belongs in the fan-out controller or in a purpose-built retry handler. The recurring design discipline is to push branching logic upstream into the appropriate layer rather than accumulate it in the worker that first encountered the need for it.
Decisions that shaped the current system, with current status and rationale.
| Area | Decision | Status | Rationale |
|---|---|---|---|
| Queue isolation model | Per-domain queues for summarization. Shared queues for ingestion and dispatch. | Stable | Domain bursts in summarization are the primary displacement source. Ingestion and dispatch do not exhibit domain-specific burst patterns at current volumes — shared queues are sufficient and avoid unnecessary operational surface area. |
| Domain classification | Rule-based for structured inputs. Model-assisted for unstructured content. Misclassification routes to review queue, not DLQ. | Stable | Misclassification is a system correction problem, not a data quality problem. Routing to review rather than dead-letter preserves the item for re-classification and prevents misclassified items from accumulating in the DLQ alongside genuine failures. |
| Backpressure primitive | Point-in-time queue depth check before fan-out commit. | Contested | Known to be insufficient under concurrent fan-out. The correct primitive is a capacity reservation ahead of commit. Capacity reservation requires coordination between workers — which the current architecture does not support. The depth check is the implementable approximation. The gap is documented in RES/ORC. |
| DLQ retention and classification | 14-day retention. Full payload + failure reason + stage + retry count per item. Manual triage on weekly cadence. | In review | 14-day window is sufficient to catch the majority of failure patterns before they are overwritten. Manual triage is the gap — automated failure type classification is in design. Until it ships, failure rate trends require manual monitoring. |
| Worker state ownership | Workers own no persistent state. All state lives in queue items and the DLQ. | Stable | A worker that owns persistent state is no longer stateless — it has a lifecycle, a failure mode, and a recovery path that the queue architecture does not manage. The constraint forces state to be explicit in the queue item, which makes it inspectable, retryable, and recoverable by the DLQ. |
| Contract enforcement placement | CTR-01 at the ingestion boundary. DSP-03 at the dispatch boundary. No contract on summarization output. | In review | The summarization output contract gap is the next expected failure surface. A schema change in summarization output propagates to the dispatch queue without validation. Adding a summarization output contract requires specifying the output shape formally — which has not been done. |
| Fan-out controller design | SUM-02 manages fan-out centrally, not per-worker. | Stable | Per-worker fan-out without a controller produces uncoordinated expansion — each worker expands independently with no shared view of downstream capacity. A central controller is the minimum coordination required for a meaningful backpressure check, even if the current check is insufficient. |
What remains unresolved after eight iterations. These are not planned features — they are architectural gaps that limit the system's reliability under conditions it will encounter as automation volume grows.
What is the correct backpressure primitive for concurrent fan-out across multiple domain workers?
The depth check fails under concurrent fan-out because multiple workers read the same depth value simultaneously, all pass the check, and all commit fan-out output before any of it is consumed. A capacity reservation primitive — where each worker claims downstream capacity before committing — would resolve this but requires worker coordination that the current architecture does not support. Implementing it requires either a shared reservation store or a fan-out gate that serializes commits. Neither has been designed.
At what throughput does the shared dispatch queue require domain isolation?
The current dispatch queue is shared across all domains. The argument for isolation — that domain bursts in dispatch would displace time-sensitive work — has not materialized at current volumes. Whether it will materialize as automation-triggered dispatch volume grows is unknown. The answer requires measuring dispatch queue depth and latency under peak automation load, which requires the automation workflows to be running at sufficient volume to produce a realistic peak.
How should the DLQ surface failure rate trends without requiring manual inspection?
The DLQ contains the classification required for automated trend detection — failure stage, failure reason, item type are all present. The gap is a triage worker that reads these fields, computes rate by type over a rolling window, and surfaces spikes before they accumulate. DLQ-02 is in design. The design question is whether the triage worker's classification logic should be rule-based against the existing structured fields or model-assisted for unstructured failure reasons. Rule-based is deterministic and fast; model-assisted handles the failure reasons that do not fit the structured taxonomy.
When does a worker's operational complexity justify redesigning it as an orchestrated sub-pipeline?
The worker autonomy boundary — how much logic a worker should contain before it is reclassified as a pipeline — has been navigated by intuition rather than a formal criterion. The current workers are simple: consume, transform, emit or reject. If a worker begins containing branching logic, conditional retries, or sub-step state, the question is whether it should be refactored into a small pipeline with its own queue topology. No criterion for this transition has been formally specified. The risk of not specifying it is workers that accumulate logic incrementally until they are pipelines in all but name.
BLD/AUT · ITERATION 008 · OPERATIONAL
Per-domain isolation is working. Cross-domain displacement is eliminated. The DLQ is a first-class observability surface. The backpressure primitive under concurrent fan-out is insufficient and known to be. The worker autonomy boundary holds in the ingestion and dispatch layers; it is under pressure in summarization. Operational means the system runs and the known gaps have not yet produced data loss — not that the gaps are closed.