The original automation design used a single shared queue for all background work. Ingestion, summarization, and dispatch competed for the same consumer pool. This worked until the first large ingestion batch — a document set that filled the queue and displaced every pending summarization and dispatch job for several hours. The failure was not a bug. It was the correct behavior of a system with no isolation primitive. The only fix was architectural: separate queues, separate consumers, separate concurrency limits per domain.
The current system runs three worker layers — ingestion, summarization, dispatch — each with per-domain queue isolation where the workload requires it. Every item entering the system passes a contract validator before any processing fires. Failed items route to a dead-letter queue with enough diagnostic context to determine root cause without re-running the pipeline. The system is designed for isolation and recoverability. Maximum throughput was not the design constraint.
The open problems are in the fan-out layer. The backpressure primitive that RES/ORC identifies as unresolved — a point-in-time depth check that does not account for in-flight committed items — is the check this system uses. It is known to be insufficient under concurrent fan-out. The operational consequence has been queue depth spikes rather than data loss, but the fix has not been implemented.
Four layers in sequence: ingestion validates and classifies inbound items; per-domain queues isolate work by domain before summarization begins; the dispatch queue holds outbound work after summarization completes; the dead-letter queue receives items that failed at any stage after retry exhaustion. The pipeline is linear for the happy path. Failure paths are explicit at every boundary.
Fig. SYS-05 — Worker Pipeline Topology · Ingestion → per-domain queues → summarization → dispatch · dead-letter recovery lane below
Every item entering the system passes through CTR-01 before classification. The contract validator checks schema version, field presence, and type constraints. A contract failure at this boundary is loud: the item is rejected to the dead-letter queue with the full payload and failure reason. Nothing downstream processes an item that failed contract validation.
ING-02 classifies each validated item into a domain and routes it to the corresponding queue. Classification is deterministic for structured inputs — items with an explicit domain tag are routed directly. For unstructured content, a model-assisted classifier assigns the domain. Misclassification has a specific handling path: items route to a review queue, not to the dead-letter queue, because misclassification is a system correction problem, not a data quality problem. Review queue items are re-classified on the next triage cycle.
The ingestion layer has no throughput throttle at the inbound seam. Rate limiting is enforced at the source — the systems that call the ingestion endpoint are responsible for their own rate. The ingestion worker pools expand to available concurrency within their per-domain limits. This design assumes that inbound rate is controlled upstream; if that assumption breaks, the contract validator becomes the first point of backpressure by default.
| Code | Name | Status | Description |
|---|---|---|---|
| ING-01 | Ingestion Worker | Stable | Consumes from the inbound stream. Validates input against the ingestion contract before any processing fires. Rejects malformed inputs to the dead-letter queue rather than propagating them downstream. |
| ING-02 | Content Classifier | Stable | Classifies each ingested item into a domain and routes it to the appropriate per-domain queue. Classification is rule-based for structured inputs and model-assisted for unstructured content. Misclassification routes to a review queue, not to the dead-letter queue. |
| CTR-01 | Input Contract Validator | Stable | Schema and type validation applied to every item at the ingestion boundary. Enforces field presence, type constraints, and format rules. Validation failures are logged with the full input payload for DLQ triage. Contract version is checked against the expected version before validation runs. |
Summarization operates on items drawn from per-domain queues. Each domain has a dedicated worker pool with a fixed concurrency ceiling. The isolation means a burst of items in the research domain does not displace summarization work queued in the content domain. Pool sizes are configured per domain based on expected throughput and acceptable latency, not on a shared resource allocation.
SUM-02 manages fan-out for items that decompose into parallel sub-tasks — a document that splits into section summaries, or a batch that expands into per-item distillation jobs. The controller checks downstream queue depth before committing fan-out output. This is the known-insufficient backpressure primitive: a point-in-time read that does not account for in-flight items already committed by concurrent workers in other domains. The queue depth can pass the check simultaneously for multiple workers and fill faster than it drains.
The operational consequence observed in practice has been queue depth spikes rather than dropped items — the downstream consumer eventually drains the spike, and the delay propagates to dispatch latency. Whether the spike becomes a data loss scenario depends on queue depth limits. Current limits are set conservatively enough that overflow has not occurred. The correct fix — a capacity reservation primitive ahead of fan-out commit — is documented in RES/ORC as an unresolved backpressure question. It has not been implemented.
| Code | Name | Status | Description |
|---|---|---|---|
| SUM-01 | Summarization Worker | Stable | Executes distillation work against items drawn from per-domain queues. One worker pool per domain. Consumer concurrency is bounded per domain — a burst in one domain cannot displace summarization work in another. |
| SUM-02 | Fan-out Controller | In review | Manages fan-out of summarization sub-tasks for items that decompose into multiple parallel units. Checks downstream queue capacity before committing fan-out output. The backpressure check is the open operational problem — see Operational Notes. |
DSP-01 is a single shared queue — unlike the summarization layer, dispatch is not domain-isolated. The rationale: dispatch is the outbound seam, not a processing layer. It is not subject to domain bursts in the same way — items arrive at dispatch after summarization has already throttled and shaped the flow. A shared dispatch queue with priority classes has been sufficient at current volumes.
DSP-02 executes against each delivery target with per-target retry and exponential backoff. Delivery semantics are at-least-once for all current targets — idempotency is enforced at the target, not at the dispatch layer. DSP-03 defines the delivery contract per target: expected payload shape, timeout bounds, and the retry exhaustion behavior. Contract violations at the delivery boundary — the target returning an unexpected shape — are logged as dispatch failures and route to the dead-letter queue.
The dispatch layer has no ordering guarantee across priority classes. Within a priority class, items are delivered in enqueue order. If delivery order across classes matters for a target, that constraint must be enforced at the target, not relied upon from the dispatch queue.
| Code | Name | Status | Description |
|---|---|---|---|
| DSP-01 | Dispatch Queue | Stable | Single outbound queue for all dispatch targets: newsletter delivery, publication triggers, external notifications. Items are ordered by enqueue time within priority class. No delivery ordering guarantee across priority classes. |
| DSP-02 | Dispatch Worker | Stable | Executes delivery against each dispatch target. Per-target retry with exponential backoff. Delivery failures after retry exhaustion route to the dead-letter queue with the target, payload, and failure reason recorded. |
| DSP-03 | Delivery Contract | In review | Per-target delivery contracts specifying expected payload shape, delivery semantics (at-least-once vs. exactly-once where supported), and timeout bounds. Contract violations at the target boundary are logged as dispatch failures, not infrastructure failures. |
The dead-letter queue is the most operationally useful surface in the system. Every item that enters it was processed far enough to generate a failure record — the payload, the stage, the failure reason, and the retry count are all preserved. The DLQ is not a graveyard. It is a structured record of every place the pipeline's assumptions were violated by real data.
DLQ-01 retains items for 14 days. Items are queryable by failure stage, failure type, and arrival time. The failure reason field is the primary triage signal: schema validation failures indicate upstream contract drift; classification failures indicate classifier gaps; delivery failures indicate target-side changes. Each failure type has a different resolution path — some require a contract update and re-queue, some require classifier retraining, some require target-side investigation.
DLQ-02 is the gap in the current system. Manual triage means failure rate trends are only visible when someone looks. The schema drift that produced the evaluation system's silent drop incident — documented in BLD/EVA — was the same class of failure that DLQ-02 automation is intended to surface proactively. The automation is in design. Until it ships, DLQ triage remains a manual process run on a weekly cadence.
| Code | Name | Status | Description |
|---|---|---|---|
| DLQ-01 | Dead-Letter Queue | Stable | Receives items that failed ingestion validation, classification, summarization, or dispatch after retry exhaustion. Retains the full item payload, the failure reason, and the pipeline stage at which failure occurred. Retention window: 14 days. |
| DLQ-02 | DLQ Triage Worker | In design | Periodically scans the dead-letter queue, classifies failures by type (schema mismatch, transient upstream error, classification failure, delivery failure), and surfaces failure rate by type. Re-queue path for items where the root cause has been resolved. Currently manual — automation of the classification step is in design. |
Per-domain queue isolation
The migration from a single shared queue to per-domain queues resolved cross-domain displacement under burst conditions. The operational cost is monitoring surface: ten queues require ten consumer health checks, ten depth alarms, and ten consumer version pins. The isolation benefit is real. So is the overhead.
Fan-out backpressure
SUM-02 checks downstream queue depth before committing fan-out output. The check is a point-in-time depth read — it does not account for in-flight items already committed by other workers. Under concurrent fan-out from multiple domains, the depth check can pass for all workers simultaneously while the downstream queue is filling faster than it drains. The correct primitive is a capacity reservation, not a depth check. This is under review.
Contract coverage
CTR-01 enforces contracts at the ingestion boundary. DSP-03 enforces contracts at the dispatch boundary. The summarization layer has no explicit contract on its output format — a schema change in the summarization output propagates to the dispatch queue without a validation gate. This is the next expected failure surface.
DLQ as observability
The dead-letter queue is the most useful operational signal in the system — it records every failure with enough context to diagnose root cause. DLQ-02 is currently manual triage. The failure classification automation in design will surface failure rate trends before they accumulate into incidents.
Continue reading