At iteration four, the human review gate was removed. A model-graded reviewer replaced it. Output volume increased immediately. Within one week, seven published pieces required retraction and full rewrite. The model reviewer had approved drafts that were technically correct and editorially wrong — well-argued pieces that were not load-bearing in the publication's architecture, correctly structured pieces that failed the publication standard the gate was designed to enforce. The automated gate was removed. The human gate was restored. It has not been touched since.
That incident is the most important thing this log documents. Not because it was the most technically interesting failure — it was not — but because every subsequent architecture decision was made in its shadow. The critique loop exists to deliver better input to the gate, not to reduce the gate's workload. The outline graph exists to catch structural problems before the gate sees them. The section spec contracts exist because a vague spec is a deferred gate failure. The trace is there because a system that cannot see its own failures cannot improve them.
Eleven iterations in, the pipeline runs: outline graph with section specs, long-context drafting against specs, voice constraint flagging, model-graded critique with targeted revision dispatch, mandatory human review, simultaneous publish-and-index. The open problems are in the critique loop's termination logic, the voice enforcer's false positive rate, and the publish-and-index atomicity gap. This log documents how the system arrived at its current state.
A publication without a pipeline is a publication whose quality is bounded by the time available and the discipline of the person writing. Both are finite and variable. A pipeline separates the structural requirements of publishable content from the moment-to-moment conditions under which it is produced. The outline graph produces the same dependency structure regardless of whether it is built on a focused afternoon or a fragmented morning. The critique loop applies the same criteria to every draft regardless of how confident the writer felt about it.
The second motivation is the trace. Writing without tracing produces content whose quality is assessable only at the output — either the piece is good or it is not, and the diagnosis for why it is not must be reconstructed from memory. Writing with tracing produces a structured record of every outline decision, draft pass, critique flag, revision dispatch, and gate review for every piece. The trace reveals that revision count correlates with spec quality. It reveals which critique criteria fail most often and at which stage. It reveals which classes of voice violation recur across multiple pieces. Without the trace, these patterns are invisible. The framework this system operationalizes is FRM/05— the knowledge forge pipeline applied to content production rather than knowledge accumulation.
Each iteration was forced by the failure of the previous one. The system did not grow by feature addition — it grew by encountering the class of failure the current architecture could not handle and building the layer that addressed it.
No pipeline. Research notes converted to articles by hand. No outline structure, no critique pass, no trace. Writing quality was a function of available time and remembered best practices.
No repeatability. A piece written in three hours when focused produced different results than the same piece written in fragments across a week. No mechanism existed to distinguish a good draft from a mediocre one before it was published.
Added a linear outline step before drafting. Outline sections listed in order. Draft written against the outline. First structured pipeline, minimal tooling.
Linear outlines hid dependency structure. A section that needed to establish an argument for a later section had no explicit relationship to it. Drafts were locally coherent but globally inconsistent — section three would make a claim that section six required but did not cite.
Introduced model-assisted drafting for first passes. Human outline, model draft, human revision. Reduced time-to-draft significantly. First exposure to the voice constraint problem.
Model drafts were tonally wrong at a consistent rate — academic where the publication requires editorial, hedged where the publication requires direct, motivational where the publication requires operational. Voice correction was a full rewrite in roughly 30% of sections.
Replaced human review with a model-graded review pass. Freed the human reviewer from the pipeline entirely. Published output increased significantly.
Published content quality degraded visibly within one week. The model reviewer approved drafts that were technically correct but editorially wrong — correctly argued but not load-bearing in the publication's cross-link architecture, well-written but without the editorial judgment to distinguish a publishable claim from a publishable-sounding one. Seven published pieces required retraction and rewrite. The automated gate was removed.
Restored mandatory human review. Added explicit review criteria: editorial judgment, factual verification, publication standard fit. Gate approval is the only path to publish.
Review gate became a bottleneck without upstream quality improvement. Draft quality entering the gate was unchanged from I03 — the gate was catching the same classes of failures on every piece. The gate was doing too much work that should have been done earlier in the pipeline.
Added a model-graded critique pass between drafting and human review. Critique evaluated against structural criteria: claim support, cross-link load, opening discipline, closing discipline. Critique-passing drafts entered human review with structural problems already resolved.
Critique criteria were too broad. A critique that flags 'weak argument in section three' is not actionable — it does not specify which claim, what evidence is missing, or what revision would satisfy the criterion. Reviewers at the human gate were receiving the same structural feedback the critique was supposed to have resolved.
Critique model output formalized: pass/fail per named criterion, specific offending passage for each failure, targeted revision instruction. One revision dispatch per failing criterion. Loop runs until pass or max cycles.
Revision instructions were often misapplied. A critique that said 'the opening does not state the claim' produced a revision that added a claim sentence but did not change the underlying argument structure — the surface complaint was addressed, the structural problem was not.
Replaced linear outlines with directed graphs. Each section is a node; edges are argument dependencies. Topological sort determines drafting order. Section specs generated per node before drafting begins.
Graph construction was slow and often over-engineered. A five-section article does not need a formal dependency graph — the overhead exceeded the benefit. Graph structure is most valuable for long-form pieces with non-obvious argument dependencies. The threshold for when to use a graph vs. a structured linear outline was not specified.
Each outline node generates a section spec: the claim, the required evidence, target cross-links, word count range. Draft model writes against the spec, not against the outline heading. Critique evaluates against the spec, not against vague quality criteria.
Spec quality became the new rate-limiting variable. An underspecified spec produced a draft that passed the spec's low bar and failed the broader critique criteria. The critique loop was now catching spec failures rather than draft failures — the fix was upstream of where the critique was applied.
Added DFT-02 post-draft pass checking against publication voice constraints. Flags violations specifically — which constraint, which passage — before the critique loop runs.
False positive flag rate was high for approximately three weeks after deployment. Stylistically unconventional passages — intentionally terse, intentionally incomplete by academic standards — were flagged as voice violations. Calibrating the flagging criteria required manual review of 40+ flagged passages before the false positive rate dropped to acceptable levels.
Publish layer routes simultaneously to publication surface and retrieval store on gate approval. Dead link checker runs at publish time. Retrieval store indexed by section with source metadata. Current operational state.
Publish-and-index operations are not atomic. A publish that succeeds on the publication surface and fails on the retrieval indexer produces an inconsistent state that is not caught until a retrieval query fails to surface the page. Atomic publish with rollback on indexer failure is in design.
The shift from linear outlines to directed graphs at I08 was forced by a specific, recurring failure: section three would make a claim that section six required, with no mechanism to detect the dependency until the draft was complete and the critique flagged the missing argument support. The fix was not to write better linear outlines — it was to represent the dependency explicitly as a graph edge, making it enforceable by the drafting order rather than by the writer's memory.
The graph structure changed what section specs could specify. A linear outline section spec can only reference adjacent sections. A graph node spec can reference any dependency node — it can say "this section requires the argument established in the reliability framework section, which is three nodes upstream in the dependency graph." That specificity is what makes the long-context draft model's cross-document consistency possible: the model is not inferring the dependency from sequence, it is being told the dependency explicitly.
The operational overhead of graph construction at I08 was real. For short-form pieces, the graph adds construction time without adding drafting quality. The current practice — graph for pieces with non-obvious dependencies and more than five sections, structured linear outline below this threshold — was arrived at empirically after six pieces where the graph overhead was not justified. Whether this threshold is correct is contested in the open questions.
The critique loop was added at I06 to reduce the class of structural failures that reached the human review gate. Before the loop, the gate was doing triple duty: editorial judgment, structural verification, and voice correction. After I06, the gate received drafts that had passed structural and voice checks — its scope narrowed to the judgment classes automation cannot exercise.
The loop's current shape — pass/fail per named criterion, specific offending passage per failure, targeted revision instruction per failure, one dispatch per criterion — emerged from the failure of I06's broader critique output. A critique that says "section three is weak" is not actionable. A critique that says "section three's opening claim is not supported by the evidence in paragraphs two and three; rewrite to either narrow the claim or add the supporting evidence" is. The formalization of the critique output at I07 was the change that made the loop useful rather than merely present.
Fig. BLD/CON — Critique Cycle Distribution · drafts by revision cycles required · escalation failure class breakdown
The three-cycle maximum before human escalation is the loop's most significant known gap. A structural claim failure — a section spec that specifies an argument that no draft can establish with the available evidence — will fail the same criterion across all three revision cycles and escalate to the human gate with no resolution. The gate reviewer must then diagnose that the failure is structural and route back to the outline layer. This diagnosis should happen at the first critique cycle, not after the third. The detection mechanism has not been implemented.
The human review gate adds the most signal per minute in three specific areas. The first is publication standard fit: whether this piece earns its place in the architecture by being load-bearing, not merely by being correct. This requires knowing what the publication is, what its cross-link architecture demands, and what "load-bearing" means in the context of the specific page being reviewed. The critique model can check whether a cross-link is present; it cannot check whether the cross-link is necessary.
The second is editorial tone above the threshold the voice constraints define. The constraints specify the floor — no startup clichés, no motivation writing, no academic hedging. They do not specify the ceiling — the difference between prose that clears the floor and prose that earns its place in a publication of this standard. That judgment requires reading the piece as a reader would, not as a constraint checker would.
The third is factual verification for domain claims. The critique model pattern-matches against known correct patterns. It does not have genuine domain expertise. A claim about distributed systems behavior, agent failure modes, or evaluation methodology that is plausible but incorrect will pass the critique model and fail a domain expert. The gate is the domain expert check. Its value is proportional to the density of domain-specific claims in the piece — which is exactly what this publication's research and system pages contain. The evaluation methodology that informs the critique criteria is in RES/EVA.
Every attempt to reduce gate scope has increased total publishing time. A gate that passes on editorial tone produces pieces that require post-publish revision. A gate that skips factual verification produces pieces with domain errors that surface in reader feedback. The gate is not the bottleneck in the pipeline — the outline and critique layers are — and the correct response to a slow gate is improving upstream quality, not reducing gate rigor. The operational reference for the current gate is SYS-02.
Decisions that shaped the current system, with current status and rationale.
| Area | Decision | Status | Rationale |
|---|---|---|---|
| Outline structure | Directed graph for complex multi-dependency pieces. Structured linear outline for short-form pieces under five sections. | Stable | Graph overhead is not justified for every piece. The complexity threshold — use a graph when argument dependencies are non-obvious and section count exceeds five — emerged from I08. Below this threshold, a structured linear outline with explicit section specs achieves equivalent draft quality with lower construction cost. |
| Section spec contracts | Every section has an explicit spec before drafting begins. Spec includes: claim, required evidence, target cross-links, word count range. | Stable | The spec is the contract between the outline and draft layers. A vague spec produces a draft that requires more critique cycles. The trace makes this cost visible: revision count correlates directly with spec specificity in the pipeline logs. Writing better specs is the highest-leverage quality improvement available upstream of the draft model. |
| Long-context drafting | Draft model receives the full outline graph and all previously completed sections in context. | Stable | Drafting with only the current section spec in context produces locally correct, globally inconsistent prose. Full-context drafting allows the model to see argument continuity requirements before they are violated, not after. |
| Voice constraint enforcement | DFT-02 flags voice violations, does not rewrite. Targeted revision returns to DFT-01. | In review | Automated rewriting of flagged passages produces technically compliant but tonally flat prose. Flagging with specific constraint and passage preserves the human reviewer's ability to see exactly what was flagged and make a judgment call about whether the flag is correct. The false positive rate after calibration is acceptable; the flagging criteria are not yet fully stable. |
| Critique loop termination | Maximum three revision cycles before escalation to human review with failing criteria flagged explicitly. | Contested | The three-cycle limit was set by intuition, not measurement. Structural claim failures do not resolve through draft revision — they require outline revision. The correct termination condition is: escalate immediately when the failure class traces to the outline layer, not after a fixed revision count. This has not been implemented. |
| Human review gate scope | Gate is mandatory and non-bypassable. Reviews for: editorial judgment, factual verification, publication standard fit. | Stable | The I04 incident established that the gate scope is not negotiable. The gate's three review classes cannot be automated — editorial judgment requires knowing what this publication is, factual verification requires domain expertise, and publication standard fit requires knowing what the cross-link architecture demands. All three are beyond the reliable reach of current critique models. |
| Publish-and-index simultaneity | PUB-01 and PUB-02 execute simultaneously on gate approval. Not atomic. | In review | Atomic publish-and-index with rollback on indexer failure is the correct design. It has not been implemented because the rollback path requires the publication surface to support revert operations that are not currently available. Simultaneity without atomicity is the interim state — the inconsistency window is accepted as a known gap. |
What remains unresolved after eleven iterations. These are operational gaps, not feature requests.
How should the critique loop detect that a failure traces to the outline layer rather than the draft layer?
A critique failure that recurs across multiple revision cycles against the same criterion is circumstantial evidence that the failure is structural — that no draft revision will satisfy the criterion because the section spec or the outline dependency is wrong. Detecting this explicitly, rather than after the third cycle, requires classifying the failure type at the first occurrence. The classification requires the critique model to distinguish 'this draft did not execute the spec' from 'this spec cannot produce a draft that satisfies this criterion.' That is a harder judgment than pass/fail scoring.
What is the correct threshold for using an outline graph vs. a structured linear outline?
The current threshold — graph for pieces with non-obvious argument dependencies and section count above five — is a heuristic without empirical backing. Whether graph structure produces measurably better draft quality at this threshold, or whether the benefit is visible only above a higher threshold, has not been measured. The trace data exists to answer this — revision count by outline type and section count — but the analysis has not been run.
How should the voice constraint enforcer handle intentionally unconventional passages that are stylistically distinct from the publication's default register?
The publication's voice constraints define a default register. Some passages deliberately depart from this register — intentionally terse, intentionally incomplete by conventional standards, intentionally structured differently than the surrounding prose. The constraint enforcer has no mechanism for distinguishing a departure that is intentional and editorially correct from one that is accidental and editorially wrong. The false positive problem is a classification problem: the enforcer needs either a signal of intentionality or a human-in-the-loop check at the flagging stage rather than only at the review gate.
What does atomic publish-and-index require from the publication surface infrastructure?
Atomic publish-and-index with rollback requires the publication surface to support a revert operation: unpublish a page that was published but whose indexer operation failed. The current publication surface does not expose this operation. Implementing it requires either a staging state (published-but-not-live, rolled back if indexing fails) or a compensating transaction pattern (publish, attempt index, unpublish if index fails). The staging state approach is cleaner but requires infrastructure changes that affect the entire publication layer.
BLD/CON · ITERATION 011 · OPERATIONAL
The pipeline runs end-to-end. The human review gate is mandatory and stable. The critique loop reduces gate workload to the judgment classes automation cannot exercise. The trace is present at every stage boundary. The outline graph catches structural dependencies before drafting begins. The open problems are known and bounded: critique loop termination logic, voice enforcer calibration, publish-and-index atomicity. Operational means the pipeline ships publishable content reliably — not that the gaps are closed.