Drain taps during the run — subscribers (stream-to-disk, fold, live consumer) replace buffer-then-drain #283
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Surfaced by the cycle-close architect review of the "Measurement as a first-class citizen" milestone (#282 declared taps).
Problem
The single-run tap path (
run_signal_r,crates/aura-cli/src/main.rs) binds each declared tap to a per-cycleRecorderand drains itsmpscchannel only after the run completes. So every tapped series buffers O(cycles) rows in the channel for the whole run — the same ~2GiB/member retention profile noted in #138. This is fine for the milestone's synthetic acceptance run (a bounded blueprint over a short window), but the next plausible use — the origin UK100 open-momentum study over full history (2014-08..2026-07, ~3120 weekdays of M1 bars) — would hit it.Direction (not prescribing)
Name the folding-sink-vs-per-cycle-
Recorderchoice for the tap path: a tap that only needs an aggregate (a count, a mean, an R-expectancy) could fold incrementally into a scalar instead of retaining every row, mirroring how the reduce-mode sweep path already strips per-cycle recording. A tap that genuinely needs the full series (for charting) keeps the per-cycle recorder but could stream to disk rather than buffering in-channel. The decision is which tap consumers need the row-level trace vs a fold.Not a milestone blocker
The declared-tap mechanism ships correct and verified this cycle; this is a scaling follow-up for running a real study over full history, filed per the forward-queue discipline. No behaviour is wrong today — the buffer is bounded by the run length.
Design direction, 2026-07-20 — the fold-vs-record fork restated as a subscription model.
No second tap kind is needed. A tap is already pure declaration: a name plus a resolved
OutPortinComposite.taps(crates/aura-engine/src/builder.rs:161; the declared observation points from #284). The O(cycles) retention described in this issue is created entirely behind the declaration —run_signal_r(crates/aura-cli/src/main.rs) wires a per-cycleRecorderper tap and drains its mpsc channel only after the run completes. The memory profile is therefore a property of the drain wiring, not of the tap. Splitting the concept into a "folding tap" and a "recording tap" would encode a consumer-side choice into the graph vocabulary.The core problem is drain timing, not trace-vs-fold. With consumers that consume during the run, nothing accumulates O(cycles) unless a consumer itself chooses to retain. The trace/fold split then falls out as consumer instances of one mechanism — the tap stays an observation point, consumers subscribe to it:
Where fold logic lives — the all-logic-is-Rust boundary. For headless, artifact-driven runs the measurement intent must travel as data, so a fold is referenced by name from a closed vocabulary, each name bound to a Rust implementation (the
SeriesReducer-style machinery already in place); a fold the vocabulary lacks escalates to a new named Rust block, never to inline code in a data artifact (the C25 rule: closed, typed vocabularies as serialized data are fine, open logic-bearing holes are not). A live in-process consumer crosses no serialization boundary, so its fold is plain Rust — a closure owned by the component; routing it through the registry would add ceremony without purpose. A supporting smell that drain policy wants a declared home rather than name-inference:reduce_for_tap(crates/aura-cli/src/main.rs:197) currently infers decimation policy from the tap name (exposure).Backpressure is a subscription property, not a tap property. A lossless subscriber (the recorder) may make the sim wait on a bounded channel — determinism is unaffected, only wall time. A live view may drop rows when it falls behind. Lossless-vs-lossy is declared per subscription.
One design decision remains open before this is spec-ready: where the drain policy for the headless path is declared — per run invocation / measurement document, or on the tap declaration itself. Leaning (2026-07-20): the run/measurement side, so the same tapped blueprint serves a charting question in one run and an aggregate question in the next without re-authoring, keeping graph authorship decoupled from measurement intent.
Single-run tap drain buffers O(cycles) rows — fold before a full-history declared-tap studyto Drain taps during the run — subscribers (stream-to-disk, fold, live consumer) replace buffer-then-drainRefinement, 2026-07-20 — the subscription model needs no series semantics; the open question from the design-direction comment is resolved by narrowing.
A tap declares nothing but its name. The consumer's knowledge comes from authorship, not from stream self-description: whoever writes the subscription also authored (or chose) the graph and knows what the tapped port emits — a tap is a faucet; whoever turns the handle knows the water will come out warm, the water does not have to announce it. Each candidate consumer of series semantics has a semantics-free path: fold-applicability checks are a typo-class guardrail the author's own knowledge covers; a generic chart renders any unknown tap safely (min/max envelope decimation is aliasing-safe for any series, a per-tap y-scale is always correct); a live in-process consumer is specialized by construction.
reduce_for_tap(crates/aura-cli/src/main.rs:197) stays as a tolerated name-convention shortcut for the wrap-convention taps.Producer-declared, graph-propagated semantic kinds — the architecture that would replace that shortcut — are parked as #293 (idea) with a named trigger; no part of this issue depends on them.
Scope of this issue, settled: (1) drain during the run instead of buffer-then-drain; (2) one tap concept, consumers as subscribers — stream-to-disk recorder (lossless), named fold from a closed vocabulary for the headless/artifact path, live in-process consumer with consumer-owned logic; (3) lossless-vs-lossy as a per-subscription property. The remaining decision from the earlier comment (where the headless drain policy is declared: run invocation / measurement document vs. tap declaration) stands, with the recorded leaning to the run/measurement side unchanged.
Decision, 2026-07-20 — the declaration-site question left open in the design-direction comment is closed, and was implicitly closed by the no-semantics refinement: a tap that declares nothing but its name cannot be the carrier of a drain policy. The headless drain policy is declared on the run/measurement side (run invocation / measurement document), never on the tap declaration. The same tapped blueprint thus serves a charting question in one run and an aggregate question in the next without re-authoring; graph authorship stays decoupled from measurement intent.
No open design fork remains on this issue — ready for spec production.
Re-anchoring after the #295 shell-boundary extraction (merged
4ed6455, 2026-07-21): the tap path this issue describes moved out of the CLI.run_signal_r, crates/aura-cli/src/main.rs.crates/aura-runner/src/member.rs(functionrun_signal_r, currently starting around line 481; the tap-binding/drain sequence this issue targets is at lines 508-538 and 560-576).The mechanism is otherwise unchanged and the problem still stands: each declared tap is bound to a fresh
mpsc::channelviaRecorderbeforeHarness::bootstrap/run_bound, and every channel is drained withrx.try_iter().collect()only after the run completes, so a tapped series still retains O(cycles) rows in-channel for the run's full duration.Note for whoever picks this up: the existing per-cycle-fold precedent (
SeriesReducerfolding eq/ex to one summary row,GatedRecorderretaining only gated R rows in reduce mode — same file, around lines 269-291) only applies to the three built-in taps (eq/ex/r) and only in reduce mode. It is not available to user-declared taps (#282/C27), which is what this issue is about; those always retain the full series today, in both modes.The #295 extraction also settles where a fix belongs:
aura-runneris now the C28 'assembly' library crate that both the CLI and any future World program or host/MCP face call into for the standard run recipe (design ledger C25 amendment, docs/design/INDEX.md). A fold-vs-stream tap redesign is runtime/library work, not CLI or document-vocabulary work, and should land incrates/aura-runner/src/member.rs(and any shared subscriber abstraction it grows), independent of the document-first CLI convergence tracked in #300.Design reconciliation (specify)
Spec: tap-subscribers (in production, combined cycle with #77). The 2026-07-20 design comments settled the model (one tap concept, consumers as subscribers, per-subscription loss policy, drain policy on the run/measurement side); this records the remaining mechanism-level forks, derived while producing the spec.
Fork: fold delivery shape -> an in-graph folding sink bound at the tap via
FlatGraph::bind_tap(the SeriesReducer/GatedRecorder precedent), accumulating owned state per fired cycle and emitting one summary row onfinalize; no per-cycle channel traffic at all.Basis: derived - the design comment itself points at "the SeriesReducer-style machinery already in place"; an in-graph fold removes the drain-timing problem entirely, which is strictly stronger than draining during the run.
Fork: record delivery shape -> a bounded sync-channel of
(Timestamp, Cell)plus one writer thread per record-subscribed tap, streaming into the trace store incrementally; the canonicalColumnarTraceJSON is assembled by streaming concatenation of two temp column streams at finish, so the on-disk format and every reader (chart,read,read_family) stay byte-compatible. Lossless backpressure = the sim blocks on a full channel - wall time only, never determinism (C1: the writer consumes one-way, feeds nothing back into the graph).Basis: derived - "lossless full trace at constant memory" is the named requirement; this is the smallest shape that achieves it without an on-disk format migration.
Fork: live delivery shape -> an in-graph sink holding a consumer-owned closure (
FnMut(Timestamp, Cell)), called per fired cycle on the sim thread; loss policy is the closure's own (per the no-semantics refinement: consumer-owned logic and state, no registry ceremony).Fork: declaration surface this cycle -> the library seam only: a
TapPlan(tap name -> subscription: record | fold(kind) | live(consumer)) parameter onrun_signal_rinaura-runner(the C28 assembly crate, per the re-anchor comment). The CLI passes record-all, preserving today'saura runsemantics (every declared tap persisted; tap-free run byte-identical, noruns/write) at constant memory. A document-vocabulary surface for per-tap measurement intent belongs to the #300 line's successors, per this issue's own re-anchor ("independent of the document-first CLI convergence").Fork: fold vocabulary v1 ->
Count | Sum | Mean | Min | Max | Last(Sum/Mean/Min/Max bind only at f64 taps - a bind-time refusal otherwise; Count at any kind; Last kind-preserving). R-expectancy is deferred: it needs trade-gating semantics (a GatedRecorder-class reduction over a multi-column record), not a scalar column fold - a future vocabulary entry per C25 when a study needs it.Fork: fold output surface -> a one-row
ColumnarTraceunder the tap's own name in the same trace store (uniform tooling; the aggregate answer replaces the series for that run, matching the settled "same tapped blueprint, charting question in one run, aggregate question in the next"). A never-warm tap folds to an empty trace for every fold kind.Fold vocabulary v1 amendment (user decision)
FoldKindgainsFirst— kind-preserving, binds at any kind, its emitted row carries the first warm value's timestamp (every other fold's row carries the last folded value's ts). v1 is nowCount | Sum | Mean | Min | Max | First | Last.Basis: user decision, 2026-07-21 — during spec review Brummel asked whether first/last were missing from the fold vocabulary;
Lastwas already in v1,Firstwas not and is added. This supersedes the "Fork: fold vocabulary v1" line of the earlier reconciliation comment (2026-07-21).Scope amendment (specify, grounding-check finding)
The fresh-context grounding-check surfaced a second declared-tap entry point this issue (and C27's Current state) never mentioned:
run_measurement(crates/aura-runner/src/measure.rs, theaura measureverb) mirrorsrun_signal_r's Recorder-per-tap bind + post-runtry_iterdrain byte-for-byte, so it carries the identical O(cycles) retention profile.run_measurement? -> YES — extended, not scoped out. Both entry points take theTapPlanparameter and share one wiring helper pair (bind_tap_plan/BoundTaps::finish) inaura-runner, so the mechanism cannot drift between them; both CLI verbs pass record-all.Basis: derived — the paths are self-described mirrors with the same retention problem this issue exists to fix; a shared helper makes the extension near-free, while scoping the measure path out would leave the "Measurement as a first-class citizen" milestone's own verb carrying exactly the profile this issue retires. (This extends the "declaration surface" line of the 2026-07-21 reconciliation comment: the seam is
run_signal_rANDrun_measurement.)Ledger note for cycle close: C27 "Current state" names only
run_signal_ras the single-run bind/record path — the audit lift should addrun_measurement.Spec sign-off (user decision)
The tap-subscribers spec passed the grounding-check (all load-bearing assumptions ratified by named green tests) and was approved by Brummel, 2026-07-21.
One question was raised and consciously settled during review: whether the fold vocabulary v1 should grow trading-specific entries (Std/Drawdown/SignFlips fit the single-column online-fold mechanism; R-expectancy/win-rate/profit-factor/SQN are trade-level and belong to the deferred gated-reduction family; Sharpe/correlations are periodized/multi-column and out of mechanism). Decision: keep v1 as specced (
Count | Sum | Mean | Min | Max | First | Last) — the vocabulary grows additively per C25 when a study needs it.Basis: user decision, 2026-07-21 (review conversation; options presented with a recommendation to add the three fitting folds, user chose keep-as-is).
Proceeding to plan production.
Design revision (user decision, 2026-07-21) — one consumer mechanism, layered fold registry
During implementation review the user re-examined the signed design's three parallel subscription variants (
Record | Fold(kind) | Live(closure)) and superseded them. This comment records the revision; the previously signed spec is superseded on the points below. Already-landed sink code is reworked forward (no history rewind).Fork: three subscription variants vs one consumer mechanism → one mechanism. Record, fold, and live are technically the same thing — a consumer of the tap's
(timestamp, value)stream; the only real axis is data-layer serializability (a label is data; a closure is not). The subscription surface becomesNamed { label, params } | Live(closure):Namedis fully serializable,Livethe single deliberately non-data variant (consumer-owned logic and loss policy, unchanged).Basis: user decision, 2026-07-21 — identified the three-variant split as an instance of the closed-enum trap and chose the unified model.
Fork: closed fold enum vs layered registry → a layered fold registry, mirroring the node-vocabulary pattern (std vocabulary + environment resolve). Core seeds:
record,count,sum,mean,min,max,first,last— the ratified v1 vocabulary, withrecordnow simply an entry. Higher layers (e.g. a future trading layer addingdrawdown,std) register additional entries without bleeding into the core — dependency injection; the registry travels with the tap plan the way the environment travels with a blueprint resolve. A genuinely new aggregate remains a new Rust entry, never inline logic in data (C25).Basis: user decision, 2026-07-21.
Record consumer holds its writer in-graph → the
recordentry's consumer appends to the streaming trace writer directly from the graph (buffered file append per fired cycle); the signed spec's writer-thread + bounded-channel + drop-before-join protocol is retired. The streamed-writer seam in the trace store is unchanged and stays pinned byte-equal to the legacy write path.Basis: derived, 2026-07-21 — with the consumer living in the graph there is no cross-thread hand-off to protect; a thread adds deadlock surface (join/drop ordering) and buys nothing at the sim's write rate; determinism (C1) untouched, since file I/O feeds nothing back into state evolution.
Fork: lifecycle → the node contract gains
initialize()(default no-op), called once per node in topological order before the source loop — the mirror of the existing end-of-runfinalizehook. Registry consumers acquire their run resources (the record writer's streams) ininitialize; it is infallible by signature, so an acquisition error is stored, the consumer degrades to inert, and exactly one terminal message surfaces through the existing "writing tap traces failed" register at finish. Pre-run directory creation stays the typed pre-run refusal point.Basis: user proposal, 2026-07-21 ("give finalize an initialize counterpart"), ratified against the finalize lifecycle contract (C8/#138).
Fork: registry entry docs → every entry carries a
docstring; the registry becomes the single source for help generation ("which methods are supported?") and for roster-enumerating refusals (an unknown label lists the available labels, following the established enumerate-the-roster refusal register).Basis: user proposal, 2026-07-21, ratified.
Fork: construction parameters → every entry declares a scalar-typed parameter schema (name + kind); a
Namedsubscription carries label + parameter bindings, validated at bind time in the same refusal register (unknown / missing / kind-mismatched parameter). This mirrors the node vocabulary's existing parameter pattern (schema params bound as scalars) and stays inside C25's closed-vocabulary tier: a parameter that would need to carry logic is a new entry, never a freetext hole. All v1 core entries are parameter-less; the seam ships now because it sits in the build signature every registered entry implements (retrofitting later would break every layer contribution), and generic validation is testable without a parameterized entry.Basis: user question, 2026-07-21 ("what if entries need construction parameters?"); resolution derived from the node-vocabulary precedent and ratified.
Status: design settled; the signed spec of 2026-07-21 is superseded on the points above and due for re-production and re-gating. The streaming trace-store write path and the fold/live accumulation cores survive unchanged.
Spec auto-sign (grounding-check PASS, no human in the loop)
The re-produced tap-subscribers spec — the registry model recorded in the design-revision comment (issues/283#issuecomment-4218) — passed the fresh-context grounding check: 13 load-bearing assumptions about current codebase behaviour were each ratified by a named, currently-green test (among them the finalize-epilogue anchor for the new initialize mirror, the streamed-writer byte-equality pin, both declared-tap entry-point anchors, and the layering guard that forces the record consumer's move into the assembly crate); the workspace type-checks clean. Two self-disclosures in the spec were verified accurate rather than blocking: the "writing tap traces failed" register is untested today (the cycle schedules its coverage), and the run-completes-on-writer-failure semantics is a commitment of the new design, not a claim about current behaviour.
Under the autonomous run policy this PASS is the signature; the spec was signed without a human review pause and the user retains an after-the-fact veto.
Status: spec signed — ready for plan production.