Aura/docs/design/INDEX.history.md

aura design ledger — INDEX history (frozen record)

FROZEN HISTORICAL RECORD. Each block below was true as of its stamp and may be superseded; this file is NOT current truth and NOT a grounding surface. Current ledger: INDEX.md. Per-contract history lives in contracts/*.history.md.

Header provenance and vocabulary gloss (pre-refactor wording)

Moved out of the INDEX header in the 2026-07-21 ledger refactor; the walking-skeleton milestone and the C1–C18 interview are completed history.

Provenance: contracts C1–C18 were settled in the initial rough-sketch design interview (2026-06-03), walking the design tree root-to-leaf (C16–C18 and the C10 refinement to a broker-independent position table came in follow-up turns; C10 was reframed in cycle 0007 to an exposure stream, then again 2026-06-23 (#117) to a bias stream with signal quality in R, the position table a derived layer — see C10). C19–C22 were added as the construction / World / playground layer; C23 and the C9/C19 compilation refinements were settled 2026-06-05 for the Construction-layer milestone (the blueprint→flat-graph reading of composites and graph optimisation). C24 (the blueprint as a serializable, World-owned data value) was settled 2026-06-29, resolving the long-deferred #109 fork in favour of topology-as-data (the game-engine principle, C16): the engine owns topology as content it serializes / loads / generates, not as Rust source baked into the binary.

Vocabulary: a contract is one ledger entry. A cycle is one pipeline round; a milestone is a tracker container spanning many cycles (the first milestone is the walking skeleton: ingest → one signal → exposure → sim-optimal broker → synthetic pip-equity signal-quality metric — the pre-reframe substrate, now reframed to bias → R-evaluator → R-expectancy, see C10).

External components — the RustAst pre-authoring reading list

Written before aura-core existed as a guide for authoring it; aura-core has long been built, so the file-by-file reading list is a historical aid. RustAst's living role (conceptual reference, never a dependency) stays in INDEX.md.

But its src/ast/rtl/ layer is a working reference implementation of the very streaming substrate aura rebuilds, and is worth reading before authoring aura-core — these are not just concepts, they exist as code:

• rtl/series/data.rs — RingBuffer<T> with financial-style indexing (index 0 = newest), total_count (the run-count of C5) and lookback_limit (C8's pre-sized window); the ScalarValue marker trait ("flat scalars only, no String/Record" = C7's closed scalar set); ScalarSeries<f64|i64|bool> and the SoA RecordSeries for composites (C7's "OHLCV = a bundle of base columns").
• rtl/series/mod.rs — create_typed_series, dispatching element type → storage backend (the type-specialized factory of C19).
• rtl/streams/mod.rs — Signal { cycle_id, value } (C4's cycle clock) and the Stream / Observer / ObservableStream push traits (the reactive model of C4/C5).
• rtl/streams/register.rs — the RTL "register" / delay node (the one explicit feedback path of C5/C9) plus a seeded, reproducible OHLC generator (C12's seed-as-input).

aura reimplements these natively and sharpens them: types are monomorphized and edges type-erased to the four scalar kinds with direct dispatch (C7) instead of carrying boxed Values, and input history is shared zero-copy as Arc<[T]> (C12) instead of a VecDeque<Value>. RustAst shows the shape; aura makes it fast and deterministic.

Open threads — resolved or superseded entries (pre-refactor wording)

Each entry below is the pre-refactor text. Where a thread stays genuinely open, INDEX.md carries a terse live successor; resolved threads live only here (their outcomes are recorded in the named contracts).

• Playground & World UI surface — the playground is core (C22); the surface is a web frontend served from disk-persisted traces (revised 2026-06, issue #101; not egui). Settled for the first cut: raw per-tap trace persistence to disk (columnar/SoA form, C7) + serve-time join_on_ts alignment + a static self-contained HTML chart page (uPlot, vendored like the render_html Graphviz-WASM blob), feeds overlaid or timestamp-aligned in panels. The families-comparison view — aura chart <family> overlaying one tap across a family's members on a shared y-scale — shipped (#107, see the C22 amendment). Still open: richer comparison meta-views (sweep-surface heatmaps, cross-family / multi-strategy comparison, multi-column tap selection #47), a local server, and the run/replay clock controls.

• aura new scaffolder and the experiment-builder API — aura new scaffolds a Rust project crate (node / strategy blueprints) against the engine (C16/C20); the experiment-builder API surface (harness wiring, structural axes, sweep combinators) is not yet designed. Aura.toml's schema was settled paths-only in cycle 0102 (data archive root, runs dir — see the C13 realization note); the load boundary itself shipped there.

• The analysis meta-level — composable orchestration + project-as-crate authoring (tracked: #109). aura's differentiator (C21) had two unbuilt halves: (1) the orchestration axes (sweep / Monte-Carlo / walk-forward) becoming composable tools a user wires into an analysis workflow, rather than today's hard-wired CLI verbs (sweep_family / walkforward_family in aura-cli) — still open; and (2) the project-as-crate authoring layer above — its load boundary landed in cycle 0102 (Aura.toml discovery + cdylib loading + merged vocabulary, C13 realization note) and the aura new scaffolder in cycle 0103; of that layer only the experiment-builder API remains open (C16/C17). On the meta-level, authoring a strategy is wiring existing nodes (C9 composition) + defining the analysis framework (C21), not writing new nodes; the user wants it driven via the CLI, later the interactive server / playground (C14/C22). Resolved (2026-06-29) → C24. The fork was mis-posed as "(a) run-a-Rust-experiment-via-CLI vs (b) wire-nodes-via-CLI". The real axis is topology = compile-time Rust source vs runtime, serializable, World-owned data value, settled as the value (C24, the game-engine principle): node logic stays Rust (C17), topology becomes data the World generates / serializes / reproduces. "Wire-nodes-via-CLI" is dropped — interactive wiring through the CLI is a non-goal; it collapses to a file + parser, which is C24's load path. What remains under #109: the blueprint data format itself (C24 Status — its own brainstorm / milestone), and the composable-orchestration half — the axes (sweep / MC / walk-forward) becoming tools applied to a World-constructed harness instead of the hard-wired aura-cli verbs (sweep_family / walkforward_family). Status (cycle 0106, #189 — the artifact half shipped v1). The #188 role-model pass re-cut the "experiment-builder API" reading of this thread (role-6b work mis-typed in role-2 technique): experiment intent is now data, not a Rust API. Cycle 0106 shipped both document vocabularies (process document, campaign document — aura-research), two-tier validation, the op-script-precedent introspection contract (aura process|campaign introspect --vocabulary/--block/--unwired/--content-id), and content-addressed stores (C18 realization note) — authorable and checkable headless, no compile, no run. Status update (cycles 0107–0110). The executor question resolved and shipped (cycles 0107/0108, #198/#200: aura-campaign executes the v2 pipeline shape over the MemberRunner seam); the #188 amendment package landed 2026-07-03 (C25 role-model entry, C20/C22 refinements, invariant-10 clarification). The "once it carries" dissolution of the hard-wired verbs (user decision 2026-07-03 on #188) is running as the milestone "Verb dissolution" (#210): the fork triage of 2026-07-04 decided its design (dissolve the real-data blueprint branches only, built-in/synthetic branches stay verb-wired until #159; generated documents auto-registered; full behaviour parity modulo the recorded additive instrument stamp), and the verb set dissolved in the #210-decision-7 order: cycle 0110 took aura sweep <bp.json> --real … (enabled by the std::sweep selection group becoming optional — all-or-nothing, selection-free = terminal-stage-only; wire form and every stored content id unchanged), then generalize, walkforward, and mc's R-bootstrap path, each now sugar over a generated, content-addressed campaign document through the one executor, plus a structural risk-regime axis (CampaignDoc.risk: [RiskRegime], sole vol{length,k} variant — the stop compared as a matrix dimension, stamped into every member manifest, never argmax-swept). Ad-hoc verb intent no longer evaporates into shell history — the #188 diagnosis's cure applied to the verbs themselves. The dissolved form is per-verb, by intended scope (ratified at milestone close, fieldtest 2026-07-07): for sweep it is the blueprint file (<bp.json> --real), while aura sweep --strategy r-sma --real stays the inline built-in path (its member lines carry no instrument/topology_hash/selection stamp and register no campaign document) — the built-in --strategy demo surface is #159's hard-wired-harness retirement target, not the dissolution's; for generalize/walkforward/mc the dissolved form is now the same generic grammar as sweep — an arbitrary blueprint positional plus --real and repeatable --axis <wrapped-name>=<csv> (#220). [HISTORY — superseded in two steps. #159 (cuts 1b-4, post-2026-07-07) retired the built-in --strategy sweep/demo surface: the inline aura sweep --strategy r-sma --real path no longer exists. #220 then removed the verbs' weld — at milestone close the dissolved form of generalize/walkforward/mc was still the welded --strategy r-sma --real; #220 made the three verbs blueprint-generic (arbitrary blueprint + arbitrary --axis; mc keeps the synthetic --seeds family unchanged beside the new --real --axis campaign mode; generalize takes --real <SYM1,SYM2,…>, >=2, with one value per axis), deleted the welded --strategy grammar (clap rejects it, exit 2), dissolved RGrid, and unified the verbs on the #214 invocation struct. The surviving form everywhere is aura <verb> <blueprint.json> --real … --axis … over examples/r_*.json.] Milestone closed 2026-07-07 on a green end-to-end fieldtest (0 bugs; behaviour preservation, campaign-substrate reach-through, and the risk-regime axis all confirmed); residual findings are discoverability/ergonomics follow-ups (#216 risk-axis discoverability, #217 verb knob asymmetry, #218 no-project store litter). Amendment (2026-07-14, #256 fork B): the dissolved walkforward/mc translations' leading sweep became the enumerate-only std::grid stage — a fieldless vocabulary block, legal only as the first stage and immediately before std::walk_forward. Only the grid's parameter points ever crossed the stage seam (the wf stage re-sweeps them per IS window itself), so the leading stage no longer executes members or persists a Sweep registry family: persisted dissolved-walkforward/mc campaign documents lose that family, while the visible grades are byte-identical (the exact-grade E2E pins are unchanged). The executor's inter-stage seam is now a typed two-armed flow (points-only vs executed members); report-consuming stages are fenced by preflight. translate_generalize keeps its executed std::sweep(argmax) leading stage — its generalize stage consumes the argmax winner report as the cell nominee.

• Inferential honesty of the World — family-selection without false-discovery control (tracked: milestone "Inferential validation (defend against false discovery at sweep scale)", #144 / #145 / #146; adjacent #139). The World's massively-parallel sweep (C12 axes 1–2, C21) is aura's differentiator and its most direct route to false discovery: optimize / rank_by select the single best family member by a bare argmax, with no penalty for the number of configurations tried, no preference for a robust parameter neighbourhood over a sharp in-sample peak, and no cross-instrument generalization read; mc is a descriptive seed-sweep, not an inferential significance test. The hygiene invariants keep one backtest honest (C1 determinism, C2 no look-ahead); they do not make a selection across a family honest. Recognized direction: the selection/aggregation layer must deflate a winner for the family size (#144), prefer a plateau over a peak (#145), and score cross-instrument generalization (#146), with a per-candidate out-of-sample bootstrap as the adjacent significance read (#139, landed cycle 0075). Distinct from the two threads above — neither orchestration composability (#109) nor a search policy (Bayesian/ genetic), but the statistical validity of the selection itself. Not a C-invariant until built; recorded as the World's third half — now structurally complete (all three pieces built, cycle 0078). Status (cycle 0076): the trials-deflation piece (#144) landed — optimize_deflated (aura-registry) wraps optimize and records, additively on the winning member's manifest (C18, the new RunManifest.selection), a deflated score + an overfit probability for the family size, without changing which member wins (C23; optimize / rank_by stay a bare argmax — the deflation is recorded provenance, not a re-ranking). The R arm is a centred moving-block reality-check (reusing the r_bootstrap kernel); the total_pips arm a closed-form expected-max-of-K dispersion floor. Status (cycle 0077): the plateau-over-peak piece (#145) landed — optimize_plateau (aura-registry) argmaxes the neighbourhood-smoothed metric surface (mean or worst-case over each member's closed mixed-radix grid neighbourhood) instead of the bare peak, recorded on the same RunManifest.selection carrier, now an orthogonal rule × annotation: SelectionMode::{Argmax, PlateauMean, PlateauWorst} with either the deflation annotation (#144) or the plateau annotation (neighbourhood_score / n_neighbours). It is opt-in via a --select flag — default argmax stays byte-identical (C23). The grid lattice surfaces from the engine (SweepBinder::sweep_with_lattice); the policy stays in aura-registry with walk_forward selection-agnostic (C9). Selection is now a pluggable objective (bare argmax → trials-deflated → plateau), not a single hard-wired argmax. Status (cycle 0078): the last piece — cross-instrument generalization (#146) — landed, completing the inferential half. Unlike #144/#145 (which select + annotate a within-family winner on RunManifest.selection), #146 is an aggregator/ validator, not a selector: the aura generalize subcommand runs a brought candidate across an instrument list and generalization (aura-registry) reduces the per-instrument R-metrics to a worst-case floor (min over instruments — the cross-instrument sibling of PlateauMode::Worst, R-only per C10) + a sign-agreement count + the per-instrument breakdown. It is a recomputable aggregate over a new FamilyKind::CrossInstrument family (C12's comparison axis, realized), each member self-identifying via the new RunManifest.instrument lineage field (C18) — not stamped on FamilySelection (there is no within-family winner to annotate). The inferential half is now structurally built; the per-candidate OOS bootstrap (#139, cycle 0075) is its adjacent significance read.

• aura-std contents — substantively populated (~30 node modules): SMA/EMA, arithmetic (Add/Sub/Mul/Sqrt/LinComb), logic (And/Gt/Latch/EqConst), the Delay z⁻¹ register, Resample, RollingMin/RollingMax, Session, the R chain (Bias/FixedStop/Sizer/PositionManagement), the legacy SimBroker pip yardstick, the cost-model graph (CostNode/CostRunner/CostSum + ConstantCost/VolSlippageCost/ CarryCost), and the Recorder/GatedRecorder/SeriesReducer sinks. Further blocks land demand-driven as the walking-skeleton needs them.

• Sequencing — the runnable single-harness substrate comes first (walking skeleton: a closed harness that runs deterministically and records via a sink); the World/meta layer (C21) and the playground trace-explorer (C22) are the differentiating layers that follow — you orchestrate and visualize a thing that must first run once.