Aura/docs/design/INDEX.history.md

# aura design ledger — INDEX history (frozen record)

> FROZEN HISTORICAL RECORD. Each block below was true as of its stamp and may
> be superseded; this file is NOT current truth and NOT a grounding surface.
> Current ledger: [INDEX.md](INDEX.md). Per-contract history lives in
> `contracts/*.history.md`.

## Header provenance and vocabulary gloss (pre-refactor wording)

Moved out of the INDEX header in the 2026-07-21 ledger refactor; the
walking-skeleton milestone and the C1–C18 interview are completed history.

Provenance: contracts C1–C18 were settled in the initial rough-sketch design
interview (2026-06-03), walking the design tree root-to-leaf (C16–C18 and the
C10 refinement to a broker-independent position table came in follow-up turns;
C10 was reframed in cycle 0007 to an exposure stream, then again 2026-06-23
(#117) to a **bias** stream with signal quality in **R**, the position table a
derived layer — see C10). C19–C22 were added as the
construction / World / playground layer; C23 and the C9/C19 compilation
refinements were settled 2026-06-05 for the **Construction-layer** milestone (the
blueprint→flat-graph reading of composites and graph optimisation). C24 (the
blueprint as a serializable, World-owned data value) was settled 2026-06-29,
resolving the long-deferred #109 fork in favour of **topology-as-data** (the
game-engine principle, C16): the engine owns topology as content it serializes /
loads / generates, not as Rust source baked into the binary.

Vocabulary: a *contract* is one ledger entry. A *cycle* is one pipeline round; a
*milestone* is a tracker container spanning many cycles (the first milestone is
the **walking skeleton**: ingest → one signal → exposure → sim-optimal broker →
synthetic pip-equity signal-quality metric — the *pre-reframe* substrate, now
reframed to bias → R-evaluator → R-expectancy, see C10).

## External components — the RustAst pre-authoring reading list

Written before `aura-core` existed as a guide for authoring it; `aura-core`
has long been built, so the file-by-file reading list is a historical aid.
RustAst's living role (conceptual reference, never a dependency) stays in
[INDEX.md](INDEX.md).

  But its **`src/ast/rtl/` layer is a working reference implementation of the very
  streaming substrate aura rebuilds**, and is worth reading before authoring
  `aura-core` — these are not just concepts, they exist as code:
  - `rtl/series/data.rs` — `RingBuffer<T>` with financial-style indexing (index 0
    = newest), `total_count` (the **run-count** of C5) and `lookback_limit` (C8's
    pre-sized window); the `ScalarValue` marker trait ("flat scalars only, no
    String/Record" = C7's closed scalar set); `ScalarSeries<f64|i64|bool>` and the
    **SoA** `RecordSeries` for composites (C7's "OHLCV = a bundle of base columns").
  - `rtl/series/mod.rs` — `create_typed_series`, dispatching element type → storage
    backend (the type-specialized **factory** of C19).
  - `rtl/streams/mod.rs` — `Signal { cycle_id, value }` (C4's cycle clock) and the
    `Stream` / `Observer` / `ObservableStream` push traits (the reactive model of
    C4/C5).
  - `rtl/streams/register.rs` — the RTL **"register" / delay node** (the one
    explicit feedback path of C5/C9) plus a seeded, reproducible OHLC generator
    (C12's seed-as-input).

  aura reimplements these natively and **sharpens** them: types are monomorphized
  and edges type-erased to the four scalar kinds with direct dispatch (C7) instead
  of carrying boxed `Value`s, and input history is shared zero-copy as `Arc<[T]>`
  (C12) instead of a `VecDeque<Value>`. RustAst shows the *shape*; aura makes it
  fast and deterministic.

## Open threads — resolved or superseded entries (pre-refactor wording)

Each entry below is the pre-refactor text. Where a thread stays genuinely
open, INDEX.md carries a terse live successor; resolved threads live only
here (their outcomes are recorded in the named contracts).

- **Playground & World UI surface** — the playground is core (C22); the surface
  is a **web frontend served from disk-persisted traces** (revised 2026-06, issue
  #101; not egui). Settled for the first cut: raw per-tap trace persistence to
  disk (columnar/SoA form, C7) + serve-time `join_on_ts` alignment + a static
  self-contained HTML chart page (uPlot, vendored like the `render_html`
  Graphviz-WASM blob), feeds overlaid or timestamp-aligned in panels. The
  families-comparison **view** — `aura chart <family>` overlaying one tap across a
  family's members on a shared y-scale — shipped (#107, see the C22 amendment).
  Still open: richer comparison meta-views (sweep-surface heatmaps, cross-family /
  multi-strategy comparison, multi-column tap selection #47), a local server, and the
  run/replay clock controls.

- **`aura new` scaffolder and the experiment-builder API** — `aura new`
  scaffolds a Rust project *crate* (node / strategy blueprints) against the
  engine (C16/C20); the experiment-builder API surface (harness wiring,
  structural axes, sweep combinators) is not yet designed. `Aura.toml`'s
  schema was settled paths-only in cycle 0102 (data archive root, runs dir —
  see the C13 realization note); the load boundary itself shipped there.

- **The analysis meta-level — composable orchestration + project-as-crate authoring
  (tracked: #109).** aura's differentiator (C21) had two unbuilt halves: (1) the
  orchestration axes (sweep / Monte-Carlo / walk-forward) becoming *composable tools*
  a user wires into an analysis workflow, rather than today's hard-wired CLI verbs
  (`sweep_family` / `walkforward_family` in `aura-cli`) — still open; and (2) the
  project-as-crate authoring layer above — its **load boundary landed in cycle 0102**
  (`Aura.toml` discovery + cdylib loading + merged vocabulary, C13 realization note)
  and the **`aura new` scaffolder in cycle 0103**; of that layer only the
  experiment-builder API remains open (C16/C17).
  On the meta-level, authoring a strategy is *wiring
  existing nodes* (C9 composition) + *defining the analysis framework* (C21), not
  writing new nodes; the user wants it driven via the CLI, later the interactive
  server / playground (C14/C22). **Resolved (2026-06-29) → C24.** The fork was
  mis-posed as "(a) run-a-Rust-experiment-via-CLI vs (b) wire-nodes-via-CLI". The real
  axis is **topology = compile-time Rust source vs runtime, serializable, World-owned
  data value**, settled as the **value** (C24, the game-engine principle): node
  *logic* stays Rust (C17), *topology* becomes data the World generates / serializes /
  reproduces. "Wire-nodes-via-CLI" is dropped — interactive wiring through the CLI is a
  non-goal; it collapses to a file + parser, which *is* C24's load path. What
  **remains** under #109: the **blueprint data format** itself (C24 Status — its own
  brainstorm / milestone), and the **composable-orchestration** half — the axes
  (sweep / MC / walk-forward) becoming tools applied to a World-constructed harness
  instead of the hard-wired `aura-cli` verbs (`sweep_family` / `walkforward_family`).
  **Status (cycle 0106, #189 — the artifact half shipped v1).** The #188 role-model pass
  re-cut the "experiment-builder API" reading of this thread (role-6b work mis-typed in
  role-2 technique): experiment intent is now **data**, not a Rust API. Cycle 0106 shipped
  both document vocabularies (process document, campaign document — `aura-research`),
  two-tier validation, the op-script-precedent introspection contract
  (`aura process|campaign introspect --vocabulary/--block/--unwired/--content-id`), and
  content-addressed stores (C18 realization note) — authorable and checkable headless, no
  compile, no run. **Status update (cycles 0107–0110).** The executor question resolved and
  shipped (cycles 0107/0108, #198/#200: `aura-campaign` executes the v2 pipeline shape over
  the `MemberRunner` seam); the #188 amendment package landed 2026-07-03 (C25 role-model
  entry, C20/C22 refinements, invariant-10 clarification). The "once it carries" dissolution
  of the hard-wired verbs (user decision 2026-07-03 on #188) is **running as the milestone
  "Verb dissolution" (#210)**: the fork triage of 2026-07-04 decided its design (dissolve
  the real-data blueprint branches only, built-in/synthetic branches stay verb-wired until
  #159; generated documents auto-registered; full behaviour parity modulo the recorded
  additive instrument stamp), and the verb set dissolved in the #210-decision-7 order: cycle 0110 took `aura sweep
  <bp.json> --real …` (enabled by the `std::sweep` selection group becoming optional —
  all-or-nothing, selection-free = terminal-stage-only; wire form and every stored content
  id unchanged), then generalize, walkforward, and mc's R-bootstrap path, each now sugar over
  a generated, content-addressed campaign document through the one executor, plus a structural
  risk-regime axis (`CampaignDoc.risk: [RiskRegime]`, sole `vol{length,k}` variant — the stop
  compared as a matrix dimension, stamped into every member manifest, never argmax-swept).
  Ad-hoc verb intent no longer evaporates into shell history — the #188 diagnosis's cure
  applied to the verbs themselves. **The dissolved form is per-verb, by intended scope**
  (ratified at milestone close, fieldtest 2026-07-07): for sweep it is the blueprint file
  (`<bp.json> --real`), while `aura sweep --strategy r-sma --real` stays the inline built-in
  path (its member lines carry no instrument/topology_hash/selection stamp and register no
  campaign document) — the built-in `--strategy` demo surface is #159's hard-wired-harness
  retirement target, not the dissolution's; for generalize/walkforward/mc the dissolved
  form is now the same generic grammar as sweep — an arbitrary blueprint positional plus
  `--real` and repeatable `--axis <wrapped-name>=<csv>` (#220). [HISTORY — superseded in
  two steps. #159 (cuts 1b-4, post-2026-07-07) retired the built-in `--strategy`
  sweep/demo surface: the inline `aura sweep --strategy r-sma --real` path no longer
  exists. #220 then removed the verbs' weld — at milestone close the dissolved form of
  generalize/walkforward/mc was still the welded `--strategy r-sma --real`; #220 made
  the three verbs blueprint-generic (arbitrary blueprint + arbitrary `--axis`; mc keeps
  the synthetic `--seeds` family unchanged beside the new `--real --axis` campaign mode;
  generalize takes `--real <SYM1,SYM2,…>`, >=2, with one value per axis), deleted the
  welded `--strategy` grammar (clap rejects it, exit 2), dissolved `RGrid`, and unified
  the verbs on the #214 invocation struct. The surviving form everywhere is `aura <verb>
  <blueprint.json> --real … --axis …` over examples/r_*.json.] Milestone closed 2026-07-07 on a green end-to-end fieldtest
  (0 bugs; behaviour preservation, campaign-substrate reach-through, and the risk-regime axis
  all confirmed); residual findings are discoverability/ergonomics follow-ups (#216 risk-axis
  discoverability, #217 verb knob asymmetry, #218 no-project store litter).
  **Amendment (2026-07-14, #256 fork B):** the dissolved walkforward/mc
  translations' leading sweep became the enumerate-only **`std::grid`** stage —
  a fieldless vocabulary block, legal only as the first stage and immediately
  before `std::walk_forward`. Only the grid's parameter points ever crossed the
  stage seam (the wf stage re-sweeps them per IS window itself), so the leading
  stage no longer executes members or persists a `Sweep` registry family:
  persisted dissolved-walkforward/mc campaign documents lose that family, while
  the visible grades are byte-identical (the exact-grade E2E pins are
  unchanged). The executor's inter-stage seam is now a typed two-armed flow
  (points-only vs executed members); report-consuming stages are fenced by
  preflight. `translate_generalize` keeps its executed `std::sweep(argmax)`
  leading stage — its generalize stage consumes the argmax winner report as the
  cell nominee.

- **Inferential honesty of the World — family-selection without false-discovery
  control (tracked: milestone "Inferential validation (defend against false
  discovery at sweep scale)", #144 / #145 / #146; adjacent #139).** The World's
  massively-parallel sweep (C12 axes 1–2, C21) is aura's differentiator *and* its
  most direct route to false discovery: `optimize` / `rank_by` select the single
  best family member by a bare argmax, with no penalty for the number of
  configurations tried, no preference for a robust parameter neighbourhood over a
  sharp in-sample peak, and no cross-instrument generalization read; `mc` is a
  descriptive seed-sweep, not an inferential significance test. The hygiene
  invariants keep one backtest honest (C1 determinism, C2 no look-ahead); they do
  **not** make a *selection across a family* honest. Recognized direction: the
  selection/aggregation layer must deflate a winner for the family size (#144),
  prefer a plateau over a peak (#145), and score cross-instrument generalization
  (#146), with a per-candidate out-of-sample bootstrap as the adjacent
  significance read (#139, landed cycle 0075). Distinct from the two threads above —
  neither orchestration *composability* (#109) nor a search *policy* (Bayesian/
  genetic), but the statistical *validity* of the selection itself. Not a
  C-invariant until built; recorded as the World's third half — **now structurally
  complete** (all three pieces built, cycle 0078).
  **Status (cycle 0076):** the trials-deflation piece (#144) landed —
  `optimize_deflated` (aura-registry) wraps `optimize` and records, additively on
  the winning member's manifest (C18, the new `RunManifest.selection`), a deflated
  score + an overfit probability for the family size, **without changing which
  member wins** (C23; `optimize` / `rank_by` stay a bare argmax — the deflation is
  recorded provenance, not a re-ranking). The R arm is a centred moving-block
  reality-check (reusing the `r_bootstrap` kernel); the `total_pips` arm a
  closed-form expected-max-of-K dispersion floor. **Status (cycle 0077):** the
  plateau-over-peak piece (#145) landed — `optimize_plateau` (aura-registry)
  argmaxes the **neighbourhood-smoothed** metric surface (mean or worst-case over
  each member's closed mixed-radix grid neighbourhood) instead of the bare peak,
  recorded on the same `RunManifest.selection` carrier, now an orthogonal *rule ×
  annotation*: `SelectionMode::{Argmax, PlateauMean, PlateauWorst}` with either the
  deflation annotation (#144) or the plateau annotation (`neighbourhood_score` /
  `n_neighbours`). It is opt-in via a `--select` flag — default argmax stays
  byte-identical (C23). The grid lattice surfaces from the engine
  (`SweepBinder::sweep_with_lattice`); the policy stays in aura-registry with
  `walk_forward` selection-agnostic (C9). Selection is now a *pluggable objective*
  (bare argmax → trials-deflated → plateau), not a single hard-wired argmax.
  **Status (cycle 0078):** the last piece — cross-instrument generalization (#146) —
  landed, completing the inferential half. Unlike #144/#145 (which *select + annotate*
  a within-family winner on `RunManifest.selection`), #146 is an **aggregator/
  validator**, not a selector: the `aura generalize` subcommand runs a *brought*
  candidate across an instrument list and `generalization` (aura-registry) reduces the
  per-instrument R-metrics to a **worst-case floor** (min over instruments — the
  cross-instrument sibling of `PlateauMode::Worst`, R-only per C10) + a sign-agreement
  count + the per-instrument breakdown. It is a *recomputable aggregate* over a new
  `FamilyKind::CrossInstrument` family (C12's comparison axis, realized), each member
  self-identifying via the new `RunManifest.instrument` lineage field (C18) — *not*
  stamped on `FamilySelection` (there is no within-family winner to annotate). The
  inferential half is now structurally built; the per-candidate OOS bootstrap (#139,
  cycle 0075) is its adjacent significance read.

- **`aura-std` contents** — substantively populated (~30 node modules): SMA/EMA,
  arithmetic (`Add`/`Sub`/`Mul`/`Sqrt`/`LinComb`), logic (`And`/`Gt`/`Latch`/`EqConst`),
  the `Delay` z⁻¹ register, `Resample`, `RollingMin`/`RollingMax`, `Session`, the R chain
  (`Bias`/`FixedStop`/`Sizer`/`PositionManagement`), the legacy `SimBroker` pip yardstick,
  the cost-model graph (`CostNode`/`CostRunner`/`CostSum` + `ConstantCost`/`VolSlippageCost`/
  `CarryCost`), and the `Recorder`/`GatedRecorder`/`SeriesReducer` sinks. Further blocks
  land demand-driven as the walking-skeleton needs them.

- **Sequencing** — the runnable single-harness *substrate* comes first (walking
  skeleton: a closed harness that runs deterministically and records via a sink);
  the World/meta layer (C21) and the playground trace-explorer (C22) are the
  differentiating layers that follow — you orchestrate and visualize a thing that
  must first run once.