diff --git a/docs/specs/0045-registry-lineage-families.md b/docs/specs/0045-registry-lineage-families.md new file mode 100644 index 0000000..0cc18b5 --- /dev/null +++ b/docs/specs/0045-registry-lineage-families.md @@ -0,0 +1,429 @@ +# Registry lineage for orchestration families — Design Spec + +**Date:** 2026-06-15 +**Status:** Draft — awaiting user spec review +**Authors:** orchestrator + Claude + +## Goal + +Make an orchestration **family** (sweep / Monte-Carlo / walk-forward) a +first-class, persisted unit in the run registry (C18 / C21). Today +`aura-registry` stores flat `RunReport` lines with no links between them: a +sweep's points and an MC's draws land as loose runs, indistinguishable from +standalone `aura run` records. This cycle adds a **lineage link** — a shared +*family id* (the parent reference) stamped on each member record — so a family +can be re-listed, ranked, and re-derived **as a unit**, and surfaces it through +the CLI (`aura mc`, family-aware `aura runs`). + +Two invariants shape the design: + +- **Producer-agnostic lineage (C18 + C6, #71 firewall).** Re-deriving a family + member means **re-running** it from `(commit + params + window + seed)` — the + fields its `RunManifest` already carries — never replaying a stored input + artifact. No recorded blob / file path / byte payload enters a manifest or a + lineage record. The member's manifest *is* its re-derivation recipe. +- **Producer-supplied window (the comment's named anti-pattern).** The + `(Timestamp, Timestamp)` window stored in a lineage member must come from the + **producer**, not from scanning a materialized `Vec` at the run call-site + (today's `prices.first()/.last()`). "A lazy producer knows its bounds without + materializing, so the stored lineage stays byte-identical whether the source + is eager or streamed." This cycle adds a `Source::bounds()` accessor and routes + every family-feeding run window through it. + +## Architecture + +Three layers move, none of them the strategy graph (brokers/sinks/strategies are +untouched): + +1. **Engine — producer-supplied window (`aura-engine`).** The `Source` trait + gains a read-only `bounds()` accessor reporting the inclusive data extent a + producer will stream, known without materialization. A free `window_of` + helper folds it across a run's sources (the union extent). The CLI's + `prices.first()/.last()` manifest-window scans are replaced by `window_of`. + +2. **Registry — the lineage record + family store (`aura-registry`).** A new + `FamilyRunRecord` (a `RunReport` stamped with `family_id` + `kind` + + `ordinal`) is the persisted lineage unit. A family is stored as its **related + member records** in a sibling JSONL store (`families.jsonl`), leaving the flat + `runs.jsonl` path and its `append`/`load`/`rank_by`/`optimize` API byte-for- + byte unchanged. The `family_id` is **`"{name}-{counter}"`**: a user-supplied + `name` (a builder-function argument, surfaced on the CLI as `--name` with a + per-kind default) plus a per-name `counter` (a run-index) assigned at + persistence. `group_families` re-derives the families by grouping on + `family_id` (the round-trip). + +3. **CLI — family surface (`aura-cli`).** `aura mc` is added (it runs + persists + a Monte-Carlo family — `McFamily` exists in the engine but had no CLI). `aura + sweep` / `aura walkforward` / `aura mc` persist to the family store via + `append_family`, each taking an optional `--name`. `aura runs` becomes family- + aware: `aura runs families` lists family headers, `aura runs family [rank + ]` lists/ranks one family's members as a unit. + +Layering (C9 preserved): `aura-engine` gains no registry/lineage concept — it +only learns to report a producer's bounds. Lineage lives in `aura-registry` +(which already depends on `aura-engine` and imports `SweepFamily`/`SweepPoint`); +it is "the World's memory" (C18/C22), not an engine concern. + +### Why name + counter (the id form) + +The `family_id` is used for exactly two things: the **grouping key** for the +round-trip (`group_families`) and the **user-facing lookup handle** for the CLI +(`aura runs family `). Neither needs the id to be *reproducible* from the +family's inputs — grouping needs only shared-within / distinct-across, and the +user reads the id from `aura runs families` and pastes it, never computes it. + +A **user-supplied name** makes the registry human-navigable (`aura runs family +macd-robustness-0`), matching the "World = named experiments" framing (C21). A +**counter** suffix guarantees uniqueness without a new dependency and without +nondeterministic entropy (a UUID would introduce the system's first unseeded +randomness, against the engine's seeded-determinism discipline — it hand-rolls +`SplitMix64` precisely to avoid that). Re-running a family mints a *new* id +(`name-1` after `name-0`) — a fresh family record, which is the honest meaning of +"I ran the experiment again". A content-addressed (reproducible-hash) identity +plus replay-dedup is a deliberate later refinement, not this cycle (Non-goals). + +## Concrete code shapes + +### User-facing program (the acceptance evidence) + +The researcher drives families from the CLI and gets back records that round-trip +as a unit. New / changed invocations: + +```console +$ aura mc # NEW: run + persist a Monte-Carlo family (default name "mc") +{"family_id":"mc-0","seed":1,"report":{"manifest":{...,"seed":1,...},"metrics":{...}}} +{"family_id":"mc-0","seed":2,"report":{...}} +{"family_id":"mc-0","seed":3,"report":{...}} +{"mc_aggregate":{"total_pips":{"mean":...,"p50":...},"max_drawdown":{...},"exposure_sign_flips":{...}}} + +$ aura mc --name macd-robustness # NEW: author-supplied name; counter disambiguates re-runs +{"family_id":"macd-robustness-0","seed":1,"report":{...}} +... + +$ aura runs families # NEW: one header line per stored family +{"family_id":"mc-0","kind":"MonteCarlo","members":3} +{"family_id":"sweep-0","kind":"Sweep","members":4} +{"family_id":"walkforward-0","kind":"WalkForward","members":3} + +$ aura runs family mc-0 rank total_pips # NEW: rank one family as a unit +{"family_id":"mc-0","seed":2,"report":{...,"metrics":{"total_pips":4.1,...}}} +{"family_id":"mc-0","seed":1,"report":{...,"metrics":{"total_pips":2.7,...}}} +{"family_id":"mc-0","seed":3,"report":{...,"metrics":{"total_pips":-0.4,...}}} + +$ aura runs list # UNCHANGED: standalone (family-less) runs only +{"manifest":{...},"metrics":{...}} +``` + +The Rust authoring shape (the World is authored in Rust, C10) — the lineage +round-trip a harness-family program writes: + +```rust +use aura_registry::{group_families, mc_member_reports, FamilyKind, Registry}; + +let family = aura_engine::monte_carlo(&base_point, &[1, 2, 3], run_draw); +let reg = Registry::open("runs/runs.jsonl"); +let id = reg.append_family("macd-robustness", FamilyKind::MonteCarlo, + &mc_member_reports(&family))?; // -> "macd-robustness-0" + +// round-trip: the stored links re-derive the family as a unit +let families = group_families(reg.load_family_members()?); +assert_eq!(families.len(), 1); +assert_eq!(families[0].id, "macd-robustness-0"); +assert_eq!(families[0].kind, FamilyKind::MonteCarlo); +assert_eq!(families[0].members.len(), 3); // re-listed as a unit, ordinal-ordered + +// a second run of the same name mints a fresh family id (a new experiment run) +let id2 = reg.append_family("macd-robustness", FamilyKind::MonteCarlo, + &mc_member_reports(&family))?; // -> "macd-robustness-1" +assert_ne!(id, id2); +``` + +### Implementation shapes (secondary — before → after) + +**1. `Source::bounds()` + `window_of` (`aura-engine`, harness.rs).** + +```rust +// BEFORE — the producer knows only its head; the window is scanned at the call site +pub trait Source { + fn peek(&self) -> Option; + fn next(&mut self) -> Option<(Timestamp, Scalar)>; +} +// in aura-cli: let window = (prices.first()?.0, prices.last()?.0); // <- the named anti-pattern + +// AFTER — the producer reports its data extent without materialization +pub trait Source { + fn peek(&self) -> Option; + fn next(&mut self) -> Option<(Timestamp, Scalar)>; + /// The inclusive `(from, to)` data extent this producer will stream, known + /// without materializing it (#71/C18): a lazy producer reports its window + /// bounds; an eager one reports its buffer's first/last. `None` = empty / + /// open-ended. Cursor-independent: a property of the data extent, not the + /// read position. + fn bounds(&self) -> Option<(Timestamp, Timestamp)>; +} + +/// The window of a run: the union extent across its sources (min from, max to). +/// `None` when every source is empty/open-ended. +pub fn window_of(sources: &[Box]) -> Option<(Timestamp, Timestamp)>; +``` + +`VecSource::bounds` → `Some((stream.first()?.0, stream.last()?.0))` (the eager +source reads its own buffer — O(1) on the slice ends; the *consumer* no longer +reaches into a `Vec`). `SyntheticSpec::source` returns a `VecSource`, so it is +covered automatically (its generated stream's first/last). `M1FieldSource::bounds` +→ the requested `(from_ms, to_ms)` normalized to epoch-ns `Timestamp` when both +are given, else `None` (open-ended; see Non-goals). + +CLI migration: `run_sample`, `sweep_family`, `sweep_over`, `run_oos`, and the new +`aura mc` build their `Vec>` first, read `window_of(&sources)`, +then `h.run(sources)` — no `prices.first()/.last()`. Byte-identical for the +synthetic demos (`VecSource::bounds` == the old first/last). + +**2. Lineage record + family store (`aura-registry`).** + +```rust +// NEW types +/// Which C12 orchestration axis produced a family. +#[derive(Clone, Copy, Debug, PartialEq, Eq, serde::Serialize, serde::Deserialize)] +pub enum FamilyKind { Sweep, MonteCarlo, WalkForward } + +/// One persisted family member: a run record stamped with its lineage link. +/// `family_id` is the shared parent reference; `ordinal` is the member's +/// position in the family's canonical order (odometer / seed-input / roll). +#[derive(Clone, Debug, PartialEq, serde::Serialize, serde::Deserialize)] +pub struct FamilyRunRecord { + pub family_id: String, + pub kind: FamilyKind, + pub ordinal: usize, + pub report: RunReport, +} + +/// A re-derived family: the members sharing one `family_id`, ordinal-sorted — +/// the round-trip result of [`group_families`]. +#[derive(Clone, Debug, PartialEq)] +pub struct Family { + pub id: String, + pub kind: FamilyKind, + pub members: Vec, +} + +impl Registry { + /// Assign a fresh `family_id = "{name}-{counter}"` — `counter` is the smallest + /// `k` for which `"{name}-{k}"` is not already in the family store (a per-name + /// run-index) — stamp `reports` as ordinal-ordered members, append them as N + /// JSONL lines to the family store (a sibling of the flat runs store, + /// `self.path.with_file_name("families.jsonl")`, so the flat path is + /// untouched), and return the assigned id. Reads the store once to pick the + /// counter (a read-before-write; single-process CLI invocations do not race — + /// see Error handling). + pub fn append_family(&self, name: &str, kind: FamilyKind, reports: &[RunReport]) + -> Result; + /// Parse every stored family member, in file order. A missing file is empty. + pub fn load_family_members(&self) -> Result, RegistryError>; +} + +/// Group member records into families by `family_id` (first-seen family order; +/// each family's members ordinal-sorted). The round-trip: `append_family(...)` +/// then `group_families(load_family_members())` recovers the families — the +/// stored links re-derive the family. +pub fn group_families(records: Vec) -> Vec; + +/// The member reports of an engine family, in canonical order — the trivial +/// extractors the CLI hands to `append_family` (they locate the per-kind member +/// report: `SweepPoint.report` / `McDraw.report` / `WindowRun.oos_report`). +pub fn sweep_member_reports(family: &SweepFamily) -> Vec; +pub fn mc_member_reports(family: &McFamily) -> Vec; +pub fn walkforward_member_reports(result: &WalkForwardResult) -> Vec; +``` + +`next_family_id` (module-private) generates `"{name}-0"`, `"{name}-1"`, … and +returns the first not present in the loaded id set — robust to `-` inside `name` +(it tests membership of generated candidates, never parses the string). The flat +`Registry::append(&RunReport)` / `load() -> Vec` and `rank_by` / +`optimize` are unchanged (no migration of `runs.jsonl`). + +**3. CLI subcommands (`aura-cli`, main.rs).** + +```rust +// BEFORE +const USAGE: &str = "usage: aura run [...] | ... | aura sweep | aura walkforward | aura runs list | aura runs rank "; +match args { + ["sweep"] => run_sweep(), // appends flat reports to runs.jsonl + ["walkforward"] => run_walkforward(), // appends flat reports to runs.jsonl + ["runs", "list"] => runs_list(), + ["runs", "rank", metric] => runs_rank(metric), + ... +} + +// AFTER +const USAGE: &str = "usage: aura run [...] | ... | aura sweep [--name ] | aura mc [--name ] | aura walkforward [--name ] | aura runs list | aura runs rank | aura runs families | aura runs family [rank ]"; +match args { + ["sweep"] => run_sweep("sweep"), + ["sweep", "--name", n] => run_sweep(n), + ["mc"] => run_mc("mc"), // NEW: monte_carlo over a built-in seed set + append_family + ["mc", "--name", n] => run_mc(n), + ["walkforward"] => run_walkforward("walkforward"), + ["walkforward", "--name", n] => run_walkforward(n), + ["runs", "list"] => runs_list(), // unchanged — standalone runs + ["runs", "rank", metric] => runs_rank(metric), + ["runs", "families"] => runs_families(), // NEW + ["runs", "family", id] => runs_family(id, None), // NEW + ["runs", "family", id, "rank", metric] => runs_family(id, Some(metric)), // NEW + ... +} +``` + +`aura sweep`/`walkforward`/`mc` move from flat-appending to family-appending via +`append_family(name, kind, &_member_reports(&family))`, printing the +returned `family_id` on each member line (the pre-lineage flat-storage stopgap is +replaced by proper family storage — the point of the cycle). `runs_family`'s +`rank` reuses the existing `rank_by` over the family's member reports (within- +family ranking). A non-persisting render helper (`mc_report`, mirroring the +existing `sweep_report`/`walkforward_report` `#[cfg(test)]` helpers) carries the +C1-determinism test of the family *computation*, separate from the store- +dependent id. + +## Components + +- **`aura-engine::harness`** — `Source::bounds()` (new trait method, object-safe), + `VecSource`/`SyntheticSpec` covered, `window_of` free function. No new dep + (engine already has serde). +- **`aura-ingest::M1FieldSource`** — store the requested `(from_ms, to_ms)` at + `open`; impl `bounds()` (requested window → epoch-ns when bounded, else `None`). +- **`aura-registry`** — new `lineage` module: `FamilyKind`, `FamilyRunRecord`, + `Family`, `group_families`, `next_family_id`, the three `*_member_reports` + extractors; `Registry::append_family` / `load_family_members`. New dep: + `serde = { workspace = true, features = ["derive"] }` (the crate has + `serde_json` but not `serde` derive; the lineage types need it — admitted under + the per-case dependency policy, same basis as `serde_json` already is). `HashSet` + for the counter is `std` (no dep). +- **`aura-cli`** — `run_mc`, `runs_families`, `runs_family`, `mc_report` test + helper; `run_sweep` / `run_walkforward` switch to `append_family` and take a + `name`; the `window_of` migration of the manifest-window scans; `USAGE` + `match` + arms (incl. `--name`). + +## Data flow + +``` +author Rust family program / aura {sweep,mc,walkforward} [--name N] + -> engine: monte_carlo / sweep / walk_forward (disjoint C1 runs) + per member: build sources -> window_of(&sources) -> h.run(sources) + -> RunReport{ manifest{ commit, params, window(producer-supplied), seed }, metrics } + -> registry: _member_reports(&family) -> Vec (canonical order) + -> Registry::append_family(name, kind, &reports): + counter = smallest k with "{name}-{k}" absent in the store + family_id = "{name}-{counter}" + -> Vec -> runs/families.jsonl (N lines) + -> returns family_id + +read back: + Registry::load_family_members -> Vec + -> group_families -> Vec + CLI: runs_families (headers) | runs_family(id) (members) | runs_family(id, rank metric) (rank_by member reports) +``` + +## Error handling + +- `append_family` / `load_family_members` reuse `RegistryError` (`Io`, `Parse{line,…}`). + A missing `families.jsonl` is an empty family store (`Ok(vec![])`), exactly as + `load` treats a missing `runs.jsonl`. +- **Counter read-before-write.** `append_family` loads the family store to pick + the next counter, so it is a read-then-append (not a blind append). Under + concurrent writers two families of the same name could race to the same counter; + the CLI runs one command per process, so this is a latent boundary, not an + active one — noted, not guarded this cycle (a future concern if the World gains + concurrent in-process family persistence). +- `aura runs family ` for an unknown id: print nothing (an empty family) and + exit 0 — consistent with `runs list` over an empty registry (a non-existent + family is not an error, just empty). `aura runs family rank ` + surfaces `RegistryError::UnknownMetric` (stderr + exit 2), as `runs rank` does. +- `window_of` returning `None` (all sources empty/open-ended) at a family call + site is a wiring/usage condition: the synthetic demos never hit it (their + `VecSource`s are non-empty); a defensive `expect` with a clear message guards it + (the same shape as today's `prices.first().expect("non-empty stream")`). + +## Testing strategy + +Engine (`aura-engine`): +1. `VecSource::bounds` returns `Some((first_ts, last_ts))` of its buffer and + `None` for an empty buffer; cursor-independent (bounds unchanged after `next`). +2. `window_of` folds the union extent across multiple sources (min from, max to) + and is `None` for all-empty. +3. `SyntheticSpec::source(seed).bounds()` equals the analytic + `(Timestamp(1), Timestamp((len-1)*step+1))`. + +Ingest (`aura-ingest`): +4. `M1FieldSource::bounds()` over an explicit `[from_ms, to_ms]` returns those + bounds normalized to epoch-ns (no bar streamed); data-gated like the existing + `streaming_seam` tests (skip when the local archive is absent). + +Registry (`aura-registry`) — the headline round-trips: +5. **Lineage round-trips:** `append_family("f", kind, reports)` then + `group_families(load_family_members())` recovers one `Family` whose `members` + equal the stamped reports (ordinal-ordered), `kind` matches, and `id == "f-0"` + (the stored links re-derive the family — acceptance bullet 3). +6. **Per-name counter:** two `append_family("x", …)` calls yield `"x-0"` then + `"x-1"`; an `append_family("y", …)` yields `"y-0"` (independent per-name + counters); `group_families` then returns three families. +7. **Distinct families stay distinct:** an interleaved multi-family file regroups + into the right families, members ordinal-sorted, families in first-seen order. +8. **Flat path untouched:** the existing `append`/`load`/`rank_by`/`optimize` + tests stay green; a `families.jsonl` write leaves `runs.jsonl` empty and vice + versa (the two stores are disjoint siblings). +9. **Rank a family as a unit:** `rank_by(family.members → reports, "total_pips")` + orders the family best-first (reuses the existing `rank_by` proof). + +CLI (`aura-cli`): +10. `aura mc` renders one member line per seed sharing one `family_id` plus the + aggregate line; the non-persisting `mc_report()` render is deterministic + across two calls (C1 of the family computation, mirroring + `sweep_report_is_deterministic`). +11. `run_sweep`/`run_walkforward`/`run_mc` persist a family (against a fresh temp + store) that `group_families` recovers as one unit of the right `kind` and + member count, with id `"{name}-0"`. +12. The `window_of` migration keeps `run_sample` byte-identical (`window == (1,7)`) + — the existing `run_sample_is_deterministic_and_non_trivial` stays green. + +## Acceptance criteria + +- **A family run writes related records:** `aura sweep` / `aura mc` / + `aura walkforward` persist `FamilyRunRecord` lines sharing one `family_id` + (`"{name}-{counter}"`) to the family store (test 11). +- **The family can be listed and ranked as a unit:** `aura runs families` lists + headers; `aura runs family [rank ]` lists/ranks one family's + members (tests 9, 10, 11). +- **Lineage round-trips:** the stored links re-derive the family + (`group_families` ∘ `load_family_members` recovers the written family — test 5); + a same-name re-run mints a fresh `"{name}-{counter}"` (test 6). +- **Producer-supplied window:** every family-feeding run's manifest window comes + from `Source::bounds()` via `window_of`, not a `Vec` scan; lineage is byte- + identical whether the source is eager or streamed (tests 1–3, 12). +- **Producer-agnostic re-derivation (C18 + #71):** no input-stream field (blob / + path / payload) is added to `RunManifest` or any lineage record; a member's + `(commit, params, window, seed)` manifest is its sole re-derivation recipe. +- **No regression to the flat registry or C9 layering:** `runs.jsonl` and its API + are byte-for-byte unchanged (test 8); `aura-engine` gains no registry/lineage + dependency. + +## Non-goals + +- **Reproducible / content-addressed family identity.** The `family_id` is an + assigned `"{name}-{counter}"`, not a hash of the family's inputs; re-running the + same family mints a new id. A content-addressed identity (same inputs → same id) + with the replay-dedup it implies against an append-only store is a deliberate + later refinement, deferred to a follow-up issue. +- **Unbounded real-data archive extent.** `M1FieldSource::bounds()` reports the + *requested* `[from_ms, to_ms]` window; an open-ended source (no window given) + returns `None`. Deriving a symbol's full archive extent without streaming (a + `DataServer` extent query) is deferred — `aura run --real ` without + `--from/--to` is a *single* run, not a family, and keeps its current probe-based + window. No lineage member this cycle is open-ended (all are `VecSource`-backed + synthetic demos). +- **Cross-family comparison / run-diff.** C18's "run-diff" depth and ranking + *families against each other* (vs. members within one family) are a later + refinement; this cycle delivers within-family listing/ranking + family headers. +- **A typed param-space in the manifest.** `RunManifest.params` stays the + `Vec<(String, f64)>` precursor (unchanged). +- **Random param-sweep enumeration (#52), sweep named-binding (#57).** Other + milestone items; #70 persists the families they (and the existing builders) + produce.