spec: 0045 registry lineage for orchestration families (boss-signed)

Persist sweep / Monte-Carlo / walk-forward runs as named, linked families (C18 lineage / C21): a FamilyRunRecord stamps each member with a shared family_id = "{name}-{counter}" in a sibling families.jsonl store, group_families re-derives a family as a unit, and the CLI gains aura mc + family-aware aura runs. Producer-supplied window via Source::bounds()/window_of (the #71 firewall: no Vec scan, no input artifact). family_id form (name + counter, not a content-hash) resolved by the user in-session; reconciliation comment on the issue records it. Auto-signed under the boss spec-auto-sign gate: all objective gates green (precondition, self-review, grounding-check PASS) and a unanimous five-lens spec-skeptic panel (criterion, grounding, scope-fork, ambiguity, plan-readiness). refs #70
2026-06-15 17:29:27 +02:00
parent eeba2182d7
commit bf829f605b
1 changed files with 429 additions and 0 deletions
@@ -0,0 +1,429 @@
+# Registry lineage for orchestration families — Design Spec
+
+**Date:** 2026-06-15
+**Status:** Draft — awaiting user spec review
+**Authors:** orchestrator + Claude
+
+## Goal
+
+Make an orchestration **family** (sweep / Monte-Carlo / walk-forward) a
+first-class, persisted unit in the run registry (C18 / C21). Today
+`aura-registry` stores flat `RunReport` lines with no links between them: a
+sweep's points and an MC's draws land as loose runs, indistinguishable from
+standalone `aura run` records. This cycle adds a **lineage link** — a shared
+*family id* (the parent reference) stamped on each member record — so a family
+can be re-listed, ranked, and re-derived **as a unit**, and surfaces it through
+the CLI (`aura mc`, family-aware `aura runs`).
+
+Two invariants shape the design:
+
+- **Producer-agnostic lineage (C18 + C6, #71 firewall).** Re-deriving a family
+  member means **re-running** it from `(commit + params + window + seed)` — the
+  fields its `RunManifest` already carries — never replaying a stored input
+  artifact. No recorded blob / file path / byte payload enters a manifest or a
+  lineage record. The member's manifest *is* its re-derivation recipe.
+- **Producer-supplied window (the comment's named anti-pattern).** The
+  `(Timestamp, Timestamp)` window stored in a lineage member must come from the
+  **producer**, not from scanning a materialized `Vec` at the run call-site
+  (today's `prices.first()/.last()`). "A lazy producer knows its bounds without
+  materializing, so the stored lineage stays byte-identical whether the source
+  is eager or streamed." This cycle adds a `Source::bounds()` accessor and routes
+  every family-feeding run window through it.
+
+## Architecture
+
+Three layers move, none of them the strategy graph (brokers/sinks/strategies are
+untouched):
+
+1. **Engine — producer-supplied window (`aura-engine`).** The `Source` trait
+   gains a read-only `bounds()` accessor reporting the inclusive data extent a
+   producer will stream, known without materialization. A free `window_of`
+   helper folds it across a run's sources (the union extent). The CLI's
+   `prices.first()/.last()` manifest-window scans are replaced by `window_of`.
+
+2. **Registry — the lineage record + family store (`aura-registry`).** A new
+   `FamilyRunRecord` (a `RunReport` stamped with `family_id` + `kind` +
+   `ordinal`) is the persisted lineage unit. A family is stored as its **related
+   member records** in a sibling JSONL store (`families.jsonl`), leaving the flat
+   `runs.jsonl` path and its `append`/`load`/`rank_by`/`optimize` API byte-for-
+   byte unchanged. The `family_id` is **`"{name}-{counter}"`**: a user-supplied
+   `name` (a builder-function argument, surfaced on the CLI as `--name` with a
+   per-kind default) plus a per-name `counter` (a run-index) assigned at
+   persistence. `group_families` re-derives the families by grouping on
+   `family_id` (the round-trip).
+
+3. **CLI — family surface (`aura-cli`).** `aura mc` is added (it runs + persists
+   a Monte-Carlo family — `McFamily` exists in the engine but had no CLI). `aura
+   sweep` / `aura walkforward` / `aura mc` persist to the family store via
+   `append_family`, each taking an optional `--name`. `aura runs` becomes family-
+   aware: `aura runs families` lists family headers, `aura runs family <id> [rank
+   <metric>]` lists/ranks one family's members as a unit.
+
+Layering (C9 preserved): `aura-engine` gains no registry/lineage concept — it
+only learns to report a producer's bounds. Lineage lives in `aura-registry`
+(which already depends on `aura-engine` and imports `SweepFamily`/`SweepPoint`);
+it is "the World's memory" (C18/C22), not an engine concern.
+
+### Why name + counter (the id form)
+
+The `family_id` is used for exactly two things: the **grouping key** for the
+round-trip (`group_families`) and the **user-facing lookup handle** for the CLI
+(`aura runs family <id>`). Neither needs the id to be *reproducible* from the
+family's inputs — grouping needs only shared-within / distinct-across, and the
+user reads the id from `aura runs families` and pastes it, never computes it.
+
+A **user-supplied name** makes the registry human-navigable (`aura runs family
+macd-robustness-0`), matching the "World = named experiments" framing (C21). A
+**counter** suffix guarantees uniqueness without a new dependency and without
+nondeterministic entropy (a UUID would introduce the system's first unseeded
+randomness, against the engine's seeded-determinism discipline — it hand-rolls
+`SplitMix64` precisely to avoid that). Re-running a family mints a *new* id
+(`name-1` after `name-0`) — a fresh family record, which is the honest meaning of
+"I ran the experiment again". A content-addressed (reproducible-hash) identity
+plus replay-dedup is a deliberate later refinement, not this cycle (Non-goals).
+
+## Concrete code shapes
+
+### User-facing program (the acceptance evidence)
+
+The researcher drives families from the CLI and gets back records that round-trip
+as a unit. New / changed invocations:
+
+```console
+$ aura mc                       # NEW: run + persist a Monte-Carlo family (default name "mc")
+{"family_id":"mc-0","seed":1,"report":{"manifest":{...,"seed":1,...},"metrics":{...}}}
+{"family_id":"mc-0","seed":2,"report":{...}}
+{"family_id":"mc-0","seed":3,"report":{...}}
+{"mc_aggregate":{"total_pips":{"mean":...,"p50":...},"max_drawdown":{...},"exposure_sign_flips":{...}}}
+
+$ aura mc --name macd-robustness   # NEW: author-supplied name; counter disambiguates re-runs
+{"family_id":"macd-robustness-0","seed":1,"report":{...}}
+...
+
+$ aura runs families            # NEW: one header line per stored family
+{"family_id":"mc-0","kind":"MonteCarlo","members":3}
+{"family_id":"sweep-0","kind":"Sweep","members":4}
+{"family_id":"walkforward-0","kind":"WalkForward","members":3}
+
+$ aura runs family mc-0 rank total_pips     # NEW: rank one family as a unit
+{"family_id":"mc-0","seed":2,"report":{...,"metrics":{"total_pips":4.1,...}}}
+{"family_id":"mc-0","seed":1,"report":{...,"metrics":{"total_pips":2.7,...}}}
+{"family_id":"mc-0","seed":3,"report":{...,"metrics":{"total_pips":-0.4,...}}}
+
+$ aura runs list                # UNCHANGED: standalone (family-less) runs only
+{"manifest":{...},"metrics":{...}}
+```
+
+The Rust authoring shape (the World is authored in Rust, C10) — the lineage
+round-trip a harness-family program writes:
+
+```rust
+use aura_registry::{group_families, mc_member_reports, FamilyKind, Registry};
+
+let family = aura_engine::monte_carlo(&base_point, &[1, 2, 3], run_draw);
+let reg = Registry::open("runs/runs.jsonl");
+let id = reg.append_family("macd-robustness", FamilyKind::MonteCarlo,
+                           &mc_member_reports(&family))?;  // -> "macd-robustness-0"
+
+// round-trip: the stored links re-derive the family as a unit
+let families = group_families(reg.load_family_members()?);
+assert_eq!(families.len(), 1);
+assert_eq!(families[0].id, "macd-robustness-0");
+assert_eq!(families[0].kind, FamilyKind::MonteCarlo);
+assert_eq!(families[0].members.len(), 3);                  // re-listed as a unit, ordinal-ordered
+
+// a second run of the same name mints a fresh family id (a new experiment run)
+let id2 = reg.append_family("macd-robustness", FamilyKind::MonteCarlo,
+                            &mc_member_reports(&family))?; // -> "macd-robustness-1"
+assert_ne!(id, id2);
+```
+
+### Implementation shapes (secondary — before → after)
+
+**1. `Source::bounds()` + `window_of` (`aura-engine`, harness.rs).**
+
+```rust
+// BEFORE — the producer knows only its head; the window is scanned at the call site
+pub trait Source {
+    fn peek(&self) -> Option<Timestamp>;
+    fn next(&mut self) -> Option<(Timestamp, Scalar)>;
+}
+// in aura-cli: let window = (prices.first()?.0, prices.last()?.0);  // <- the named anti-pattern
+
+// AFTER — the producer reports its data extent without materialization
+pub trait Source {
+    fn peek(&self) -> Option<Timestamp>;
+    fn next(&mut self) -> Option<(Timestamp, Scalar)>;
+    /// The inclusive `(from, to)` data extent this producer will stream, known
+    /// without materializing it (#71/C18): a lazy producer reports its window
+    /// bounds; an eager one reports its buffer's first/last. `None` = empty /
+    /// open-ended. Cursor-independent: a property of the data extent, not the
+    /// read position.
+    fn bounds(&self) -> Option<(Timestamp, Timestamp)>;
+}
+
+/// The window of a run: the union extent across its sources (min from, max to).
+/// `None` when every source is empty/open-ended.
+pub fn window_of(sources: &[Box<dyn Source>]) -> Option<(Timestamp, Timestamp)>;
+```
+
+`VecSource::bounds` → `Some((stream.first()?.0, stream.last()?.0))` (the eager
+source reads its own buffer — O(1) on the slice ends; the *consumer* no longer
+reaches into a `Vec`). `SyntheticSpec::source` returns a `VecSource`, so it is
+covered automatically (its generated stream's first/last). `M1FieldSource::bounds`
+→ the requested `(from_ms, to_ms)` normalized to epoch-ns `Timestamp` when both
+are given, else `None` (open-ended; see Non-goals).
+
+CLI migration: `run_sample`, `sweep_family`, `sweep_over`, `run_oos`, and the new
+`aura mc` build their `Vec<Box<dyn Source>>` first, read `window_of(&sources)`,
+then `h.run(sources)` — no `prices.first()/.last()`. Byte-identical for the
+synthetic demos (`VecSource::bounds` == the old first/last).
+
+**2. Lineage record + family store (`aura-registry`).**
+
+```rust
+// NEW types
+/// Which C12 orchestration axis produced a family.
+#[derive(Clone, Copy, Debug, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
+pub enum FamilyKind { Sweep, MonteCarlo, WalkForward }
+
+/// One persisted family member: a run record stamped with its lineage link.
+/// `family_id` is the shared parent reference; `ordinal` is the member's
+/// position in the family's canonical order (odometer / seed-input / roll).
+#[derive(Clone, Debug, PartialEq, serde::Serialize, serde::Deserialize)]
+pub struct FamilyRunRecord {
+    pub family_id: String,
+    pub kind: FamilyKind,
+    pub ordinal: usize,
+    pub report: RunReport,
+}
+
+/// A re-derived family: the members sharing one `family_id`, ordinal-sorted —
+/// the round-trip result of [`group_families`].
+#[derive(Clone, Debug, PartialEq)]
+pub struct Family {
+    pub id: String,
+    pub kind: FamilyKind,
+    pub members: Vec<FamilyRunRecord>,
+}
+
+impl Registry {
+    /// Assign a fresh `family_id = "{name}-{counter}"` — `counter` is the smallest
+    /// `k` for which `"{name}-{k}"` is not already in the family store (a per-name
+    /// run-index) — stamp `reports` as ordinal-ordered members, append them as N
+    /// JSONL lines to the family store (a sibling of the flat runs store,
+    /// `self.path.with_file_name("families.jsonl")`, so the flat path is
+    /// untouched), and return the assigned id. Reads the store once to pick the
+    /// counter (a read-before-write; single-process CLI invocations do not race —
+    /// see Error handling).
+    pub fn append_family(&self, name: &str, kind: FamilyKind, reports: &[RunReport])
+        -> Result<String, RegistryError>;
+    /// Parse every stored family member, in file order. A missing file is empty.
+    pub fn load_family_members(&self) -> Result<Vec<FamilyRunRecord>, RegistryError>;
+}
+
+/// Group member records into families by `family_id` (first-seen family order;
+/// each family's members ordinal-sorted). The round-trip: `append_family(...)`
+/// then `group_families(load_family_members())` recovers the families — the
+/// stored links re-derive the family.
+pub fn group_families(records: Vec<FamilyRunRecord>) -> Vec<Family>;
+
+/// The member reports of an engine family, in canonical order — the trivial
+/// extractors the CLI hands to `append_family` (they locate the per-kind member
+/// report: `SweepPoint.report` / `McDraw.report` / `WindowRun.oos_report`).
+pub fn sweep_member_reports(family: &SweepFamily) -> Vec<RunReport>;
+pub fn mc_member_reports(family: &McFamily) -> Vec<RunReport>;
+pub fn walkforward_member_reports(result: &WalkForwardResult) -> Vec<RunReport>;
+```
+
+`next_family_id` (module-private) generates `"{name}-0"`, `"{name}-1"`, … and
+returns the first not present in the loaded id set — robust to `-` inside `name`
+(it tests membership of generated candidates, never parses the string). The flat
+`Registry::append(&RunReport)` / `load() -> Vec<RunReport>` and `rank_by` /
+`optimize` are unchanged (no migration of `runs.jsonl`).
+
+**3. CLI subcommands (`aura-cli`, main.rs).**
+
+```rust
+// BEFORE
+const USAGE: &str = "usage: aura run [...] | ... | aura sweep | aura walkforward | aura runs list | aura runs rank <metric>";
+match args {
+    ["sweep"] => run_sweep(),                 // appends flat reports to runs.jsonl
+    ["walkforward"] => run_walkforward(),     // appends flat reports to runs.jsonl
+    ["runs", "list"] => runs_list(),
+    ["runs", "rank", metric] => runs_rank(metric),
+    ...
+}
+
+// AFTER
+const USAGE: &str = "usage: aura run [...] | ... | aura sweep [--name <n>] | aura mc [--name <n>] | aura walkforward [--name <n>] | aura runs list | aura runs rank <metric> | aura runs families | aura runs family <id> [rank <metric>]";
+match args {
+    ["sweep"] => run_sweep("sweep"),
+    ["sweep", "--name", n] => run_sweep(n),
+    ["mc"] => run_mc("mc"),                    // NEW: monte_carlo over a built-in seed set + append_family
+    ["mc", "--name", n] => run_mc(n),
+    ["walkforward"] => run_walkforward("walkforward"),
+    ["walkforward", "--name", n] => run_walkforward(n),
+    ["runs", "list"] => runs_list(),          // unchanged — standalone runs
+    ["runs", "rank", metric] => runs_rank(metric),
+    ["runs", "families"] => runs_families(),                       // NEW
+    ["runs", "family", id] => runs_family(id, None),              // NEW
+    ["runs", "family", id, "rank", metric] => runs_family(id, Some(metric)), // NEW
+    ...
+}
+```
+
+`aura sweep`/`walkforward`/`mc` move from flat-appending to family-appending via
+`append_family(name, kind, &<kind>_member_reports(&family))`, printing the
+returned `family_id` on each member line (the pre-lineage flat-storage stopgap is
+replaced by proper family storage — the point of the cycle). `runs_family`'s
+`rank` reuses the existing `rank_by` over the family's member reports (within-
+family ranking). A non-persisting render helper (`mc_report`, mirroring the
+existing `sweep_report`/`walkforward_report` `#[cfg(test)]` helpers) carries the
+C1-determinism test of the family *computation*, separate from the store-
+dependent id.
+
+## Components
+
+- **`aura-engine::harness`** — `Source::bounds()` (new trait method, object-safe),
+  `VecSource`/`SyntheticSpec` covered, `window_of` free function. No new dep
+  (engine already has serde).
+- **`aura-ingest::M1FieldSource`** — store the requested `(from_ms, to_ms)` at
+  `open`; impl `bounds()` (requested window → epoch-ns when bounded, else `None`).
+- **`aura-registry`** — new `lineage` module: `FamilyKind`, `FamilyRunRecord`,
+  `Family`, `group_families`, `next_family_id`, the three `*_member_reports`
+  extractors; `Registry::append_family` / `load_family_members`. New dep:
+  `serde = { workspace = true, features = ["derive"] }` (the crate has
+  `serde_json` but not `serde` derive; the lineage types need it — admitted under
+  the per-case dependency policy, same basis as `serde_json` already is). `HashSet`
+  for the counter is `std` (no dep).
+- **`aura-cli`** — `run_mc`, `runs_families`, `runs_family`, `mc_report` test
+  helper; `run_sweep` / `run_walkforward` switch to `append_family` and take a
+  `name`; the `window_of` migration of the manifest-window scans; `USAGE` + `match`
+  arms (incl. `--name`).
+
+## Data flow
+
+```
+author Rust family program / aura {sweep,mc,walkforward} [--name N]
+  -> engine: monte_carlo / sweep / walk_forward  (disjoint C1 runs)
+       per member: build sources -> window_of(&sources) -> h.run(sources)
+                   -> RunReport{ manifest{ commit, params, window(producer-supplied), seed }, metrics }
+  -> registry: <kind>_member_reports(&family) -> Vec<RunReport>  (canonical order)
+       -> Registry::append_family(name, kind, &reports):
+            counter = smallest k with "{name}-{k}" absent in the store
+            family_id = "{name}-{counter}"
+            -> Vec<FamilyRunRecord{ family_id, kind, ordinal=i, report }>  -> runs/families.jsonl (N lines)
+            -> returns family_id
+
+read back:
+  Registry::load_family_members -> Vec<FamilyRunRecord>
+  -> group_families            -> Vec<Family{ id, kind, members(ordinal-sorted) }>
+  CLI: runs_families (headers) | runs_family(id) (members) | runs_family(id, rank metric) (rank_by member reports)
+```
+
+## Error handling
+
+- `append_family` / `load_family_members` reuse `RegistryError` (`Io`, `Parse{line,…}`).
+  A missing `families.jsonl` is an empty family store (`Ok(vec![])`), exactly as
+  `load` treats a missing `runs.jsonl`.
+- **Counter read-before-write.** `append_family` loads the family store to pick
+  the next counter, so it is a read-then-append (not a blind append). Under
+  concurrent writers two families of the same name could race to the same counter;
+  the CLI runs one command per process, so this is a latent boundary, not an
+  active one — noted, not guarded this cycle (a future concern if the World gains
+  concurrent in-process family persistence).
+- `aura runs family <id>` for an unknown id: print nothing (an empty family) and
+  exit 0 — consistent with `runs list` over an empty registry (a non-existent
+  family is not an error, just empty). `aura runs family <id> rank <badmetric>`
+  surfaces `RegistryError::UnknownMetric` (stderr + exit 2), as `runs rank` does.
+- `window_of` returning `None` (all sources empty/open-ended) at a family call
+  site is a wiring/usage condition: the synthetic demos never hit it (their
+  `VecSource`s are non-empty); a defensive `expect` with a clear message guards it
+  (the same shape as today's `prices.first().expect("non-empty stream")`).
+
+## Testing strategy
+
+Engine (`aura-engine`):
+1. `VecSource::bounds` returns `Some((first_ts, last_ts))` of its buffer and
+   `None` for an empty buffer; cursor-independent (bounds unchanged after `next`).
+2. `window_of` folds the union extent across multiple sources (min from, max to)
+   and is `None` for all-empty.
+3. `SyntheticSpec::source(seed).bounds()` equals the analytic
+   `(Timestamp(1), Timestamp((len-1)*step+1))`.
+
+Ingest (`aura-ingest`):
+4. `M1FieldSource::bounds()` over an explicit `[from_ms, to_ms]` returns those
+   bounds normalized to epoch-ns (no bar streamed); data-gated like the existing
+   `streaming_seam` tests (skip when the local archive is absent).
+
+Registry (`aura-registry`) — the headline round-trips:
+5. **Lineage round-trips:** `append_family("f", kind, reports)` then
+   `group_families(load_family_members())` recovers one `Family` whose `members`
+   equal the stamped reports (ordinal-ordered), `kind` matches, and `id == "f-0"`
+   (the stored links re-derive the family — acceptance bullet 3).
+6. **Per-name counter:** two `append_family("x", …)` calls yield `"x-0"` then
+   `"x-1"`; an `append_family("y", …)` yields `"y-0"` (independent per-name
+   counters); `group_families` then returns three families.
+7. **Distinct families stay distinct:** an interleaved multi-family file regroups
+   into the right families, members ordinal-sorted, families in first-seen order.
+8. **Flat path untouched:** the existing `append`/`load`/`rank_by`/`optimize`
+   tests stay green; a `families.jsonl` write leaves `runs.jsonl` empty and vice
+   versa (the two stores are disjoint siblings).
+9. **Rank a family as a unit:** `rank_by(family.members → reports, "total_pips")`
+   orders the family best-first (reuses the existing `rank_by` proof).
+
+CLI (`aura-cli`):
+10. `aura mc` renders one member line per seed sharing one `family_id` plus the
+    aggregate line; the non-persisting `mc_report()` render is deterministic
+    across two calls (C1 of the family computation, mirroring
+    `sweep_report_is_deterministic`).
+11. `run_sweep`/`run_walkforward`/`run_mc` persist a family (against a fresh temp
+    store) that `group_families` recovers as one unit of the right `kind` and
+    member count, with id `"{name}-0"`.
+12. The `window_of` migration keeps `run_sample` byte-identical (`window == (1,7)`)
+    — the existing `run_sample_is_deterministic_and_non_trivial` stays green.
+
+## Acceptance criteria
+
+- **A family run writes related records:** `aura sweep` / `aura mc` /
+  `aura walkforward` persist `FamilyRunRecord` lines sharing one `family_id`
+  (`"{name}-{counter}"`) to the family store (test 11).
+- **The family can be listed and ranked as a unit:** `aura runs families` lists
+  headers; `aura runs family <id> [rank <metric>]` lists/ranks one family's
+  members (tests 9, 10, 11).
+- **Lineage round-trips:** the stored links re-derive the family
+  (`group_families` ∘ `load_family_members` recovers the written family — test 5);
+  a same-name re-run mints a fresh `"{name}-{counter}"` (test 6).
+- **Producer-supplied window:** every family-feeding run's manifest window comes
+  from `Source::bounds()` via `window_of`, not a `Vec` scan; lineage is byte-
+  identical whether the source is eager or streamed (tests 1–3, 12).
+- **Producer-agnostic re-derivation (C18 + #71):** no input-stream field (blob /
+  path / payload) is added to `RunManifest` or any lineage record; a member's
+  `(commit, params, window, seed)` manifest is its sole re-derivation recipe.
+- **No regression to the flat registry or C9 layering:** `runs.jsonl` and its API
+  are byte-for-byte unchanged (test 8); `aura-engine` gains no registry/lineage
+  dependency.
+
+## Non-goals
+
+- **Reproducible / content-addressed family identity.** The `family_id` is an
+  assigned `"{name}-{counter}"`, not a hash of the family's inputs; re-running the
+  same family mints a new id. A content-addressed identity (same inputs → same id)
+  with the replay-dedup it implies against an append-only store is a deliberate
+  later refinement, deferred to a follow-up issue.
+- **Unbounded real-data archive extent.** `M1FieldSource::bounds()` reports the
+  *requested* `[from_ms, to_ms]` window; an open-ended source (no window given)
+  returns `None`. Deriving a symbol's full archive extent without streaming (a
+  `DataServer` extent query) is deferred — `aura run --real <SYMBOL>` without
+  `--from/--to` is a *single* run, not a family, and keeps its current probe-based
+  window. No lineage member this cycle is open-ended (all are `VecSource`-backed
+  synthetic demos).
+- **Cross-family comparison / run-diff.** C18's "run-diff" depth and ranking
+  *families against each other* (vs. members within one family) are a later
+  refinement; this cycle delivers within-family listing/ranking + family headers.
+- **A typed param-space in the manifest.** `RunManifest.params` stays the
+  `Vec<(String, f64)>` precursor (unchanged).
+- **Random param-sweep enumeration (#52), sweep named-binding (#57).** Other
+  milestone items; #70 persists the families they (and the existing builders)
+  produce.