# Registry lineage for orchestration families — Design Spec

**Date:** 2026-06-15
**Status:** Draft — awaiting user spec review
**Authors:** orchestrator + Claude

## Goal

Make an orchestration **family** (sweep / Monte-Carlo / walk-forward) a
first-class, persisted unit in the run registry (C18 / C21). Today
`aura-registry` stores flat `RunReport` lines with no links between them: a
sweep's points and an MC's draws land as loose runs, indistinguishable from
standalone `aura run` records. This cycle adds a **lineage link** — a shared
*family id* (the parent reference) stamped on each member record — so a family
can be re-listed, ranked, and re-derived **as a unit**, and surfaces it through
the CLI (`aura mc`, family-aware `aura runs`).

Two invariants shape the design:

- **Producer-agnostic lineage (C18 + C6, #71 firewall).** Re-deriving a family
  member means **re-running** it from `(commit + params + window + seed)` — the
  fields its `RunManifest` already carries — never replaying a stored input
  artifact. No recorded blob / file path / byte payload enters a manifest or a
  lineage record. The member's manifest *is* its re-derivation recipe.
- **Producer-supplied window (the comment's named anti-pattern).** The
  `(Timestamp, Timestamp)` window stored in a lineage member must come from the
  **producer**, not from scanning a materialized `Vec` at the run call-site
  (today's `prices.first()/.last()`). "A lazy producer knows its bounds without
  materializing, so the stored lineage stays byte-identical whether the source
  is eager or streamed." This cycle adds a `Source::bounds()` accessor and routes
  every family-feeding run window through it.

## Architecture

Three layers move, none of them the strategy graph (brokers/sinks/strategies are
untouched):

1. **Engine — producer-supplied window (`aura-engine`).** The `Source` trait
   gains a read-only `bounds()` accessor reporting the inclusive data extent a
   producer will stream, known without materialization. A free `window_of`
   helper folds it across a run's sources (the union extent). The CLI's
   `prices.first()/.last()` manifest-window scans are replaced by `window_of`.

2. **Registry — the lineage record + family store (`aura-registry`).** A new
   `FamilyRunRecord` (a `RunReport` stamped with `family_id` + `kind` +
   `ordinal`) is the persisted lineage unit. A family is stored as its **related
   member records** in a sibling JSONL store (`families.jsonl`), leaving the flat
   `runs.jsonl` path and its `append`/`load`/`rank_by`/`optimize` API byte-for-
   byte unchanged. The `family_id` is **`"{name}-{counter}"`**: a user-supplied
   `name` (a builder-function argument, surfaced on the CLI as `--name` with a
   per-kind default) plus a per-name `counter` (a run-index) assigned at
   persistence. `group_families` re-derives the families by grouping on
   `family_id` (the round-trip).

3. **CLI — family surface (`aura-cli`).** `aura mc` is added (it runs + persists
   a Monte-Carlo family — `McFamily` exists in the engine but had no CLI). `aura
   sweep` / `aura walkforward` / `aura mc` persist to the family store via
   `append_family`, each taking an optional `--name`. `aura runs` becomes family-
   aware: `aura runs families` lists family headers, `aura runs family <id> [rank
   <metric>]` lists/ranks one family's members as a unit.

Layering (C9 preserved): `aura-engine` gains no registry/lineage concept — it
only learns to report a producer's bounds. Lineage lives in `aura-registry`
(which already depends on `aura-engine` and imports `SweepFamily`/`SweepPoint`);
it is "the World's memory" (C18/C22), not an engine concern.

### Why name + counter (the id form)

The `family_id` is used for exactly two things: the **grouping key** for the
round-trip (`group_families`) and the **user-facing lookup handle** for the CLI
(`aura runs family <id>`). Neither needs the id to be *reproducible* from the
family's inputs — grouping needs only shared-within / distinct-across, and the
user reads the id from `aura runs families` and pastes it, never computes it.

A **user-supplied name** makes the registry human-navigable (`aura runs family
macd-robustness-0`), matching the "World = named experiments" framing (C21). A
**counter** suffix guarantees uniqueness without a new dependency and without
nondeterministic entropy (a UUID would introduce the system's first unseeded
randomness, against the engine's seeded-determinism discipline — it hand-rolls
`SplitMix64` precisely to avoid that). Re-running a family mints a *new* id
(`name-1` after `name-0`) — a fresh family record, which is the honest meaning of
"I ran the experiment again". A content-addressed (reproducible-hash) identity
plus replay-dedup is a deliberate later refinement, not this cycle (Non-goals).

## Concrete code shapes

### User-facing program (the acceptance evidence)

The researcher drives families from the CLI and gets back records that round-trip
as a unit. New / changed invocations:

```console
$ aura mc                       # NEW: run + persist a Monte-Carlo family (default name "mc")
{"family_id":"mc-0","seed":1,"report":{"manifest":{...,"seed":1,...},"metrics":{...}}}
{"family_id":"mc-0","seed":2,"report":{...}}
{"family_id":"mc-0","seed":3,"report":{...}}
{"mc_aggregate":{"total_pips":{"mean":...,"p50":...},"max_drawdown":{...},"exposure_sign_flips":{...}}}

$ aura mc --name macd-robustness   # NEW: author-supplied name; counter disambiguates re-runs
{"family_id":"macd-robustness-0","seed":1,"report":{...}}
...

$ aura runs families            # NEW: one header line per stored family
{"family_id":"mc-0","kind":"MonteCarlo","members":3}
{"family_id":"sweep-0","kind":"Sweep","members":4}
{"family_id":"walkforward-0","kind":"WalkForward","members":3}

$ aura runs family mc-0 rank total_pips     # NEW: rank one family as a unit
{"family_id":"mc-0","seed":2,"report":{...,"metrics":{"total_pips":4.1,...}}}
{"family_id":"mc-0","seed":1,"report":{...,"metrics":{"total_pips":2.7,...}}}
{"family_id":"mc-0","seed":3,"report":{...,"metrics":{"total_pips":-0.4,...}}}

$ aura runs list                # UNCHANGED: standalone (family-less) runs only
{"manifest":{...},"metrics":{...}}
```

The Rust authoring shape (the World is authored in Rust, C10) — the lineage
round-trip a harness-family program writes:

```rust
use aura_registry::{group_families, mc_member_reports, FamilyKind, Registry};

let family = aura_engine::monte_carlo(&base_point, &[1, 2, 3], run_draw);
let reg = Registry::open("runs/runs.jsonl");
let id = reg.append_family("macd-robustness", FamilyKind::MonteCarlo,
                           &mc_member_reports(&family))?;  // -> "macd-robustness-0"

// round-trip: the stored links re-derive the family as a unit
let families = group_families(reg.load_family_members()?);
assert_eq!(families.len(), 1);
assert_eq!(families[0].id, "macd-robustness-0");
assert_eq!(families[0].kind, FamilyKind::MonteCarlo);
assert_eq!(families[0].members.len(), 3);                  // re-listed as a unit, ordinal-ordered

// a second run of the same name mints a fresh family id (a new experiment run)
let id2 = reg.append_family("macd-robustness", FamilyKind::MonteCarlo,
                            &mc_member_reports(&family))?; // -> "macd-robustness-1"
assert_ne!(id, id2);
```

### Implementation shapes (secondary — before → after)

**1. `Source::bounds()` + `window_of` (`aura-engine`, harness.rs).**

```rust
// BEFORE — the producer knows only its head; the window is scanned at the call site
pub trait Source {
    fn peek(&self) -> Option<Timestamp>;
    fn next(&mut self) -> Option<(Timestamp, Scalar)>;
}
// in aura-cli: let window = (prices.first()?.0, prices.last()?.0);  // <- the named anti-pattern

// AFTER — the producer reports its data extent without materialization
pub trait Source {
    fn peek(&self) -> Option<Timestamp>;
    fn next(&mut self) -> Option<(Timestamp, Scalar)>;
    /// The inclusive `(from, to)` data extent this producer will stream, known
    /// without materializing it (#71/C18): a lazy producer reports its window
    /// bounds; an eager one reports its buffer's first/last. `None` = empty /
    /// open-ended. Cursor-independent: a property of the data extent, not the
    /// read position.
    fn bounds(&self) -> Option<(Timestamp, Timestamp)>;
}

/// The window of a run: the union extent across its sources (min from, max to).
/// `None` when every source is empty/open-ended.
pub fn window_of(sources: &[Box<dyn Source>]) -> Option<(Timestamp, Timestamp)>;
```

`VecSource::bounds` → `Some((stream.first()?.0, stream.last()?.0))` (the eager
source reads its own buffer — O(1) on the slice ends; the *consumer* no longer
reaches into a `Vec`). `SyntheticSpec::source` returns a `VecSource`, so it is
covered automatically (its generated stream's first/last). `M1FieldSource::bounds`
→ the requested `(from_ms, to_ms)` normalized to epoch-ns `Timestamp` when both
are given, else `None` (open-ended; see Non-goals).

CLI migration: `run_sample`, `sweep_family`, `sweep_over`, `run_oos`, and the new
`aura mc` build their `Vec<Box<dyn Source>>` first, read `window_of(&sources)`,
then `h.run(sources)` — no `prices.first()/.last()`. Byte-identical for the
synthetic demos (`VecSource::bounds` == the old first/last).

**2. Lineage record + family store (`aura-registry`).**

```rust
// NEW types
/// Which C12 orchestration axis produced a family.
#[derive(Clone, Copy, Debug, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
pub enum FamilyKind { Sweep, MonteCarlo, WalkForward }

/// One persisted family member: a run record stamped with its lineage link.
/// `family_id` is the shared parent reference; `ordinal` is the member's
/// position in the family's canonical order (odometer / seed-input / roll).
#[derive(Clone, Debug, PartialEq, serde::Serialize, serde::Deserialize)]
pub struct FamilyRunRecord {
    pub family_id: String,
    pub kind: FamilyKind,
    pub ordinal: usize,
    pub report: RunReport,
}

/// A re-derived family: the members sharing one `family_id`, ordinal-sorted —
/// the round-trip result of [`group_families`].
#[derive(Clone, Debug, PartialEq)]
pub struct Family {
    pub id: String,
    pub kind: FamilyKind,
    pub members: Vec<FamilyRunRecord>,
}

impl Registry {
    /// Assign a fresh `family_id = "{name}-{counter}"` — `counter` is the smallest
    /// `k` for which `"{name}-{k}"` is not already in the family store (a per-name
    /// run-index) — stamp `reports` as ordinal-ordered members, append them as N
    /// JSONL lines to the family store (a sibling of the flat runs store,
    /// `self.path.with_file_name("families.jsonl")`, so the flat path is
    /// untouched), and return the assigned id. Reads the store once to pick the
    /// counter (a read-before-write; single-process CLI invocations do not race —
    /// see Error handling).
    pub fn append_family(&self, name: &str, kind: FamilyKind, reports: &[RunReport])
        -> Result<String, RegistryError>;
    /// Parse every stored family member, in file order. A missing file is empty.
    pub fn load_family_members(&self) -> Result<Vec<FamilyRunRecord>, RegistryError>;
}

/// Group member records into families by `family_id` (first-seen family order;
/// each family's members ordinal-sorted). The round-trip: `append_family(...)`
/// then `group_families(load_family_members())` recovers the families — the
/// stored links re-derive the family.
pub fn group_families(records: Vec<FamilyRunRecord>) -> Vec<Family>;

/// The member reports of an engine family, in canonical order — the trivial
/// extractors the CLI hands to `append_family` (they locate the per-kind member
/// report: `SweepPoint.report` / `McDraw.report` / `WindowRun.oos_report`).
pub fn sweep_member_reports(family: &SweepFamily) -> Vec<RunReport>;
pub fn mc_member_reports(family: &McFamily) -> Vec<RunReport>;
pub fn walkforward_member_reports(result: &WalkForwardResult) -> Vec<RunReport>;
```

`next_family_id` (module-private) generates `"{name}-0"`, `"{name}-1"`, … and
returns the first not present in the loaded id set — robust to `-` inside `name`
(it tests membership of generated candidates, never parses the string). The flat
`Registry::append(&RunReport)` / `load() -> Vec<RunReport>` and `rank_by` /
`optimize` are unchanged (no migration of `runs.jsonl`).

**3. CLI subcommands (`aura-cli`, main.rs).**

```rust
// BEFORE
const USAGE: &str = "usage: aura run [...] | ... | aura sweep | aura walkforward | aura runs list | aura runs rank <metric>";
match args {
    ["sweep"] => run_sweep(),                 // appends flat reports to runs.jsonl
    ["walkforward"] => run_walkforward(),     // appends flat reports to runs.jsonl
    ["runs", "list"] => runs_list(),
    ["runs", "rank", metric] => runs_rank(metric),
    ...
}

// AFTER
const USAGE: &str = "usage: aura run [...] | ... | aura sweep [--name <n>] | aura mc [--name <n>] | aura walkforward [--name <n>] | aura runs list | aura runs rank <metric> | aura runs families | aura runs family <id> [rank <metric>]";
match args {
    ["sweep"] => run_sweep("sweep"),
    ["sweep", "--name", n] => run_sweep(n),
    ["mc"] => run_mc("mc"),                    // NEW: monte_carlo over a built-in seed set + append_family
    ["mc", "--name", n] => run_mc(n),
    ["walkforward"] => run_walkforward("walkforward"),
    ["walkforward", "--name", n] => run_walkforward(n),
    ["runs", "list"] => runs_list(),          // unchanged — standalone runs
    ["runs", "rank", metric] => runs_rank(metric),
    ["runs", "families"] => runs_families(),                       // NEW
    ["runs", "family", id] => runs_family(id, None),              // NEW
    ["runs", "family", id, "rank", metric] => runs_family(id, Some(metric)), // NEW
    ...
}
```

`aura sweep`/`walkforward`/`mc` move from flat-appending to family-appending via
`append_family(name, kind, &<kind>_member_reports(&family))`, printing the
returned `family_id` on each member line (the pre-lineage flat-storage stopgap is
replaced by proper family storage — the point of the cycle). `runs_family`'s
`rank` reuses the existing `rank_by` over the family's member reports (within-
family ranking). A non-persisting render helper (`mc_report`, mirroring the
existing `sweep_report`/`walkforward_report` `#[cfg(test)]` helpers) carries the
C1-determinism test of the family *computation*, separate from the store-
dependent id.

## Components

- **`aura-engine::harness`** — `Source::bounds()` (new trait method, object-safe),
  `VecSource`/`SyntheticSpec` covered, `window_of` free function. No new dep
  (engine already has serde).
- **`aura-ingest::M1FieldSource`** — store the requested `(from_ms, to_ms)` at
  `open`; impl `bounds()` (requested window → epoch-ns when bounded, else `None`).
- **`aura-registry`** — new `lineage` module: `FamilyKind`, `FamilyRunRecord`,
  `Family`, `group_families`, `next_family_id`, the three `*_member_reports`
  extractors; `Registry::append_family` / `load_family_members`. New dep:
  `serde = { workspace = true, features = ["derive"] }` (the crate has
  `serde_json` but not `serde` derive; the lineage types need it — admitted under
  the per-case dependency policy, same basis as `serde_json` already is). `HashSet`
  for the counter is `std` (no dep).
- **`aura-cli`** — `run_mc`, `runs_families`, `runs_family`, `mc_report` test
  helper; `run_sweep` / `run_walkforward` switch to `append_family` and take a
  `name`; the `window_of` migration of the manifest-window scans; `USAGE` + `match`
  arms (incl. `--name`).

## Data flow

```
author Rust family program / aura {sweep,mc,walkforward} [--name N]
  -> engine: monte_carlo / sweep / walk_forward  (disjoint C1 runs)
       per member: build sources -> window_of(&sources) -> h.run(sources)
                   -> RunReport{ manifest{ commit, params, window(producer-supplied), seed }, metrics }
  -> registry: <kind>_member_reports(&family) -> Vec<RunReport>  (canonical order)
       -> Registry::append_family(name, kind, &reports):
            counter = smallest k with "{name}-{k}" absent in the store
            family_id = "{name}-{counter}"
            -> Vec<FamilyRunRecord{ family_id, kind, ordinal=i, report }>  -> runs/families.jsonl (N lines)
            -> returns family_id

read back:
  Registry::load_family_members -> Vec<FamilyRunRecord>
  -> group_families            -> Vec<Family{ id, kind, members(ordinal-sorted) }>
  CLI: runs_families (headers) | runs_family(id) (members) | runs_family(id, rank metric) (rank_by member reports)
```

## Error handling

- `append_family` / `load_family_members` reuse `RegistryError` (`Io`, `Parse{line,…}`).
  A missing `families.jsonl` is an empty family store (`Ok(vec![])`), exactly as
  `load` treats a missing `runs.jsonl`.
- **Counter read-before-write.** `append_family` loads the family store to pick
  the next counter, so it is a read-then-append (not a blind append). Under
  concurrent writers two families of the same name could race to the same counter;
  the CLI runs one command per process, so this is a latent boundary, not an
  active one — noted, not guarded this cycle (a future concern if the World gains
  concurrent in-process family persistence).
- `aura runs family <id>` for an unknown id: print nothing (an empty family) and
  exit 0 — consistent with `runs list` over an empty registry (a non-existent
  family is not an error, just empty). `aura runs family <id> rank <badmetric>`
  surfaces `RegistryError::UnknownMetric` (stderr + exit 2), as `runs rank` does.
- `window_of` returning `None` (all sources empty/open-ended) at a family call
  site is a wiring/usage condition: the synthetic demos never hit it (their
  `VecSource`s are non-empty); a defensive `expect` with a clear message guards it
  (the same shape as today's `prices.first().expect("non-empty stream")`).

## Testing strategy

Engine (`aura-engine`):
1. `VecSource::bounds` returns `Some((first_ts, last_ts))` of its buffer and
   `None` for an empty buffer; cursor-independent (bounds unchanged after `next`).
2. `window_of` folds the union extent across multiple sources (min from, max to)
   and is `None` for all-empty.
3. `SyntheticSpec::source(seed).bounds()` equals the analytic
   `(Timestamp(1), Timestamp((len-1)*step+1))`.

Ingest (`aura-ingest`):
4. `M1FieldSource::bounds()` over an explicit `[from_ms, to_ms]` returns those
   bounds normalized to epoch-ns (no bar streamed); data-gated like the existing
   `streaming_seam` tests (skip when the local archive is absent).

Registry (`aura-registry`) — the headline round-trips:
5. **Lineage round-trips:** `append_family("f", kind, reports)` then
   `group_families(load_family_members())` recovers one `Family` whose `members`
   equal the stamped reports (ordinal-ordered), `kind` matches, and `id == "f-0"`
   (the stored links re-derive the family — acceptance bullet 3).
6. **Per-name counter:** two `append_family("x", …)` calls yield `"x-0"` then
   `"x-1"`; an `append_family("y", …)` yields `"y-0"` (independent per-name
   counters); `group_families` then returns three families.
7. **Distinct families stay distinct:** an interleaved multi-family file regroups
   into the right families, members ordinal-sorted, families in first-seen order.
8. **Flat path untouched:** the existing `append`/`load`/`rank_by`/`optimize`
   tests stay green; a `families.jsonl` write leaves `runs.jsonl` empty and vice
   versa (the two stores are disjoint siblings).
9. **Rank a family as a unit:** `rank_by(family.members → reports, "total_pips")`
   orders the family best-first (reuses the existing `rank_by` proof).

CLI (`aura-cli`):
10. `aura mc` renders one member line per seed sharing one `family_id` plus the
    aggregate line; the non-persisting `mc_report()` render is deterministic
    across two calls (C1 of the family computation, mirroring
    `sweep_report_is_deterministic`).
11. `run_sweep`/`run_walkforward`/`run_mc` persist a family (against a fresh temp
    store) that `group_families` recovers as one unit of the right `kind` and
    member count, with id `"{name}-0"`.
12. The `window_of` migration keeps `run_sample` byte-identical (`window == (1,7)`)
    — the existing `run_sample_is_deterministic_and_non_trivial` stays green.

## Acceptance criteria

- **A family run writes related records:** `aura sweep` / `aura mc` /
  `aura walkforward` persist `FamilyRunRecord` lines sharing one `family_id`
  (`"{name}-{counter}"`) to the family store (test 11).
- **The family can be listed and ranked as a unit:** `aura runs families` lists
  headers; `aura runs family <id> [rank <metric>]` lists/ranks one family's
  members (tests 9, 10, 11).
- **Lineage round-trips:** the stored links re-derive the family
  (`group_families` ∘ `load_family_members` recovers the written family — test 5);
  a same-name re-run mints a fresh `"{name}-{counter}"` (test 6).
- **Producer-supplied window:** every family-feeding run's manifest window comes
  from `Source::bounds()` via `window_of`, not a `Vec` scan; lineage is byte-
  identical whether the source is eager or streamed (tests 1–3, 12).
- **Producer-agnostic re-derivation (C18 + #71):** no input-stream field (blob /
  path / payload) is added to `RunManifest` or any lineage record; a member's
  `(commit, params, window, seed)` manifest is its sole re-derivation recipe.
- **No regression to the flat registry or C9 layering:** `runs.jsonl` and its API
  are byte-for-byte unchanged (test 8); `aura-engine` gains no registry/lineage
  dependency.

## Non-goals

- **Reproducible / content-addressed family identity.** The `family_id` is an
  assigned `"{name}-{counter}"`, not a hash of the family's inputs; re-running the
  same family mints a new id. A content-addressed identity (same inputs → same id)
  with the replay-dedup it implies against an append-only store is a deliberate
  later refinement, deferred to a follow-up issue.
- **Unbounded real-data archive extent.** `M1FieldSource::bounds()` reports the
  *requested* `[from_ms, to_ms]` window; an open-ended source (no window given)
  returns `None`. Deriving a symbol's full archive extent without streaming (a
  `DataServer` extent query) is deferred — `aura run --real <SYMBOL>` without
  `--from/--to` is a *single* run, not a family, and keeps its current probe-based
  window. No lineage member this cycle is open-ended (all are `VecSource`-backed
  synthetic demos).
- **Cross-family comparison / run-diff.** C18's "run-diff" depth and ranking
  *families against each other* (vs. members within one family) are a later
  refinement; this cycle delivers within-family listing/ranking + family headers.
- **A typed param-space in the manifest.** `RunManifest.params` stays the
  `Vec<(String, f64)>` precursor (unchanged).
- **Random param-sweep enumeration (#52), sweep named-binding (#57).** Other
  milestone items; #70 persists the families they (and the existing builders)
  produce.
