From 4b6bc5279479322efcd2be6932ff01903a8f3301 Mon Sep 17 00:00:00 2001 From: Brummel Date: Wed, 1 Jul 2026 01:15:56 +0200 Subject: [PATCH] spec: 0094 content-addressed reproduction (boss-signed) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Restore C18 reproducibility for generated (blueprint-sweep) runs. Since cycle 0093 a sweep member's topology is a generated data value, so commit alone no longer identifies the graph and the manifest is not a complete re-derivation recipe (#158). Design: content-address the topology. The canonical blueprint JSON is stored once, keyed by the topology_hash the manifest already carries (cycle 0092), under runs/blueprints/.json (a dumb bytes-by-key store in aura-registry, no sha2 dep — the reproduce path's bit-identical compare is the integrity check). `aura reproduce ` loads each member's blueprint by hash, re-bootstraps with the member's params/window/seed, re-runs, and asserts the metrics are bit-identical (C1). acc-1 (two paths -> same content id) via a minimal `graph introspect --content-id`; acc-3 (Tier-1-stable id) composes the #156 omit-defaults pin. Derived forks (logged on #158): content-address (not carry-inline) — keeps the manifest tiny (C18) and dedups one topology across a family (C11/C12); hash domain = signal-only this cycle (the scaffolding is not yet blueprint-data, C24), boundary documented, whole-harness id deferred (#156/#159). Engine untouched (C9); store + hash off the frozen engine (invariant 8). boss-signed on a grounding-check PASS (9/9 current-behaviour assumptions ratified by green tests). refs #158 --- .../0094-content-addressed-reproduction.md | 285 ++++++++++++++++++ 1 file changed, 285 insertions(+) create mode 100644 docs/specs/0094-content-addressed-reproduction.md diff --git a/docs/specs/0094-content-addressed-reproduction.md b/docs/specs/0094-content-addressed-reproduction.md new file mode 100644 index 0000000..d84161c --- /dev/null +++ b/docs/specs/0094-content-addressed-reproduction.md @@ -0,0 +1,285 @@ +# Content-addressed reproduction of a generated run — Design Spec + +**Date:** 2026-07-01 +**Status:** Draft — awaiting user spec review +**Authors:** orchestrator + Claude + +## Goal + +Restore the C18 reproducibility guarantee for **generated** runs. Since cycle 0093 a +sweep member's topology is a *generated data value* (a `Composite` loaded from a blueprint +file, gridded over named axes), so `commit` alone no longer identifies the graph and the +member's manifest is no longer a complete re-derivation recipe — a C18 violation (#158). + +This cycle closes that gap by **content-addressing the topology**: the canonical blueprint +JSON is stored once, keyed by its `topology_hash` (already on the manifest since cycle +0092), and a reproduce path re-derives a persisted family's runs from the store and proves +each is bit-identical to the stored result (C1). The three #158 acceptance properties are +realized as tests; the user-facing capability is `aura reproduce `. + +The hash domain is **signal-only** this cycle — the fixed Stage-1-R scaffolding +(`wrap_stage1r`: broker / sinks / RiskExecutor / cost-graph) is not yet blueprint-data +(deliberately out of the round-trippable set, C24) so the content-id covers the signal and +the scaffolding stays commit-identified. The boundary is documented; whole-harness +content-addressing is deferred (#156 / #159 / project-as-crate). + +## Architecture + +Three layers, each at its correct home: + +1. **Content-addressed blueprint store (aura-registry).** The registry owns the on-disk + `runs/` layout (`runs.jsonl`, `families.jsonl` sibling, `runs/traces//`). It gains + a sibling content-addressed store `runs/blueprints/.json` and two dumb + bytes-by-key methods: `put_blueprint(hash, canonical_json)` (write-once: identical + content re-writes the identical bytes, a no-op in effect) and + `get_blueprint(hash) -> Option`. The registry does **not** parse, hash, or + validate the bytes — it stores and retrieves a String keyed by a String. No `sha2` + dependency enters the registry: the caller (the CLI) computes the key, and the + reproduce path's bit-identical metric comparison is itself the integrity check (a + corrupted blueprint re-derives different metrics → reported as a divergence). + +2. **Hash + canonical bytes + reproduce (aura-cli).** The CLI already owns `topology_hash` + (`sha2` over `blueprint_to_json`, off the frozen engine, invariant 8). On a blueprint + sweep it writes the canonical blueprint once (the bytes whose SHA256 *is* the shared + `topology_hash`); the reproduce path reads a stored family, loads each member's + blueprint by hash, reconstructs the member's sweep point from its recorded params, + re-bootstraps through the cycle-1 `wrap_stage1r` seam, re-runs, and compares. + +3. **Cross-path content-id (aura graph introspect).** A minimal read-only addition + `aura graph introspect --content-id` prints the SHA256 of the canonical blueprint an + op-script (#157) builds — making "do these two authoring paths describe the same + topology?" answerable from the CLI, and giving acceptance (1) its surface. + +C9 holds throughout: the engine is untouched (`blueprint_to_json` / `blueprint_from_json` +already exist in `aura-engine::blueprint_serde`); the new store and reproduce logic live in +the research-side crates that already depend on the engine. + +## Concrete code shapes + +### Worked user-facing capability — `aura reproduce ` + +The research user persisted a sweep family and wants to verify it still reproduces from its +manifest (the C18 "re-derives full results on demand" promise): + +```console +$ aura sweep stage1_signal_open.json --axis stage1_signal.slow.length=4,6,8,10 --name smacross +smacross-1 stage1_signal.slow.length=4 expectancy_r=1.2710 n_trades=3 +smacross-1 stage1_signal.slow.length=6 expectancy_r=... n_trades=... +... +# topology stored once at runs/blueprints/.json; 4 member manifests share that hash. + +$ aura reproduce smacross-1 +smacross-1 member stage1_signal.slow.length=4 reproduced: bit-identical +smacross-1 member stage1_signal.slow.length=6 reproduced: bit-identical +smacross-1 member stage1_signal.slow.length=8 reproduced: bit-identical +smacross-1 member stage1_signal.slow.length=10 reproduced: bit-identical +reproduced 4/4 members bit-identically from runs/blueprints/.json +$ echo $? +0 +# a divergence (or a missing stored blueprint) prints the member + exits non-zero. +``` + +This is the empirical evidence for the feature-acceptance criterion: the capability a +research user reaches for, built on the manifest + the content-addressed store + the commit +as the complete re-derivation recipe. + +### Store write at the sweep persist seam (aura-cli `run_blueprint_sweep`, ~main.rs:3063) + +Before (today — the family is persisted, the topology is not stored): + +```rust +fn run_blueprint_sweep(doc: &str, axes: &[(String, Vec)], name: &str, persist: bool, data: DataSource) { + let _ = persist; + let family = blueprint_sweep_family(doc, axes, &data).unwrap_or_else(|e| { eprintln!("aura: {e:?}"); std::process::exit(2); }); + let id = match default_registry().append_family(name, FamilyKind::Sweep, &sweep_member_reports(&family)) { /* … */ }; + for pt in &family.points { println!("{}", family_member_line(&id, &pt.report)); } +} +``` + +After (the shared topology is content-addressed once, beside the family record): + +```rust +fn run_blueprint_sweep(doc: &str, axes: &[(String, Vec)], name: &str, persist: bool, data: DataSource) { + let _ = persist; + let family = blueprint_sweep_family(doc, axes, &data).unwrap_or_else(|e| { eprintln!("aura: {e:?}"); std::process::exit(2); }); + let reg = default_registry(); + // store the canonical blueprint once, keyed by the family's shared topology_hash — + // exactly the bytes whose SHA256 the members already carry (#164 byte-canonical form). + let topo = family.points[0].report.manifest.topology_hash.clone() + .expect("a blueprint sweep stamps every member's topology_hash"); + let canonical = blueprint_to_json(&blueprint_from_json(doc, &|t| std_vocabulary(t)) + .expect("doc parse-validated at the dispatch boundary")) + .expect("a loaded blueprint re-serializes"); + reg.put_blueprint(&topo, &canonical).unwrap_or_else(|e| { eprintln!("aura: {e}"); std::process::exit(2); }); + let id = match reg.append_family(name, FamilyKind::Sweep, &sweep_member_reports(&family)) { /* … unchanged … */ }; + for pt in &family.points { println!("{}", family_member_line(&id, &pt.report)); } +} +``` + +### Content-addressed store (aura-registry, new methods on `Registry`) + +```rust +impl Registry { + /// The content-addressed blueprint store dir, a sibling of the runs store: + /// `.with_file_name("blueprints")`. + fn blueprints_dir(&self) -> PathBuf { self.path.with_file_name("blueprints") } + + /// Write-once content-addressed put: `runs/blueprints/.json` = canonical_json. + /// Idempotent — the same hash always addresses identical bytes, so an overwrite is a + /// no-op in effect. The registry does not verify `sha256(bytes) == hash`; the caller + /// owns the hash, and reproduction's bit-identical compare is the integrity check. + pub fn put_blueprint(&self, hash: &str, canonical_json: &str) -> Result<(), RegistryError> { + let dir = self.blueprints_dir(); + std::fs::create_dir_all(&dir).map_err(/* RegistryError::Io */)?; + std::fs::write(dir.join(format!("{hash}.json")), canonical_json).map_err(/* … */)?; + Ok(()) + } + + /// Read a stored blueprint by content id; `Ok(None)` if absent (treat-as-empty + /// discipline, like `load` on a missing store). + pub fn get_blueprint(&self, hash: &str) -> Result, RegistryError> { + match std::fs::read_to_string(self.blueprints_dir().join(format!("{hash}.json"))) { + Ok(s) => Ok(Some(s)), + Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(None), + Err(e) => Err(/* RegistryError::Io */), + } + } +} +``` + +### Reproduce path (aura-cli, new) + +```rust +/// Re-derive every member of a persisted sweep family from the content-addressed store and +/// compare to the stored result. Determinism (C1) makes a match bit-identical. +fn reproduce_family(id: &str, data: &DataSource) -> ReproduceReport { + let reg = default_registry(); + let family = group_families(reg.load_family_members().expect("load")) + .into_iter().find(|f| f.id == id) + .unwrap_or_else(|| { eprintln!("aura: no such family '{id}'"); std::process::exit(2); }); + + let mut outcomes = Vec::new(); + for member in &family.members { + let hash = member.manifest.topology_hash.as_deref() + .expect("a generated member carries a topology_hash"); + let doc = reg.get_blueprint(hash).expect("store read") + .unwrap_or_else(|| { eprintln!("aura: blueprint {hash} missing from store"); std::process::exit(2); }); + // reconstruct the sweep point: param_space order of the reloaded signal, values + // read from the member's recorded named params (the inverse of zip_params). + let signal = blueprint_from_json(&doc, &|t| std_vocabulary(t)).expect("stored blueprint parses"); + let point = point_from_params(&signal, &member.manifest.params); // Vec in param_space order + let rerun = run_signal_point(signal, &point, data, member.manifest.seed); // wrap_stage1r + bootstrap + run, reduce-mode + outcomes.push((member_label(member), rerun.metrics == member.metrics)); + } + ReproduceReport { id: id.to_string(), outcomes } +} +``` + +`point_from_params` walks the reloaded signal's `param_space()` and, for each knob, reads +the value the member's manifest recorded under that knob's name, yielding the `Vec` +`bootstrap_with_cells` consumes — the inverse of the `zip_params(&space, point)` the sweep +used to record them. + +### Cross-path content-id surface (aura graph introspect, ~main.rs introspect arm) + +```rust +// `aura graph introspect --content-id` over an op-script on stdin: build the blueprint +// (the #157 replay), print the SHA256 of its canonical form — the same hash `aura run` +// stamps as topology_hash. Reuses topology_hash(&composite); no new hashing. +["graph", "introspect", "--content-id"] => { + let blueprint = build_from_op_script(read_stdin())?; // existing #157 replay → Composite + println!("{}", topology_hash(&blueprint)); +} +``` + +## Components + +- **`Registry::put_blueprint` / `get_blueprint` / `blueprints_dir`** (aura-registry) — the + content-addressed bytes store under `runs/blueprints/`. Dumb, no engine-blueprint parse, + no `sha2`. +- **Store write in `run_blueprint_sweep`** (aura-cli) — one canonical-blueprint write per + sweep, keyed by the family's shared `topology_hash`. +- **`reproduce_family` + `point_from_params` + the `aura reproduce ` dispatch arm** + (aura-cli) — load family → load blueprint by hash → reconstruct point → re-run → compare. +- **`graph introspect --content-id`** (aura-cli) — the op-script content-id surface for + acceptance (1). + +## Data flow + +Persist: `aura sweep ` → `blueprint_sweep_family` (members stamped with the shared +`topology_hash`) → `run_blueprint_sweep` writes `runs/blueprints/.json` once → +`append_family` writes the member records to `families.jsonl`. + +Reproduce: `aura reproduce ` → `group_families` finds the family → per member: read +`topology_hash` → `get_blueprint(hash)` → `blueprint_from_json` → `point_from_params` → +`wrap_stage1r` + `bootstrap_with_cells(point)` + `run` (reduce-mode, exactly the sweep +member path) → `summarize_r` → compare re-run `metrics` to the stored member `metrics`. + +The manifest carries `params`, `window`, `seed`, `topology_hash`; the data source is +synthetic (deterministic) for this cycle — recorded-dataset reproduction rides the +DataServer seam (#124), out of scope. + +## Error handling + +- **Missing stored blueprint** (`get_blueprint` → `None`): the family was recorded before + the store existed, or the store dir was deleted — print `blueprint missing from + store` and exit non-zero (refuse-don't-guess, C10/C18). Never re-run against a guessed + topology. +- **Unknown family id**: `aura reproduce ` with no matching family prints `no such + family ''` and exits non-zero (a deliberate non-zero, distinct from the treat-as-empty + `runs family ` *lookup* — reproduce is an action that needs a real target). +- **Divergence** (`rerun.metrics != stored`): report the offending member and exit + non-zero. This is the integrity check; on synthetic data a divergence is a real bug, not + an expected outcome. +- **Store write failure** (I/O): propagate as `RegistryError`, surfaced as `aura: ` + + exit non-zero at the persist seam. + +## Testing strategy + +- **acc 2 (the heart) — reproduce is bit-identical, including open-at-end.** A test + persists a blueprint sweep whose grid includes the **open-at-end** param (fast=2/slow=4 is + open-at-end, `n_open_at_end=1`), then `reproduce_family` re-derives every member and + asserts `rerun.metrics == stored.metrics` for each. This exercises the open-at-end path + end-to-end through the real reproduce loop — store round-trip + C1 determinism: a stored + member re-derives bit-identically (a reduce-vs-reduce re-run, the same path the sweep + recorded). The reduce(`GatedRecorder`) vs full(`Recorder`) R-equivalence the cycle-0093 + implementer flagged is a *separate* property, already pinned by the existing keystone + `blueprint_sweep_member_equals_single_run_and_shares_topology_hash` (a reduce member == + a full single run at fast=2/slow=4, an open-at-end param) plus two unit tests — see #166; + this cycle adds no new claim there, it reuses the open-at-end param for thorough reproduce + coverage. +- **acc 2 — round-trip store.** A unit test: `put_blueprint(h, bytes)` then + `get_blueprint(h)` returns `Some(bytes)` byte-identically; `get_blueprint(absent)` is + `Ok(None)`. +- **acc 1 — two command paths, same content id.** A test builds the SMA-cross signal via + the op-script path (`graph introspect --content-id`, or hashing `graph build` output) and + via `stage1_signal`/`topology_hash`, and asserts the two content ids are equal for the + same topology and differ for a different one (e.g. fast=2 vs fast=3). +- **acc 3 — Tier-1 stability of the content id.** A test pins that `topology_hash` is + unchanged when a Tier-1 optional field the blueprint does not use is added to the format + — composing with `unknown_optional_field_is_tolerated_byte_identically` (#156): the + canonical omit-defaults form omits an absent optional, so the hashed bytes (and the id) + are byte-identical. +- **E2E (CLI):** `aura sweep --name X` then `aura reproduce X` over the built binary — + asserts the `reproduced N/N members bit-identically` line and exit 0; a missing-store and + an unknown-id path each exit non-zero with the named cause on stderr. + +## Acceptance criteria + +1. **Two command paths that build the same topology produce the same content id.** The + op-script path and `aura run`'s `stage1_signal` agree on `topology_hash` for the same + signal, and disagree for a different one. (`graph introspect --content-id` exposes it.) +2. **A serialized manifest re-derives the same FlatGraph and a bit-identical run (C1).** + `aura reproduce ` loads each member's blueprint from the content-addressed + store by its manifest's `topology_hash`, re-bootstraps with the member's params / window + / seed, re-runs, and the re-run metrics are bit-identical to the stored metrics — + including the open-at-end param. +3. **A Tier-1 format addition the blueprint does not use leaves its content id unchanged.** + An absent optional does not enter the canonical form, so `topology_hash` is byte-stable + across the addition. +4. **Boundary documented.** The content-id covers the **signal** only; the Stage-1-R + scaffolding stays commit-identified this cycle (recorded in the ledger), with + whole-harness content-addressing deferred. +5. **Invariants preserved.** Engine untouched (C9); the store + hash are off the frozen + engine (invariant 8); reproduction is deterministic on synthetic data (C1); + refuse-don't-guess on a missing stored blueprint (C10/C18).