aura reproduce re-derives real-data families over synthetic data (0/N DIVERGED; WalkForward panic) #229

New Issue

2026-07-10T14:47:03+02:00

Brummel commented

2026-07-10 14:47:03 +02:00

Surfaced by the triage-harvest milestone fieldtest (2026-07-10, evidence in fieldtests/milestone-triage-harvest/); root-caused the same day.

Symptom

A Sweep family minted with --real GER40 reports reproduced 0/N members bit-identically (every member DIVERGED, exit 1) under aura reproduce <id>.
A WalkForward family panics instead: main.rs:1123: non-empty OOS window (exit 101).
A FRESH identical sweep invocation matches the persisted metrics bit-for-bit, so the runs themselves are deterministic (C1 holds); the defect is reproduce-side.

Cause

reproduce_family (crates/aura-cli/src/main.rs:1149-1150) hardcodes DataSource::Synthetic, ignoring each member manifest's recorded instrument and window. Sweep members re-run over the 18-bar synthetic showcase and diverge; WalkForward members rebuild their OOS window by filtering the synthetic walk against real epoch-ns bounds, get an empty source, and hit the .expect("non-empty OOS window"). Every real-data family kind is affected (MonteCarlo included); gang and campaign pipeline are not load-bearing — the failing dimension is --real.

Design record (fork resolved, derived)

Fork: reproduce real-data families vs refuse cleanly. → Reconstruct the member's data source from its manifest (instrument + window, the same local-archive loader the dissolved verbs use); when the archive is absent, refuse with the verbs' established named-data refusal instead of a wrong-data run.
Basis: derived — C18's purpose is reproducibility of persisted research, the re-derivation machinery demonstrably produces bit-identical results over the local archive (fieldtest evidence), and the data-absence refusal keeps the contract honest on hosts without the archive. The prior doc-comment deferral ("rides the DataServer seam, #124") predates the dissolved verbs; reproduce reading the same local archive the verbs read introduces no new #124 dependency.
The WalkForward RED pin asserts panic-freedom + a handled verdict only (not bit-identity) — the OOS re-derivation path may legitimately differ from the sweep member path; tightening it to bit-identity is a follow-up decision once observed.

RED tests (cli_run.rs, skip-on-no-data): reproduce_real_sweep_family_re_derives_bit_identically, reproduce_real_walkforward_family_does_not_panic.

Surfaced by the triage-harvest milestone fieldtest (2026-07-10, evidence in fieldtests/milestone-triage-harvest/); root-caused the same day. ## Symptom - A Sweep family minted with `--real GER40` reports `reproduced 0/N members bit-identically` (every member DIVERGED, exit 1) under `aura reproduce <id>`. - A WalkForward family panics instead: `main.rs:1123: non-empty OOS window` (exit 101). - A FRESH identical sweep invocation matches the persisted metrics bit-for-bit, so the runs themselves are deterministic (C1 holds); the defect is reproduce-side. ## Cause `reproduce_family` (crates/aura-cli/src/main.rs:1149-1150) hardcodes `DataSource::Synthetic`, ignoring each member manifest's recorded instrument and window. Sweep members re-run over the 18-bar synthetic showcase and diverge; WalkForward members rebuild their OOS window by filtering the synthetic walk against real epoch-ns bounds, get an empty source, and hit the `.expect("non-empty OOS window")`. Every real-data family kind is affected (MonteCarlo included); gang and campaign pipeline are not load-bearing — the failing dimension is `--real`. ## Design record (fork resolved, derived) - **Fork: reproduce real-data families vs refuse cleanly.** → Reconstruct the member's data source from its manifest (instrument + window, the same local-archive loader the dissolved verbs use); when the archive is absent, refuse with the verbs' established named-data refusal instead of a wrong-data run. Basis: derived — C18's purpose is reproducibility of persisted research, the re-derivation machinery demonstrably produces bit-identical results over the local archive (fieldtest evidence), and the data-absence refusal keeps the contract honest on hosts without the archive. The prior doc-comment deferral ("rides the DataServer seam, #124") predates the dissolved verbs; reproduce reading the same local archive the verbs read introduces no new #124 dependency. - The WalkForward RED pin asserts panic-freedom + a handled verdict only (not bit-identity) — the OOS re-derivation path may legitimately differ from the sweep member path; tightening it to bit-identity is a follow-up decision once observed. RED tests (cli_run.rs, skip-on-no-data): reproduce_real_sweep_family_re_derives_bit_identically, reproduce_real_walkforward_family_does_not_panic.

Brummel added the bug label 2026-07-10 14:47:03 +02:00

Brummel referenced this issue

2026-07-10 15:32:43 +02:00

Define + deliver reproduce semantics for real-data MonteCarlo families (currently synthetic-only) #230

Brummel referenced this issue from a commit

2026-07-10 15:39:46 +02:00

test(cli): RED — reproduce over real-data families diverges/panics (#229)

Brummel referenced this issue from a commit

2026-07-10 15:39:46 +02:00

fix(cli): reproduce re-derives real-data families from the member manifest (#229)

Brummel closed this issue

2026-07-10 15:39:46 +02:00