Define + deliver reproduce semantics for real-data MonteCarlo families (currently synthetic-only) #230

New Issue

2026-07-10T15:32:43+02:00

Brummel commented

2026-07-10 15:32:43 +02:00

Carved out of #229 (reproduce over real-data families). The #229 fix reconstructs Sweep/WalkForward members' data sources from the family manifest; MonteCarlo family reproduce still always runs the synthetic seed-driven walk regardless of a real instrument on the manifest.

Not folded into #229 because the semantics are genuinely open: an mc R-bootstrap family's members are seeded resamples of an R-series — "reproduce bit-identically" may mean re-running the bootstrap over the re-derived member R-series (deep) or re-checking the resample aggregation from persisted series (shallow). Decide the contract, then pin it RED-first like #229.

depends on: nothing (the #229 manifest-source machinery exists)
context: #229 fix concern + milestone-fieldtest follow-up; the fieldtest exercised Sweep/WalkForward reproduce only.

Carved out of #229 (reproduce over real-data families). The #229 fix reconstructs Sweep/WalkForward members' data sources from the family manifest; MonteCarlo family reproduce still always runs the synthetic seed-driven walk regardless of a real instrument on the manifest. Not folded into #229 because the semantics are genuinely open: an mc R-bootstrap family's members are seeded resamples of an R-series — "reproduce bit-identically" may mean re-running the bootstrap over the re-derived member R-series (deep) or re-checking the resample aggregation from persisted series (shallow). Decide the contract, then pin it RED-first like #229. - depends on: nothing (the #229 manifest-source machinery exists) - context: #229 fix concern + milestone-fieldtest follow-up; the fieldtest exercised Sweep/WalkForward reproduce only.

Brummel added this to the Real-project readiness — role-complete research loop (no freeze) milestone 2026-07-10 17:16:15 +02:00

Brummel commented

2026-07-10 20:00:19 +02:00

Contract decided (2026-07-10): reproduce for a real-data MonteCarlo family is DEEP — re-derive the member's full chain from the manifest (real data source per the machinery the Sweep/WalkForward fix established, then the base R-series, then the seeded bootstrap resample) and compare metrics bit-identically.

Derivation, three grounds: (1) deep is already the MC reproduce semantic — the current code re-runs the seed-driven walk + bootstrap, merely against a hardcoded synthetic source, so the defect is in the source derivation, not the contract; (2) consistency — Sweep/WalkForward reproduce is a full member re-run from the manifest, and an mc member's primary inputs (base series + seed) re-derive the same way; (3) purpose — reproduce is the determinism audit (same input, same run): the shallow alternative (re-check the resample aggregation from persisted series) only catches storage corruption, not a drifted bootstrap or base run, and is a different, weaker kind of check that would silently narrow the verb's meaning for one family kind.

The shallow option is dropped, not deferred. RED-first delivery follows the established shape: mint a real-data mc family, reproduce it, pin the spurious synthetic-walk behaviour, then thread the manifest-derived source through the mc arm.

Contract decided (2026-07-10): reproduce for a real-data MonteCarlo family is DEEP — re-derive the member's full chain from the manifest (real data source per the machinery the Sweep/WalkForward fix established, then the base R-series, then the seeded bootstrap resample) and compare metrics bit-identically. Derivation, three grounds: (1) deep is already the MC reproduce semantic — the current code re-runs the seed-driven walk + bootstrap, merely against a hardcoded synthetic source, so the defect is in the source derivation, not the contract; (2) consistency — Sweep/WalkForward reproduce is a full member re-run from the manifest, and an mc member's primary inputs (base series + seed) re-derive the same way; (3) purpose — reproduce is the determinism audit (same input, same run): the shallow alternative (re-check the resample aggregation from persisted series) only catches storage corruption, not a drifted bootstrap or base run, and is a different, weaker kind of check that would silently narrow the verb's meaning for one family kind. The shallow option is dropped, not deferred. RED-first delivery follows the established shape: mint a real-data mc family, reproduce it, pin the spurious synthetic-walk behaviour, then thread the manifest-derived source through the mc arm.

Brummel commented

2026-07-10 20:08:14 +02:00

Diagnosis finding (2026-07-10): the framed bug is unreproducible — no CLI path mints a real-data FamilyKind::MonteCarlo family at all. Evidence: the only non-test producer of FamilyKind::MonteCarlo is run_blueprint_mc (main.rs:1859), called exactly once (main.rs:3208) with DataSource::Synthetic hardcoded, and aura mc --seeds rejects --real (main.rs:3121-3129); aura mc --real routes to the campaign path, where the MonteCarlo stage is a terminal annotator (aura-campaign/src/exec.rs:401-446, family_id: None, "nothing flows out") that persists no family — it prints the mc_r_bootstrap line and records only the Sweep/WalkForward families. For the mc families that can exist (synthetic seed walks), reproduce re-derives correctly (green covering test reproduce_family_re_derives_every_mc_member_bit_identically).

The deep contract decided in the comment of 2026-07-10 above therefore has nothing to bind to yet: delivering it is design-bearing feature work, not a bug fix. At least three shapes are open, none clearly default: (A) the campaign MC stage additionally persists an addressable MC family, with a manifest extended by the real instrument, real ms window, the wf pooling parameters that produced the OOS trade-R series, and the bootstrap knobs incl. the bootstrap seed (today's stamped seed is the price-walk seed, the wrong one); (B) the annotator design stays and only the bootstrap knobs + seed are recorded on the already-persisted campaign/wf artifacts, making the annotation re-derivable from reproducible families without a new persistence kind; (C) close as by-design: the campaign document + real data are deterministic, so re-running the campaign IS the deep reproduction of the annotation, and per-annotation addressability is declined. The choice interacts with the deliberately-settled verb-dissolution design (mc-as-annotator, #210/#220) and overlaps the open idea #172 (real-data block-bootstrap Monte-Carlo as a first-class capability).

Status: blocked on that design choice; not deliverable RED-first as framed.

Diagnosis finding (2026-07-10): the framed bug is unreproducible — no CLI path mints a real-data `FamilyKind::MonteCarlo` family at all. Evidence: the only non-test producer of `FamilyKind::MonteCarlo` is `run_blueprint_mc` (main.rs:1859), called exactly once (main.rs:3208) with `DataSource::Synthetic` hardcoded, and `aura mc --seeds` rejects `--real` (main.rs:3121-3129); `aura mc --real` routes to the campaign path, where the MonteCarlo stage is a terminal annotator (`aura-campaign/src/exec.rs:401-446`, `family_id: None`, "nothing flows out") that persists no family — it prints the `mc_r_bootstrap` line and records only the Sweep/WalkForward families. For the mc families that can exist (synthetic seed walks), reproduce re-derives correctly (green covering test `reproduce_family_re_derives_every_mc_member_bit_identically`). The deep contract decided in the comment of 2026-07-10 above therefore has nothing to bind to yet: delivering it is design-bearing feature work, not a bug fix. At least three shapes are open, none clearly default: (A) the campaign MC stage additionally persists an addressable MC family, with a manifest extended by the real instrument, real ms window, the wf pooling parameters that produced the OOS trade-R series, and the bootstrap knobs incl. the bootstrap seed (today's stamped seed is the price-walk seed, the wrong one); (B) the annotator design stays and only the bootstrap knobs + seed are recorded on the already-persisted campaign/wf artifacts, making the annotation re-derivable from reproducible families without a new persistence kind; (C) close as by-design: the campaign document + real data are deterministic, so re-running the campaign IS the deep reproduction of the annotation, and per-annotation addressability is declined. The choice interacts with the deliberately-settled verb-dissolution design (mc-as-annotator, #210/#220) and overlaps the open idea #172 (real-data block-bootstrap Monte-Carlo as a first-class capability). Status: blocked on that design choice; not deliverable RED-first as framed.

Brummel commented

2026-07-11 12:55:09 +02:00

Resolution (2026-07-11): the fork dissolves once the two distinct purposes of Monte-Carlo in trading research are separated.

Purpose 1 — trade-sequence bootstrap. A statistic over an already-produced result: resample the trade-R series to read off expectancy confidence intervals and drawdown/ruin distributions. Members are reshuffled arrays, not runs; no backtest executes. The campaign std::monte_carlo stage is exactly this, and the terminal-annotator design (#210/#220) is correct for it — persisting an addressable family per annotation would be a category error (no family is persisted for a mean or a percentile either).

Purpose 2 — data-resample Monte-Carlo. Block-bootstrap/permutation over the price series, each member a full backtest over a different data realization (the overfitting-falsification use, cf. permutation tests / White's Reality Check). That genuinely is a family in the C1 sense with deep reproduce semantics — and its home is #172, not this issue.

Decision, per the option letters laid out in the diagnosis comment of 2026-07-10 in this thread:

(C) adopted for addressability. The campaign document and the recorded real data are deterministic (C1), so re-running the campaign IS the deep reproduction of the annotation. Closed as by-design.
(B)'s core is already satisfied — found on inspection, no work item remains. The persisted campaign_run record carries: the campaign and process document content ids (a file target is register-then-run sugar, crates/aura-cli/src/campaign_run.rs:390, so both documents live in the content-addressed store; the process document carries resamples/block_len), the seed — the very value threaded into r_bootstrap (crates/aura-campaign/src/exec.rs:154 → :413/:434) and stamped on the record (exec.rs:214) —, the family ids of the reproducible base families (the Sweep/WalkForward reproduce machinery of the #229 fix), and the bootstrap result itself (StageBootstrap on the stage realization, crates/aura-registry/src/lineage.rs:150-176) to compare a re-derivation against. The "wrong stamped seed" observation in the diagnosis comment applies to the synthetic MC family manifest (whose seed is the member-generating price-walk seed), i.e. to option (A)'s hypothetical manifest — not to the campaign record.
(A) declined. It would bolt purpose-2 persistence onto the purpose-1 annotator — wrong place, wrong semantics. The ambition routes to #172, noted there.

The deep contract decided in the comment of 2026-07-10 (reproduce re-derives the full chain — source, base series, seeded resample — and compares bit-identically) stands unrevised for real MC families; it binds to the purpose-2 capability of #172 when that is built, not to the campaign annotator.

Resolution (2026-07-11): the fork dissolves once the two distinct purposes of Monte-Carlo in trading research are separated. **Purpose 1 — trade-sequence bootstrap.** A statistic over an already-produced result: resample the trade-R series to read off expectancy confidence intervals and drawdown/ruin distributions. Members are reshuffled arrays, not runs; no backtest executes. The campaign `std::monte_carlo` stage is exactly this, and the terminal-annotator design (#210/#220) is *correct* for it — persisting an addressable family per annotation would be a category error (no family is persisted for a mean or a percentile either). **Purpose 2 — data-resample Monte-Carlo.** Block-bootstrap/permutation over the *price series*, each member a full backtest over a different data realization (the overfitting-falsification use, cf. permutation tests / White's Reality Check). That genuinely is a family in the C1 sense with deep reproduce semantics — and its home is #172, not this issue. Decision, per the option letters laid out in the diagnosis comment of 2026-07-10 in this thread: - **(C) adopted for addressability.** The campaign document and the recorded real data are deterministic (C1), so re-running the campaign IS the deep reproduction of the annotation. Closed as by-design. - **(B)'s core is already satisfied** — found on inspection, no work item remains. The persisted `campaign_run` record carries: the campaign and process document content ids (a file target is register-then-run sugar, `crates/aura-cli/src/campaign_run.rs:390`, so both documents live in the content-addressed store; the process document carries `resamples`/`block_len`), the seed — the very value threaded into `r_bootstrap` (`crates/aura-campaign/src/exec.rs:154` → `:413`/`:434`) and stamped on the record (`exec.rs:214`) —, the family ids of the reproducible base families (the Sweep/WalkForward reproduce machinery of the #229 fix), and the bootstrap *result* itself (`StageBootstrap` on the stage realization, `crates/aura-registry/src/lineage.rs:150-176`) to compare a re-derivation against. The "wrong stamped seed" observation in the diagnosis comment applies to the synthetic MC *family* manifest (whose seed is the member-generating price-walk seed), i.e. to option (A)'s hypothetical manifest — not to the campaign record. - **(A) declined.** It would bolt purpose-2 persistence onto the purpose-1 annotator — wrong place, wrong semantics. The ambition routes to #172, noted there. The deep contract decided in the comment of 2026-07-10 (reproduce re-derives the full chain — source, base series, seeded resample — and compares bit-identically) stands unrevised for real MC *families*; it binds to the purpose-2 capability of #172 when that is built, not to the campaign annotator.

Brummel closed this issue

2026-07-11 12:55:09 +02:00

Brummel referenced this issue

2026-07-11 12:55:24 +02:00

Real-data block-bootstrap Monte-Carlo over a loaded blueprint #172

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Brummel/Aura#230