diff --git a/crates/aura-research/src/lib.rs b/crates/aura-research/src/lib.rs index da5fb75..8a9e6a6 100644 --- a/crates/aura-research/src/lib.rs +++ b/crates/aura-research/src/lib.rs @@ -1168,6 +1168,26 @@ mod tests { assert!(ca.contains(r#""values":[8.0,12.0,16.0]"#)); } + /// The intrinsic tier does NOT constrain where a population-transforming + /// stage sits relative to the annotators (terminal_seen gates only Gate + /// stages): `[sweep, monte_carlo, walk_forward]` is intrinsically valid. + /// Execution-shape refusals are the executor preflight's job, one tier up + /// — this pin is the boundary between the two tiers. + #[test] + fn validate_process_permits_walk_forward_after_an_annotator() { + let doc = parse_process( + r#"{ "format_version": 1, "kind": "process", "name": "wf-after-mc", + "pipeline": [ + { "block": "std::sweep", "metric": "sqn", "select": "argmax" }, + { "block": "std::monte_carlo", "resamples": 100, "block_len": 5 }, + { "block": "std::walk_forward", "in_sample_ms": 4000, "out_of_sample_ms": 1000, + "step_ms": 1000, "mode": "rolling", "metric": "sqn", "select": "argmax" } + ] }"#, + ) + .unwrap(); + assert_eq!(validate_process(&doc), Vec::new()); + } + #[test] fn validate_process_accepts_the_fixture_and_reports_each_fault() { let ok = parse_process(PROCESS_FIXTURE).unwrap(); diff --git a/docs/specs/0108-annotator-stages.md b/docs/specs/0108-annotator-stages.md new file mode 100644 index 0000000..2840364 --- /dev/null +++ b/docs/specs/0108-annotator-stages.md @@ -0,0 +1,280 @@ +# Annotator Stages — mc/generalize Execution — Design Spec (cycle 0108) + +**Date:** 2026-07-03 +**Status:** Draft — /boss autonomous run; sign-off gate = grounding-check PASS +**Authors:** orchestrator + Claude +**Reference issue:** #200 (this cycle's anchor). All five design forks are +decided on #200 (fork-triage comment 2026-07-03, five-lens swarm, all +convergent on cited ground); this spec encodes those decisions and adds none. + +## Goal + +`aura campaign run` executes the two annotator stages the v1 executor refused: +the v2 executable pipeline shape is +`std::sweep [std::gate]* [std::walk_forward]? [std::monte_carlo]? [std::generalize]?`. +`std::monte_carlo` bootstraps the stage's incoming R-evidence (per-survivor +after sweep/gates; the pooled per-window OOS series after a walk_forward) via +the shipped `r_bootstrap`; `std::generalize` runs at campaign scope over the +per-cell selection winners of each (strategy, window) across instruments via +the shipped `generalization`. Both are terminal annotators — nothing flows out +of them (#200 decision 3, unanimous) — and no document-schema bytes change: +every existing content id survives this cycle. + +## Architecture + +All execution changes live in **`aura-campaign`** (preflight v2 + the two +stage executions); **`aura-registry`** widens the realization records with +sparse serde-default fields (existing `campaign_runs.jsonl` lines stay +parseable — the C14/C23 widening convention) and derives serde on +`Generalization`; **`aura-engine`** derives serde on `RBootstrap`; +**`aura-cli`** gains prose for the new faults and loses the v1-boundary +refusal for mc/generalize (its pinned tests flip to the new semantics). +Determinism (C1): the bootstrap seeds from the doc's `seed` (the deflation +convention); all annotator inputs are the executor's fresh in-memory reports — +`trade_rs` is `#[serde(skip)]` and empty after any registry round-trip, so the +annotators run inside `execute` or not at all. + +## Concrete code shapes + +### The user-facing program + +The 0107 fieldtest process grows the annotator suffix — no other document +changes (axes, campaign, data as in +`fieldtests/cycle-0107-campaign-executor/c0107_2_campaign_full.json`): + +```json +{ + "format_version": 1, + "kind": "process", + "name": "screen-wf-bootstrap-generalize", + "pipeline": [ + { "block": "std::sweep", "metric": "sqn_normalized", "select": "argmax", "deflate": true }, + { "block": "std::gate", "all": [ { "metric": "expectancy_r", "cmp": "gt", "value": 0.0 } ] }, + { "block": "std::walk_forward", "in_sample_ms": 1209600000, "out_of_sample_ms": 604800000, + "step_ms": 604800000, "mode": "rolling", "metric": "sqn_normalized", "select": "argmax" }, + { "block": "std::monte_carlo", "resamples": 1000, "block_len": 5 }, + { "block": "std::generalize", "metric": "net_expectancy_r" } + ] +} +``` + +``` +$ aura campaign run campaign.json # >= 2 instruments for generalize +…family/selection lines as today… +{"campaign_run":{"campaign":"…","process":"…","run":0,"seed":42,"cells":[…, + {"block":"std::walk_forward","family_id":"…"}, + {"block":"std::monte_carlo","bootstrap":{"pooled_oos":{"e_r":{…},"prob_le_zero":0.031,"n_trades":57,"block_len":5,"n_resamples":1000}}}…], + "generalizations":[{"strategy_ordinal":0,"window_ordinal":0,"generalization":{"selection_metric":"net_expectancy_r","n_instruments":2,"worst_case":0.04,"sign_agreement":2,"per_instrument":[["EURUSD",0.04],["GER40",0.11]]},"winners":[["EURUSD",[["sma_cross.fast.length",{"I64":3}],…]],["GER40",[…]]]}]}} +$ echo $? +0 +``` + +A process without walk_forward: the mc stage annotates **each surviving +member** instead (`"bootstrap":{"per_survivor":[[0,{…RBootstrap…}],[2,{…}]]}`, +ordinals indexing the population family — the `survivor_ordinals` convention). + +### aura-engine + +```rust +// mc.rs — RBootstrap currently derives Clone/Debug/PartialEq only: +#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] +pub struct RBootstrap { … unchanged fields … } +// (MetricStats, its field type, gains the same derives if it lacks them.) +``` + +### aura-registry + +```rust +// lib.rs — Generalization gains serde derives (fields unchanged). + +// lineage.rs — the mc annotation, payload mirroring the input shape (#200 d1/d4): +#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum StageBootstrap { + /// After sweep/gates: one bootstrap per surviving member, + /// `(ordinal into the population family, result)`. + PerSurvivor(Vec<(usize, RBootstrap)>), + /// After a walk_forward: one bootstrap over the pooled per-window + /// OOS trade-R series (roll order). + PooledOos(RBootstrap), +} + +pub struct StageRealization { + …existing fields… + /// std::monte_carlo stages only. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub bootstrap: Option, +} + +/// The campaign-scope generalize annotation (#200 d2): recorded at the scope +/// it is computed, keyed (strategy, window) across instruments. +#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] +pub struct CampaignGeneralization { + pub strategy_ordinal: usize, + pub window_ordinal: usize, + /// None when fewer than 2 instrument cells produced a winner (gate + /// truncation) — the shortfall is recorded, never computed around. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub generalization: Option, + /// Per-instrument winning params — divergent winners are exposed, + /// never averaged away. Present whenever >= 1 winner exists. + pub winners: Vec<(String, Vec<(String, Scalar)>)>, + /// Instrument cells that contributed no winner (gate-truncated), by name. + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub missing: Vec, +} + +pub struct CampaignRunRecord { + …existing fields… + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub generalizations: Vec, +} +``` + +### aura-campaign — preflight v2 + +The v1 shape check is replaced by the v2 shape +`sweep (gate)* (walk_forward)? (monte_carlo)? (generalize)?` (each suffix +stage at most once, order fixed). `UnsupportedStage` disappears; new faults: + +```rust +pub enum ExecFault { + …existing minus UnsupportedStage… + /// generalize needs >= 2 instruments in the campaign (static). + GeneralizeNeedsInstruments { available: usize }, + /// generalize's metric must be an R metric (static — the intrinsic tier + /// accepts any vocabulary name; the shipped generalization() would only + /// refuse at runtime). + GeneralizeNonRMetric { metric: String }, + /// monte_carlo with resamples == 0 or block_len == 0 (static, instead of + /// the engine's defined all-zero degenerate). + ZeroBootstrapParam { stage: usize, field: &'static str }, +} +``` + +Shape violations (annotator before a population stage, duplicates, generalize +not last, mc before wf) stay `PipelineShape { detail }` with precise prose. +The generalize R-metric check reuses the registry's `check_r_metric`. + +### aura-campaign — execution + +```rust +// run_cell, new arm (replaces the unreachable! for MonteCarlo): +StageBlock::MonteCarlo { resamples, block_len } => { + let bootstrap = match &wf_family { + // after walk_forward: pool the wf family's OOS trade_rs in roll + // order (the shipped pooled_oos_trade_rs pattern), one bootstrap + Some(fam) => StageBootstrap::PooledOos(r_bootstrap( + &pooled_trade_rs(&fam.reports), *resamples as usize, *block_len as usize, seed)), + // after sweep/gates: one bootstrap per surviving member over its + // fresh in-memory trade_rs; zero-trade members record the defined + // all-zero degenerate explicitly + None => StageBootstrap::PerSurvivor( + survivors.iter().map(|(ordinal, _, report)| (*ordinal, r_bootstrap( + report.metrics.r.as_ref().map(|r| r.trade_rs.as_slice()).unwrap_or(&[]), + *resamples as usize, *block_len as usize, seed))).collect()), + }; + stages.push(StageRealization { block: "std::monte_carlo".into(), + family_id: None, survivor_ordinals: None, selection: None, + bootstrap: Some(bootstrap) }); +} + +// execute(), after the cell loop — the campaign-scope generalize phase: +// for each (strategy_ordinal, window_ordinal): collect each instrument +// cell's FINAL selection winner report (the wf stage's per-window winners +// pooled? NO — the cell's nominated candidate is the last selection-bearing +// stage's winner REPORT: the sweep winner when no wf ran, else the wf +// family's last-window OOS report); >= 2 winners -> generalization(&pairs, +// metric) with winners+missing recorded; < 2 -> generalization: None, +// shortfall in `missing`. +``` + +The cell's **nominated report** for generalize is defined as: the OOS report +of the *last* walk-forward window when a wf stage ran (the freshest +out-of-sample evidence for the finally-selected params), else the sweep +stage's winner report. This is a definition this spec fixes so the executor, +the record, and the tests agree; it is recorded implicitly by `winners` +carrying the nominated params per instrument. + +`run_cell` therefore returns (in `CellOutcome`) what the campaign phase needs: +the nominated winner report + params per cell (a small `nominee: +Option<(Vec<(String, Scalar)>, RunReport)>` on `CellOutcome`), `None` for +gate-truncated cells. + +### aura-cli + +- `exec_fault_prose`: arms for the three new faults (house style, e.g. + `campaign has {available} instrument(s); std::generalize needs at least 2`, + `process stage {stage}: monte_carlo {field} must be > 0`, + `std::generalize metric "{metric}" is not an R metric (gross/net R families + generalize; pip metrics do not)`); the `UnsupportedStage` arm and its + "not executable in v1" prose disappear (compile-driven by the variant + removal). +- The `campaign_run_v1_boundary_refuses_mc_process` seam test flips: an + mc-bearing process now RUNS (fake or real data permitting) — replaced by + pins on the new refusals (mc-before-wf shape, generalize single-instrument). +- `PROCESS_BLOCKS` doc strings in aura-research: "(terminal annotator in v1)" + → "(terminal annotator)" — introspection TEXT only, no document bytes, no + content-id movement (verify: no golden pins the parenthetical; the describe + tests pin slot lines). + +## Components + +| Change | Crate | +|---|---| +| `RBootstrap` (+`MetricStats` if needed) serde derives | aura-engine | +| `Generalization` serde derives; `StageBootstrap`, `CampaignGeneralization`, `StageRealization.bootstrap`, `CampaignRunRecord.generalizations` | aura-registry | +| preflight v2 (shape + 3 static guards), mc arm, campaign-scope generalize phase, `CellOutcome.nominee`, `pooled_trade_rs` helper | aura-campaign | +| fault prose swap, seam-test flips, doc-string tidy | aura-cli, aura-research | + +## Data flow + +``` +cell loop (unchanged) … wf stage + └─ mc stage: wf ran ? pooled OOS series → one RBootstrap + : per-survivor trade_rs → Vec<(ordinal, RBootstrap)> + → StageRealization.bootstrap +after all cells: for each (strategy, window): + nominees across instruments (>= 2 ? generalization() : shortfall) + → CampaignRunRecord.generalizations +stdout: the campaign_run line carries both (serde) — no new emit kinds +``` + +## Error handling + +Exit codes unchanged (refusals 1 before any member runs; realization exit 0). +New refusals are static preflight faults with Debug-free prose. A zero-trade +bootstrap input is NOT an error: the engine's defined all-zero `RBootstrap` +is recorded explicitly (a null result is a valid research result). + +## Testing strategy + +1. **aura-registry:** record round-trip with `bootstrap`/`generalizations` + populated; a pre-widening `campaign_runs.jsonl` line (no new fields) still + parses (serde-default pin). +2. **aura-campaign (fake runner):** per-survivor mc (ordinals + per-member + RBootstrap against hand-computed `r_bootstrap` on planted `trade_rs`); + pooled-OOS mc after wf (input = wf family reports' trade_rs in roll + order); zero-trade degenerate recorded; deterministic twice (doc seed); + generalize over 2 fake instruments (worst_case/sign_agreement against the + shipped function, winners + params recorded); gate-truncated shortfall + (`generalization: None`, `missing` named); all new preflight refusals + (shape suffix order, duplicates, mc-before-wf, generalize-not-last, + single-instrument, non-R metric, zero resamples/block_len). +3. **aura-cli seam:** new-fault prose pins (Debug-free); the flipped + v1-boundary test; `--metrics`/introspection untouched. +4. **Gated real-data e2e:** extend the 0107 e2e process with the annotator + suffix where local data exists (2 instruments if a second archive exists, + else the mc-only suffix + a generalize-refusal assert); skip elsewhere. + +## Acceptance criteria + +1. The worked process above runs end-to-end (data permitting): exit 0, the + `campaign_run` line and stored record carry `bootstrap` and (with >= 2 + instruments) `generalizations`. +2. A no-wf pipeline annotates every surviving member (`per_survivor`). +3. All new preflight refusals fire before any member runs, with precise + prose; `[sweep, mc, walk_forward]` — intrinsically valid — is refused. +4. Existing stored `campaign_runs.jsonl` lines parse unchanged; no document + content id moves this cycle. +5. Suite green, clippy clean, doc build clean.