diff --git a/docs/design/INDEX.md b/docs/design/INDEX.md index 3eba19a..55b457e 100644 --- a/docs/design/INDEX.md +++ b/docs/design/INDEX.md @@ -644,6 +644,22 @@ reconstruction loses precision; the stored `realized_r` is exact). Composites li acyclic. **Stage 2** (currency P&L, fixed-fractional compounding through the z⁻¹ fill-edge register, realistic brokers consuming the position-event table) remains the deferred downstream layer, entered only after `E[R] > 0`. +**Realization (cycle 0066).** SQN is now the operational single-number objective for +ranking a Stage-1 sweep family by signal quality — C12 **axis-2 (argmax-metric)** over +the C18 family store. `metric_cmp` (`aura-registry`) learns the higher-is-better R +metrics `sqn`, `expectancy_r`, `net_expectancy_r`; a member with no `r` block sorts +last (`NEG_INFINITY`), so a pip-only family ranked by an R metric degrades to ordinal +order rather than erroring. `aura sweep --strategy stage1-r` produces the rankable R +family, each member folding `summarize_r` into `RunMetrics.r`. The default grid varies +**only the signal** (`fast`/`slow` SMA lengths), holding the stop and sizing fixed: +`risk_budget` is R-invariant and `bias.scale` is sign-only under flat-1R (both +degenerate axes for the ranked metric), and — load-bearing — the **stop defines 1R**, +so varying it would change what R *means* per member and break cross-member SQN +comparability (the motivation for the deferred n-normalized SQN100, #130). Each swept +member's manifest records the fixed R-defining params (stop, scale) beside the floated +knobs, so a member is reproducible from its own manifest (C18) and the constant-stop +basis of comparability is auditable from the family record. Per-member `--trace` for +the stage1-r sweep is refused explicitly (not a silent no-op) pending #135. ### C11 — Generalized sources; record-then-replay determinism boundary **Guarantee.** A source is anything that produces timestamped scalar streams — diff --git a/docs/glossary.md b/docs/glossary.md index 04ad192..3bb0c53 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -131,6 +131,10 @@ A broker-independent, time-ordered table of position events (scalar columns: `ev **Avoid:** — The native unit of strategy performance: 1R = the loss if the protective stop is hit, so a trade's outcome is a multiple of R (−1R stopped out, +2R, …) and the primary question is *"how much R out per 1R risked?"*. Risk-normalized and account-/instrument-agnostic — the Stage-1 yardstick replacing pips, defined by the stop (Van Tharp R-multiples). +### R metrics +**Avoid:** — +The R-multiple summary metrics `summarize_r` folds into `RunMetrics.r`, which double as the higher-is-better rank vocabulary for `aura runs family rank ` (cycle 0066): **`sqn`** (the System Quality Number — see SQN), **`expectancy_r`** (mean realized R per trade, the headline E[R]), and **`net_expectancy_r`** (expectancy net of one round-trip cost per trade — churn-honest; equals `expectancy_r` while Stage 1 is frictionless). A family member with no `r` block ranks last under any of them, so a pip-only family ranked by an R metric falls back to ordinal order. + ### R-evaluator **Avoid:** — The Stage-1 signal-quality node: integrates the per-trade R-outcomes of a flat-1R `RiskExecutor` into an R-expectancy / R-curve — the deterministic, account-/instrument-agnostic yardstick. The R-reframe successor of the `sim-optimal broker` (pip integral); unlike it the R-evaluator requires a stop, so it consumes a RiskExecutor, not raw bias. @@ -195,6 +199,10 @@ The columnar Structure-of-Arrays layout in which the four scalar base types are **Avoid:** — Anything that produces timestamped scalar streams — market data and non-financial feeds (e.g. a news-bias node) are treated identically; a pure producer node. `data-server` is aura's first source. +### SQN +**Avoid:** system-quality-number (spell out once, then SQN) +System Quality Number — `√n · mean_R / stdev_R`: the dispersion- and trade-count-aware single-number objective for ranking a sweep family by R signal quality (cycle 0066; van Tharp). Higher is better; `n < 2` or zero-variance → 0. One of the R metrics; an n-normalized variant (SQN100) is deferred (#130). + ### Stage 1 / Stage 2 **Avoid:** — The two-stage cleave of evaluation: **Stage 1** (R / signal quality) is flat-1R, feed-forward (no equity feedback), measured by the R-evaluator in R-expectancy — the primary research loop; **Stage 2** (currency / deploy viability) is fixed-fractional sizing that reads equity and compounds, with realistic brokers and real frictions, entered only once `E[R] > 0`. The cleave quarantines the only feedback into Stage 2 (flat-1R vs compounding is a structural axis, C11); aura's own synthesis, closest precedent vectorbt's signal vs money-management split. diff --git a/docs/plans/0066-r-sweep-rank.md b/docs/plans/0066-r-sweep-rank.md deleted file mode 100644 index ae0a68d..0000000 --- a/docs/plans/0066-r-sweep-rank.md +++ /dev/null @@ -1,542 +0,0 @@ -# R-sweep families + rank-by-SQN — Implementation Plan - -> **Parent spec:** `docs/specs/0066-r-sweep-rank.md` -> -> **For agentic workers:** REQUIRED SUB-SKILL: use the `implement` skill to run -> this plan. Steps use `- [ ]` checkboxes for tracking. - -**Goal:** Make SQN the single-number objective for ranking a Stage-1 sweep family -by R signal quality: `aura sweep --strategy stage1-r` produces an R-bearing family, -and `aura runs family rank sqn` (also `expectancy_r`, `net_expectancy_r`) ranks -it best-first. - -**Architecture:** Two independent crates. `aura-registry` — `metric_cmp` learns -three higher-is-better R metrics that reach into `RunReport.metrics.r`, with a -`None` member sorting to the worst position. `aura-cli` — `enum Strategy` gains -`Stage1R`; the stage1-r topology is extracted into one helper so the single run -keeps its identical binding (byte-output unchanged, verified by a new golden) while -the sweep floats `fast.length`/`slow.length` over a 4-point grid, folding `r: -Some(..)` per member. The family store, its persistence, and the `runs family … -rank …` routing are unchanged — they already carry full `RunReport`s and route any -metric string through `metric_cmp`. - -**Tech Stack:** `aura-registry/src/lib.rs` (Metric/metric_cmp/Display + tests), -`aura-cli/src/main.rs` (Strategy/parse_sweep_args/run_sweep/stage1_r_graph/ -stage1_r_sweep_family + unit tests), `aura-cli/tests/cli_run.rs` (golden + -integration test), read-only fixture source `aura-engine/src/report.rs`. - -**Files this plan creates or modifies:** - -- Modify: `crates/aura-cli/tests/cli_run.rs` — new golden char. test + new sweep+rank integration test -- Modify: `crates/aura-cli/src/main.rs` — `stage1_r_graph` helper, `stage1_r_sweep_family`, `Strategy::Stage1R`, parse + dispatch, usage strings, unit test -- Modify: `crates/aura-registry/src/lib.rs` — `Metric` variants, `metric_cmp` arms, `UnknownMetric` Display, R-metric tests -- Read-only: `crates/aura-engine/src/report.rs` — `RMetrics` struct literal (fixture source) - -**Task order (dependencies):** Task 1 (golden, GREEN on current HEAD) → Task 2 -(helper extraction; golden + existing stage1-r tests stay GREEN) → Task 3 -(registry metric_cmp, self-contained RED→GREEN) → Task 4 (sweep family + CLI -wiring, RED→GREEN; needs Task 2's helper and Task 3's rank-by-sqn). - ---- - -### Task 1: Golden characterization test for the stage1-r single run - -Pins the EXACT current `metrics` output so Task 2's refactor is *verified* -byte-preserving (the grounding-check found no such golden exists). Captured against -current HEAD → GREEN immediately; Task 2 must keep it GREEN. - -**Files:** -- Modify: `crates/aura-cli/tests/cli_run.rs` (append a test near the stage1-r tests ~1600) - -- [ ] **Step 1: Capture the current metrics literal** - -Run: `cargo run -q -p aura-cli -- run --harness stage1-r` -This prints one JSON line `{"manifest":{...},"metrics":{...}}`. Copy the substring -from `"metrics":` to the end of the line (the trailing `}` included) — this is the -golden literal for Step 2. Example shape (DO NOT use these numbers — paste the real -captured bytes): -`"metrics":{"total_pips":,"max_drawdown":,"bias_sign_flips":,"r":{"expectancy_r":,...,"conviction_terciles_r":[,,]}}` - -- [ ] **Step 2: Add the golden test, pasting the captured literal** - -```rust -/// Golden characterization (cycle 0066): pins the EXACT metric output of the -/// stage1-r single run, so the `stage1_r_graph` helper extraction (Task 2) is -/// VERIFIED byte-preserving, not assumed. Captured against HEAD before the -/// refactor; the refactor must keep it identical (a drift is a behaviour change → -/// debug). Commit-agnostic: pins only the `metrics` object, since the manifest -/// carries the volatile build commit. -#[test] -fn stage1_r_single_run_output_golden() { - let out = Command::new(BIN).args(["run", "--harness", "stage1-r"]).output().unwrap(); - assert!(out.status.success(), "exit: {:?}", out.status); - let s = String::from_utf8(out.stdout).expect("utf-8 stdout"); - let key = "\"metrics\":"; - let start = s.find(key).expect("stdout has a metrics object"); - let metrics = s[start..].trim_end(); - assert_eq!( - metrics, - r#"PASTE_THE_CAPTURED_METRICS_LITERAL_FROM_STEP_1"#, - "stage1-r single-run metric output drifted from the golden: {s}" - ); -} -``` - -- [ ] **Step 3: Run the golden against current HEAD — must already pass** - -Run: `cargo test -p aura-cli --test cli_run stage1_r_single_run_output_golden` -Expected: PASS (it characterizes the unmodified code). If it FAILS here, the pasted -literal is wrong — re-capture from Step 1. - ---- - -### Task 2: Extract `stage1_r_graph` helper; float fast/slow; single run binds Some(2)/Some(4) - -Behaviour-preserving refactor. One wiring source; the single-run binding is -structurally identical, so the Task-1 golden + the existing stage1-r tests stay -GREEN. - -**Files:** -- Modify: `crates/aura-cli/src/main.rs:1737-1788` (rename `stage1_r_blueprint` → `stage1_r_graph`, add the two `Option` params) -- Modify: `crates/aura-cli/src/main.rs:1806-1808` (`run_stage1_r` calls the helper with `Some(2)`/`Some(4)`) - -- [ ] **Step 1: Replace `stage1_r_blueprint` with `stage1_r_graph`** - -Replace the whole `fn stage1_r_blueprint(...) -> Composite { ... }` (1737-1788) -with the helper below. Only the `fast`/`slow` builder lines change (made -conditional on the `Option`); every other line is verbatim from the original body. - -```rust -/// The Stage-1 R harness topology, shared by the single run and the sweep. The two -/// signal knobs are bound when `Some` (single run — identical to the old -/// build-time bind, output byte-unchanged) and left free when `None` (sweep — they -/// land in `param_space` as `fast.length` / `slow.length`). Four taps: equity -/// (SimBroker), exposure (Bias), the 14-column R-record (→ summarize_r), and -/// r_equity = cum_realized_r + unrealized_r. -#[allow(clippy::type_complexity, clippy::too_many_arguments)] -fn stage1_r_graph( - tx_eq: mpsc::Sender<(Timestamp, Vec)>, - tx_ex: mpsc::Sender<(Timestamp, Vec)>, - tx_r: mpsc::Sender<(Timestamp, Vec)>, - tx_req: mpsc::Sender<(Timestamp, Vec)>, - fast_len: Option, - slow_len: Option, -) -> Composite { - let mut g = GraphBuilder::new("stage1_r"); - // SMA-cross signal → Bias (the same signal the pip sample uses). - let mut fast_b = Sma::builder().named("fast"); - if let Some(l) = fast_len { - fast_b = fast_b.bind("length", Scalar::i64(l)); - } - let fast = g.add(fast_b); - let mut slow_b = Sma::builder().named("slow"); - if let Some(l) = slow_len { - slow_b = slow_b.bind("length", Scalar::i64(l)); - } - let slow = g.add(slow_b); - let spread = g.add(Sub::builder()); - let exposure = g.add(Bias::builder().named("bias").bind("scale", Scalar::f64(0.5))); - // pip branch (verbatim from sample_blueprint_with_sinks). - let broker = g.add(SimBroker::builder(SYNTHETIC_PIP_SIZE)); - let eq = g.add(Recorder::builder(vec![ScalarKind::F64], Firing::Any, tx_eq)); - let ex = g.add(Recorder::builder(vec![ScalarKind::F64], Firing::Any, tx_ex)); - // R branch: bias + price → RiskExecutor(vol_stop) → dense R-record. - let exec = g.add(risk_executor( - StopRule::Vol { length: STAGE1_R_STOP_LENGTH, k: STAGE1_R_STOP_K }, - 1.0, - )); - let rrec = g.add(Recorder::builder(PM_RECORD_KINDS.to_vec(), Firing::Any, tx_r)); - // r_equity = cum_realized_r + unrealized_r — one tapped series for charting. - let r_equity = g.add( - LinComb::builder(2) - .bind("weights[0]", Scalar::f64(1.0)) - .bind("weights[1]", Scalar::f64(1.0)), - ); - let req = g.add(Recorder::builder(vec![ScalarKind::F64], Firing::Any, tx_req)); - let price = g.source_role("price", ScalarKind::F64); - g.feed( - price, - [fast.input("series"), slow.input("series"), broker.input("price"), exec.input("price")], - ); - g.connect(fast.output("value"), spread.input("lhs")); - g.connect(slow.output("value"), spread.input("rhs")); - g.connect(spread.output("value"), exposure.input("signal")); - // bias fans to: broker (pip), exposure sink, and the RiskExecutor (R). - g.connect(exposure.output("bias"), broker.input("exposure")); - g.connect(exposure.output("bias"), ex.input("col[0]")); - g.connect(exposure.output("bias"), exec.input("bias")); - g.connect(broker.output("equity"), eq.input("col[0]")); - // tap the dense R-record (all PM fields) for summarize_r. - for (i, field) in PM_FIELD_NAMES.iter().enumerate() { - g.connect(exec.output(field), rrec.input(COL_PORTS[i].as_str())); - } - // r_equity sum + tap. - g.connect(exec.output("cum_realized_r"), r_equity.input("term[0]")); - g.connect(exec.output("unrealized_r"), r_equity.input("term[1]")); - g.connect(r_equity.output("value"), req.input("col[0]")); - g.build().expect("stage1_r wiring resolves") -} -``` - -- [ ] **Step 2: Update `run_stage1_r` to call the helper** - -In `run_stage1_r` (~1806), change the build line: -```rust - let flat = stage1_r_graph(tx_eq, tx_ex, tx_r, tx_req, Some(2), Some(4)) - .compile_with_params(&[]) - .expect("valid stage1-r blueprint"); -``` -(was `stage1_r_blueprint(tx_eq, tx_ex, tx_r, tx_req)`.) - -- [ ] **Step 3: Build — confirm no other caller of the old name** - -Run: `cargo build -p aura-cli` -Expected: clean (0 errors). A compile error naming `stage1_r_blueprint` means -another caller exists — update it to `stage1_r_graph(.., Some(..), Some(..))`. - -- [ ] **Step 4: Run the golden + all stage1-r tests — must stay GREEN (byte-preserved)** - -Run: `cargo test -p aura-cli --test cli_run stage1_r` -Expected: PASS — `stage1_r_single_run_output_golden`, `run_harness_stage1_r_*`, -`stage1_r_trace_*` all green. A RED golden here means the extraction was NOT -byte-preserving → STOP and bounce to `debug` (do not adjust the golden to match). - ---- - -### Task 3: `metric_cmp` learns sqn / expectancy_r / net_expectancy_r (aura-registry) - -Self-contained RED→GREEN in `aura-registry`. All three higher-is-better, reaching -into `RunReport.metrics.r`; a `None` member sorts to the worst position. - -**Files:** -- Modify: `crates/aura-registry/src/lib.rs:100-104` (`enum Metric`), `:116-135` (`metric_cmp`), `:184-187` (`UnknownMetric` Display), `mod tests` (~200, fixture + tests) - -- [ ] **Step 1: Add the RED tests + an R fixture helper** - -In `mod tests`, extend the `use` and add the helper + four tests: -```rust -// add to the existing test `use aura_engine::{...}` line: -use aura_engine::RMetrics; - -/// A family member carrying an R block — mirrors `report_with` but sets -/// `metrics.r = Some(..)`. Only sqn / expectancy_r / net_expectancy_r vary per -/// call (the rank keys); the rest are fixed plausible values. -fn report_with_r(sqn: f64, expectancy_r: f64, net_expectancy_r: f64) -> RunReport { - let mut rep = report_with(0.0, 0.0, 0); - rep.metrics.r = Some(RMetrics { - expectancy_r, - n_trades: 4, - win_rate: 0.5, - avg_win_r: 1.5, - avg_loss_r: -0.5, - profit_factor: 3.0, - max_r_drawdown: 0.5, - n_open_at_end: 0, - sqn, - net_expectancy_r, - conviction_terciles_r: [0.0, 0.0, 0.0], - }); - rep -} - -#[test] -fn rank_by_sqn_orders_members_descending() { - let reports = vec![ - report_with_r(0.5, 0.1, 0.1), - report_with_r(2.0, 0.4, 0.4), - report_with_r(1.0, 0.2, 0.2), - ]; - let ranked = rank_by(reports, "sqn").expect("rank sqn"); - let sqns: Vec = ranked.iter().map(|r| r.metrics.r.as_ref().unwrap().sqn).collect(); - assert_eq!(sqns, vec![2.0, 1.0, 0.5], "sqn must rank highest-first"); -} - -#[test] -fn rank_by_expectancy_and_net_expectancy_r() { - let reports = vec![report_with_r(1.0, 0.1, 0.05), report_with_r(1.0, 0.3, 0.25)]; - let by_exp = rank_by(reports.clone(), "expectancy_r").expect("rank expectancy_r"); - assert_eq!(by_exp[0].metrics.r.as_ref().unwrap().expectancy_r, 0.3); - let by_net = rank_by(reports, "net_expectancy_r").expect("rank net_expectancy_r"); - assert_eq!(by_net[0].metrics.r.as_ref().unwrap().net_expectancy_r, 0.25); -} - -#[test] -fn rank_r_metric_sorts_none_member_last() { - // a pip-only member (r: None) ranks below every member with an R block. - let reports = vec![report_with(1.0, 0.5, 1), report_with_r(0.5, 0.1, 0.1)]; - let ranked = rank_by(reports, "sqn").expect("rank sqn with a None member"); - assert!(ranked[0].metrics.r.is_some(), "the R member ranks first"); - assert!(ranked[1].metrics.r.is_none(), "the None member sorts last"); -} - -#[test] -fn unknown_metric_message_lists_r_metrics() { - let msg = RegistryError::UnknownMetric("nope".to_string()).to_string(); - assert!(msg.contains("sqn"), "msg: {msg}"); - assert!(msg.contains("expectancy_r"), "msg: {msg}"); - assert!(msg.contains("net_expectancy_r"), "msg: {msg}"); -} -``` - -- [ ] **Step 2: Run the new tests — confirm they FAIL** - -Run: `cargo test -p aura-registry rank_by_sqn_orders_members_descending` -Expected: FAIL — `rank_by(.., "sqn")` returns `Err(UnknownMetric("sqn"))`, so -`.expect("rank sqn")` panics. (The other three fail likewise / on the message.) - -- [ ] **Step 3: Extend `enum Metric`** - -```rust -enum Metric { - TotalPips, - MaxDrawdown, - BiasSignFlips, - Sqn, - ExpectancyR, - NetExpectancyR, -} -``` - -- [ ] **Step 4: Extend `metric_cmp` (string arms + comparator arms + None→-inf)** - -In `metric_cmp`, add the string arms before the `other =>` catch-all: -```rust - "sqn" => Metric::Sqn, - "expectancy_r" => Metric::ExpectancyR, - "net_expectancy_r" => Metric::NetExpectancyR, -``` -Add a helper above the returned closure and the three comparator arms (a missing -`r` block sorts to the bottom for these higher-is-better metrics): -```rust - fn r_get(rep: &RunReport, f: impl Fn(&aura_engine::RMetrics) -> f64) -> f64 { - rep.metrics.r.as_ref().map(f).unwrap_or(f64::NEG_INFINITY) - } - Ok(move |a: &RunReport, b: &RunReport| match metric { - // ... existing TotalPips / MaxDrawdown / BiasSignFlips arms unchanged ... - Metric::Sqn => r_get(b, |r| r.sqn).total_cmp(&r_get(a, |r| r.sqn)), - Metric::ExpectancyR => r_get(b, |r| r.expectancy_r).total_cmp(&r_get(a, |r| r.expectancy_r)), - Metric::NetExpectancyR => r_get(b, |r| r.net_expectancy_r).total_cmp(&r_get(a, |r| r.net_expectancy_r)), - }) -``` -(`total_cmp` so `NEG_INFINITY` orders deterministically and no `NaN` can panic. If -`RMetrics` is not already in scope in `lib.rs`, use the fully-qualified -`aura_engine::RMetrics` as written.) - -- [ ] **Step 5: Extend the `UnknownMetric` Display known-list** - -At `lib.rs:~186`, change the known-list literal: -`(known: total_pips, max_drawdown, bias_sign_flips)` → -`(known: total_pips, max_drawdown, bias_sign_flips, sqn, expectancy_r, net_expectancy_r)` - -- [ ] **Step 6: Run the registry suite — all GREEN** - -Run: `cargo test -p aura-registry` -Expected: PASS (the four new tests + all existing, including -`rank_by_orders_best_first_per_metric` and `rank_by_unknown_metric_is_an_error`). - ---- - -### Task 4: `stage1_r_sweep_family` + `Strategy::Stage1R` + parse + dispatch - -RED→GREEN. Adds the R sweep family and wires `--strategy stage1-r` end-to-end. -Needs Task 2's `stage1_r_graph` and Task 3's `rank sqn`. - -**Files:** -- Modify: `crates/aura-cli/tests/cli_run.rs` (new integration test ~after 1033) -- Modify: `crates/aura-cli/src/main.rs:1122-1125` (`enum Strategy`), `:1149` (parse), `:1228-1231` (dispatch), new `stage1_r_sweep_family` (~after 1098), usage strings (1128/1136/1215/1974) - -- [ ] **Step 1: Add the RED integration test** - -Append to `crates/aura-cli/tests/cli_run.rs`: -```rust -/// Property (#133): `aura sweep --strategy stage1-r` runs the stage1-r harness over -/// the fast×slow signal grid (4 members), each carrying an R block, persisted as a -/// rankable family; `runs family rank sqn` orders them highest-SQN first. -#[test] -fn sweep_strategy_stage1_r_ranks_a_family_by_sqn() { - let cwd = temp_cwd("sweep-stage1-r"); - let out = Command::new(BIN) - .args(["sweep", "--strategy", "stage1-r"]) - .current_dir(&cwd) - .output() - .expect("spawn aura sweep --strategy stage1-r"); - assert!( - out.status.success(), - "stage1-r sweep exit: {:?}; stderr: {}", - out.status, - String::from_utf8_lossy(&out.stderr) - ); - let stdout = String::from_utf8(out.stdout).expect("utf-8"); - let lines: Vec<&str> = stdout.lines().collect(); - assert_eq!(lines.len(), 4, "stage1-r sweep must print 4 member lines: {stdout:?}"); - for line in &lines { - assert!(line.contains("\"r\":{"), "member must carry an r block: {line}"); - assert!(line.contains("\"sqn\""), "member r block must carry sqn: {line}"); - } - // rank the family by sqn — highest first. - let rank = Command::new(BIN) - .args(["runs", "family", "sweep-0", "rank", "sqn"]) - .current_dir(&cwd) - .output() - .expect("spawn rank sqn"); - assert!(rank.status.success(), "rank sqn exit: {:?}", rank.status); - let rank_out = String::from_utf8(rank.stdout).expect("utf-8"); - assert_eq!(rank_out.lines().count(), 4, "rank: {rank_out:?}"); - let sqns: Vec = rank_out - .lines() - .map(|l| { - let key = "\"sqn\":"; - let start = l.find(key).expect("line has sqn") + key.len(); - let tail = &l[start..]; - let end = tail.find([',', '}']).expect("token end"); - tail[..end].parse().expect("sqn is an f64") - }) - .collect(); - assert!(sqns.windows(2).all(|w| w[0] >= w[1]), "rank not descending by sqn: {sqns:?}"); - // an unknown metric is a usage error (exit 2). - let bogus = Command::new(BIN) - .args(["runs", "family", "sweep-0", "rank", "bogus"]) - .current_dir(&cwd) - .output() - .expect("spawn bogus"); - assert_eq!(bogus.status.code(), Some(2), "bogus exit: {:?}", bogus.status); - let _ = std::fs::remove_dir_all(&cwd); -} -``` - -- [ ] **Step 2: Run it — confirm it FAILS** - -Run: `cargo test -p aura-cli --test cli_run sweep_strategy_stage1_r_ranks_a_family_by_sqn` -Expected: FAIL — `--strategy stage1-r` is rejected by `parse_sweep_args` (exit 2), -so `out.status.success()` is false. (Build is clean; this is a runtime failure.) - -- [ ] **Step 3: Add `Strategy::Stage1R`** - -```rust -enum Strategy { - SmaCross, - Momentum, - Stage1R, -} -``` - -- [ ] **Step 4: Accept `--strategy stage1-r` in `parse_sweep_args`** - -In the `--strategy` value `match *value` (~1149), add before the `_ =>` arm: -```rust - "stage1-r" => Strategy::Stage1R, -``` - -- [ ] **Step 5: Dispatch in `run_sweep`** - -In the `match strategy` (~1228), add the arm: -```rust - Strategy::Stage1R => stage1_r_sweep_family(persist.then_some(name), &data), -``` - -- [ ] **Step 6: Add `stage1_r_sweep_family` (mirror `momentum_sweep_family`)** - -Insert after `momentum_sweep_family` (~1098): -```rust -/// `aura sweep --strategy stage1-r`: sweep the stage1-r harness over a fast×slow -/// SIGNAL grid (the stop + sizing stay fixed — see the cycle-0066 / #133 decision -/// log: risk_budget is R-invariant and bias.scale is sign-only under flat-1R, both -/// degenerate axes; the stop defines the R unit, so varying it would break -/// cross-member SQN comparability). Each member folds the dense R-record via -/// summarize_r, so its RunReport carries `r: Some(..)` and the family is rankable -/// by sqn / expectancy_r / net_expectancy_r. Per-member tracing is a follow-up; the -/// `_trace` slot keeps the dispatch shape uniform with the other sweeps. -fn stage1_r_sweep_family(_trace: Option<&str>, data: &DataSource) -> SweepFamily { - let pip = data.pip_size(); - let window = data.full_window(); - // a throwaway floated build, only to resolve param_space + the named axes - // (its taps are never drained — the per-point run_one rebuilds with live ones). - let floated = || { - let (tx_eq, _) = mpsc::channel(); - let (tx_ex, _) = mpsc::channel(); - let (tx_r, _) = mpsc::channel(); - let (tx_req, _) = mpsc::channel(); - stage1_r_graph(tx_eq, tx_ex, tx_r, tx_req, None, None) - }; - let space = floated().param_space(); - floated() - .axis("fast.length", [2, 3]) - .axis("slow.length", [6, 12]) - .sweep(|point| { - let (tx_eq, rx_eq) = mpsc::channel(); - let (tx_ex, rx_ex) = mpsc::channel(); - let (tx_r, rx_r) = mpsc::channel(); - let (tx_req, _rx_req) = mpsc::channel(); - let mut h = stage1_r_graph(tx_eq, tx_ex, tx_r, tx_req, None, None) - .bootstrap_with_cells(point) - .expect("stage1-r grid points are kind-checked against param_space"); - h.run(data.run_sources()); - let eq_rows: Vec<(Timestamp, Vec)> = rx_eq.try_iter().collect(); - let ex_rows: Vec<(Timestamp, Vec)> = rx_ex.try_iter().collect(); - let r_rows: Vec<(Timestamp, Vec)> = rx_r.try_iter().collect(); - let named = zip_params(&space, point); - let manifest = sim_optimal_manifest(named, window, 0, pip); - let mut metrics = summarize(&f64_field(&eq_rows, 0), &f64_field(&ex_rows, 0)); - metrics.r = Some(summarize_r(&r_rows, 0.0)); - RunReport { manifest, metrics } - }) - .expect("the stage1-r named grid matches the stage1-r param-space") -} -``` -Note: the `.axis(...)` calls must cover EVERY free knob in `param_space`. With only -`fast`/`slow` floated, `param_space` is exactly `{fast.length, slow.length}`. If -`.sweep(...)` panics on an unbound knob, an extra knob is free — inspect -`floated().param_space()` and either bind it in `stage1_r_graph` or add a -single-value axis. (Expected: exactly the two axes.) - -- [ ] **Step 7: Update the sweep usage strings to list stage1-r** - -At each `` in the sweep usage/doc text (`main.rs:1128`, `:1136` the -`usage` closure, `:1215` doc, `:1974` the top-level usage literal), change to -``. - -- [ ] **Step 8: Build + run the integration test — GREEN** - -Run: `cargo build -p aura-cli` -Expected: clean (the `run_sweep` match is now exhaustive). -Run: `cargo test -p aura-cli --test cli_run sweep_strategy_stage1_r_ranks_a_family_by_sqn` -Expected: PASS — 4 members each with an `r` block, ranked descending by sqn, -unknown metric exits 2. - -- [ ] **Step 9: Full workspace gate** - -Run: `cargo test --workspace` -Expected: PASS (no regressions; the new tests green). -Run: `cargo clippy --workspace --all-targets -- -D warnings` -Expected: clean. - ---- - -## Self-review - -1. **Spec coverage:** Half A (metric_cmp R metrics + None-worst + extended message) - = Task 3. Half B (stage1-r sweep family + --strategy + dispatch) = Tasks 2+4. - Golden (byte-preservation guard) = Task 1. Acceptance #1–#5 all covered. -2. **Placeholder scan:** the only deferred literal is the golden's captured metrics - string (Task 1 Step 1→2) — a characterization value that is *by definition* - captured from the running binary, with an exact capture command + paste slot. - This is the legitimate golden-capture pattern, not a TBD; flagged here so it is - not mistaken for one. No other placeholders. -3. **Type consistency:** `stage1_r_graph` (not `stage1_r_blueprint`) used in both - Task 2 and Task 4; `Strategy::Stage1R`, `report_with_r`, `r_get`, `RMetrics` - consistent across tasks. -4. **Step granularity:** each step is one action (one edit / one run). -5. **No commit steps:** none — the orchestrator commits the iter. -6. **Pin/replacement contiguity:** the golden pins the `metrics` substring captured - from real output (contiguous by construction); the integration test extracts - `"sqn":` tokens (no soft-wrap pin). OK. -7. **Compile-gate ordering:** Task 2's signature change (`stage1_r_blueprint` → - `stage1_r_graph` + 2 params) threads its one caller (`run_stage1_r`) in the same - task before the build gate (Step 3). Task 4's `Strategy::Stage1R` threads both - the dispatch arm (compiler-flagged) and the parse arm (silent) before the build - gate (Step 8). No deferred caller past a build gate. -8. **Verification-filter strings resolve:** `stage1_r_single_run_output_golden`, - `rank_by_sqn_orders_members_descending`, `sweep_strategy_stage1_r_ranks_a_family_by_sqn` - are the exact new test names; `stage1_r` / `-p aura-registry` are real targets; - the final gates are unfiltered `--workspace`. One positional filter per `cargo - test` (project memory). OK. diff --git a/docs/specs/0066-r-sweep-rank.md b/docs/specs/0066-r-sweep-rank.md deleted file mode 100644 index c4a7474..0000000 --- a/docs/specs/0066-r-sweep-rank.md +++ /dev/null @@ -1,283 +0,0 @@ -# R-sweep families + rank-by-SQN — Design Spec - -**Date:** 2026-06-24 -**Status:** Draft — awaiting user spec review -**Authors:** orchestrator + Claude -**Cycle:** 0066 · **Reference issue:** #133 (decision log) · **Contract:** C12 (axes 1+2), C18 (family store), C10 (R) - -## Goal - -Make SQN the operational single-number objective for ranking a Stage-1 sweep -family by R signal quality. Two halves, both deferred from cycle 0065: - -1. **An R sweep family** — `aura sweep --strategy stage1-r` runs the stage1-r - harness over a signal-param grid and persists each member's full `RunReport` - (including the `r: Some(RMetrics)` block) into the `families.jsonl` store. -2. **R-aware ranking** — `aura runs family rank ` learns three R - metrics — `sqn`, `expectancy_r`, `net_expectancy_r` (all higher-is-better) — - so a family is ranked best-first by signal quality. - -This realizes C12 axis-2 (argmax-metric) over the C18 family store for the R -yardstick. It is **not** a new mechanism: the store already persists the `r` -block; the sweep builder is already harness-agnostic; only the CLI harness -selection and the `metric_cmp` vocabulary need extending. - -## Non-goals (explicit scope fence) - -- **No new grid syntax.** The grid is a hardcoded default like the existing - `sweep_family` / `momentum_sweep_family`. CLI-driven custom grids are a - separate feature; no existing sweep has them. -- **No sweep of the stop or sizing.** `bias.scale`, `risk_budget`, and the - vol-stop `length`/`k` are held fixed — see the reference-issue decision log - (degenerate axes for the ranked R metric; varying the stop changes what R - means and motivates the deferred SQN100 #130). -- **No `--harness`/`--strategy` flag unification.** The sweep keeps its own - `--strategy` grammar; unifying it with `run --harness` is an orthogonal - pre-existing CLI divergence, a candidate follow-up `idea`. -- **No n-normalized SQN100** (#130, sibling) and **no friction** (`round_trip_cost` - stays 0.0 — Stage-1 is frictionless; `net_expectancy_r` therefore equals - `expectancy_r` on a stage1-r family, but the metric is wired so #120's friction - lights it up unchanged). - -## Architecture - -Two crates change, independently: - -- **`aura-registry`** (`src/lib.rs`) — `metric_cmp` gains three R-metric arms - that reach into `RunReport.metrics.r: Option`; a `None` member sorts - to the worst position (treated as `f64::NEG_INFINITY` for the higher-is-better - R metrics). The `UnknownMetric` "known: …" message is extended. No persistence - change — the store already carries `r`. -- **`aura-cli`** (`src/main.rs`) — `enum Strategy` gains `Stage1R`; the sweep - argument parser accepts `--strategy stage1-r`; `run_sweep` dispatches to a new - `stage1_r_sweep_family`. The stage1-r topology is extracted into a shared helper - parameterized by the signal knobs: the **single run keeps binding** `fast=2` / - `slow=4` (the same `.bind(...)` calls, now routed through the helper — its code - path and recorded output are unchanged *by construction*), while the **sweep - floats** `fast.length` / `slow.length` so they appear in `param_space` and are - varied via `.axis(...)`. A new **golden characterization test** of the single-run - output is added and committed against current HEAD *first*, so the refactor is - *verified* byte-preserving (the grounding-check correctly noted no such golden - exists today — adding it closes a real coverage gap). - -The sweep-member run path, the `FamilyRunRecord` shape, `append_family`, -`load_family_members`, `group_families`, and the `runs family … rank …` CLI -routing are all **unchanged** — they already flow full `RunReport`s and route any -metric string through `metric_cmp`. - -## Concrete code shapes - -### User-facing program (the acceptance evidence) - -```console -$ aura sweep --strategy stage1-r --name r-sweep -# runs the stage1-r harness over the fast×slow signal grid (4 members), -# persists 4 FamilyRunRecord lines to runs/families.jsonl, each with r: Some(..) - -$ aura runs family r-sweep-0 rank sqn -{"manifest":{...,"params":{"sma_fast":2,"sma_slow":6,...}},"metrics":{"total_pips":..,"r":{"sqn":1.83,"expectancy_r":0.42,...}}} -{"manifest":{...,"params":{"sma_fast":3,"sma_slow":12,...}},"metrics":{...,"r":{"sqn":1.10,...}}} -{"manifest":{...},"metrics":{...,"r":{"sqn":0.50,...}}} -{"manifest":{...},"metrics":{...,"r":{"sqn":-0.20,...}}} -# members printed best-first by SQN (descending). expectancy_r / net_expectancy_r likewise. - -$ aura runs family r-sweep-0 rank wibble -aura: unknown metric 'wibble' (known: total_pips, max_drawdown, bias_sign_flips, sqn, expectancy_r, net_expectancy_r) -# exit 2 — vocabulary now lists the R metrics -``` - -### `metric_cmp` — before → after (`aura-registry/src/lib.rs:99-135`) - -Before: `enum Metric { TotalPips, BiasSignFlips, MaxDrawdown }`, all reading -top-level `a.metrics.*`. After: - -```rust -enum Metric { TotalPips, BiasSignFlips, MaxDrawdown, Sqn, ExpectancyR, NetExpectancyR } - -fn metric_cmp(metric: &str) - -> Result Ordering, RegistryError> -{ - let metric = match metric { - "total_pips" => Metric::TotalPips, - "max_drawdown" => Metric::MaxDrawdown, - "bias_sign_flips" | "exposure_sign_flips" => Metric::BiasSignFlips, - "sqn" => Metric::Sqn, - "expectancy_r" => Metric::ExpectancyR, - "net_expectancy_r" => Metric::NetExpectancyR, - other => return Err(RegistryError::UnknownMetric(other.to_string())), - }; - // higher-is-better R metrics: a missing r block sorts to the bottom (-inf). - fn r_get(rep: &RunReport, f: impl Fn(&RMetrics) -> f64) -> f64 { - rep.metrics.r.as_ref().map(f).unwrap_or(f64::NEG_INFINITY) - } - Ok(move |a: &RunReport, b: &RunReport| match metric { - Metric::TotalPips => b.metrics.total_pips.partial_cmp(&a.metrics.total_pips).unwrap(), - Metric::MaxDrawdown => a.metrics.max_drawdown.partial_cmp(&b.metrics.max_drawdown).unwrap(), - Metric::BiasSignFlips => a.metrics.bias_sign_flips.cmp(&b.metrics.bias_sign_flips), - Metric::Sqn => r_get(b, |r| r.sqn).total_cmp(&r_get(a, |r| r.sqn)), - Metric::ExpectancyR => r_get(b, |r| r.expectancy_r).total_cmp(&r_get(a, |r| r.expectancy_r)), - Metric::NetExpectancyR => r_get(b, |r| r.net_expectancy_r).total_cmp(&r_get(a, |r| r.net_expectancy_r)), - }) -} -``` - -`total_cmp` (not `partial_cmp().unwrap()`) for the R metrics so `NEG_INFINITY` -orders deterministically and a stray `NaN` cannot panic. The `UnknownMetric` -Display string gains `, sqn, expectancy_r, net_expectancy_r`. - -### stage1-r blueprint — extract a shared helper, float the signal knobs only for the sweep (`aura-cli/src/main.rs:1745-1746, 1806-1808`) - -Today `stage1_r_blueprint` binds `fast`/`slow` at build, and the single run -compiles with an empty point: - -```rust -let fast = g.add(Sma::builder().named("fast").bind("length", Scalar::i64(2))); -let slow = g.add(Sma::builder().named("slow").bind("length", Scalar::i64(4))); -// ... -let flat = stage1_r_blueprint(..).compile_with_params(&[]).expect(..); -``` - -After: the topology moves into one helper; the ONLY difference between the -single-run and the sweep build is whether the two signal knobs are bound or left -free — one wiring source (no drift), and the single-run path keeps the *identical* -`.bind(...)` calls (so its output is byte-unchanged by construction): - -```rust -// shared wiring; a None length floats the knob (it lands in param_space), a -// Some(len) binds it at build exactly as today. -fn stage1_r_graph(.., fast_len: Option, slow_len: Option) -> Composite { - let mut fast_b = Sma::builder().named("fast"); - if let Some(l) = fast_len { fast_b = fast_b.bind("length", Scalar::i64(l)); } - let fast = g.add(fast_b); - // …slow likewise; the rest of the topology is verbatim from today's body… -} - -// single run — binds the same constants; code path + output byte-identical to today: -let flat = stage1_r_graph(.., Some(2), Some(4)).compile_with_params(&[]).expect(..); - -// sweep — floats them, binds per grid point via .axis(...): -let bp = stage1_r_graph(.., None, None); // fast.length / slow.length now in param_space -bp.axis("fast.length", [2, 3]).axis("slow.length", [6, 12]).sweep(run_one) -``` - -(`bias.scale`, the `risk_executor` stop and `risk_budget` stay bound inside their -builders — only `fast`/`slow` lengths are made floatable. The exact helper -signature is the planner's to pin against the real `stage1_r_blueprint` body.) - -### stage1-r sweep family — new, mirrors `momentum_sweep_family` (`aura-cli/src/main.rs`) - -```rust -enum Strategy { SmaCross, Momentum, Stage1R } // + Stage1R - -// in run_sweep dispatch: -let family = match strategy { - Strategy::SmaCross => sweep_family(persist.then_some(name), &data), - Strategy::Momentum => momentum_sweep_family(persist.then_some(name), &data), - Strategy::Stage1R => stage1_r_sweep_family(persist.then_some(name), &data), -}; - -fn stage1_r_sweep_family(trace: Option<&str>, data: &DataSource) -> SweepFamily { - // build the stage1-r blueprint with fast/slow floated; bind a 4-point grid: - // fast.length ∈ {2, 3} × slow.length ∈ {6, 12} (fast < slow always) - // run_one bootstraps each point, drains the four taps, folds RunMetrics with - // metrics.r = Some(summarize_r(&r_rows, 0.0)) // same fold as run_stage1_r - // returns a SweepFamily of 4 members, each report carrying r: Some(..). -} -``` - -`--strategy stage1-r` is accepted by `parse_sweep_args` (it already tokenizes -`--strategy `; the value match gains a `stage1-r => Strategy::Stage1R` arm). - -## Components - -| Component | File | Change | -|---|---|---| -| `Metric` enum + `metric_cmp` | `aura-registry/src/lib.rs` | +3 variants, +3 string arms, +3 comparator arms with `None`→worst; extend `UnknownMetric` Display | -| `Strategy` enum | `aura-cli/src/main.rs` | +`Stage1R` | -| `parse_sweep_args` | `aura-cli/src/main.rs` | accept `--strategy stage1-r` | -| `run_sweep` dispatch | `aura-cli/src/main.rs` | +`Strategy::Stage1R` arm | -| `stage1_r_blueprint` → `stage1_r_graph` helper | `aura-cli/src/main.rs` | extract shared wiring; `fast`/`slow` lengths floatable via `Option` args | -| `run_stage1_r` | `aura-cli/src/main.rs` | call the helper with `Some(2)`/`Some(4)` — binding + output unchanged | -| golden char. test (new) | `aura-cli` (`tests/cli_run.rs` or unit) | pin current stage1-r single-run output against HEAD *before* the refactor | -| `stage1_r_sweep_family` (new) | `aura-cli/src/main.rs` | the 4-point R sweep, folding `r: Some(..)` per member | - -## Data flow - -`aura sweep --strategy stage1-r --name r-sweep` -→ `stage1_r_sweep_family` builds the floated blueprint, `.axis("fast.length",[2,3]).axis("slow.length",[6,12]).sweep(run_one)` -→ each `run_one(point)` bootstraps a disjoint sim (C1), drains equity/exposure/R-record/r-equity taps, folds a `RunReport { manifest, metrics{ r: Some(summarize_r(..)) } }` -→ `Registry::append_family("r-sweep", Sweep, reports)` writes 4 `FamilyRunRecord` lines to `runs/families.jsonl`. - -`aura runs family r-sweep-0 rank sqn` -→ `load_family_members` → `group_families` → find `r-sweep-0` -→ `rank_by(reports, "sqn")` → `metric_cmp("sqn")` → sort descending by `r.sqn` (None→-inf) -→ print each `RunReport` JSON line best-first. - -## Error handling - -- **Unknown metric** — unchanged path: `RegistryError::UnknownMetric`, `eprintln!`, - `exit(2)`; the "known:" list now includes the three R metrics. -- **Ranking a pip-only family by an R metric** — every member is `r: None` → all - `NEG_INFINITY` → stable tie → ordinal order. No error (documented, by the - None-sorts-worst decision). -- **Mixed Some/None** (not reachable from a homogeneous family today, but defined): - `Some` members rank above `None` members. -- **Empty / degenerate R block** — `summarize_r` already returns `0.0` for - `sqn`/`expectancy_r` on `n<2`/empty (cycle 0065); a member with no trades has a - well-defined `r: Some(RMetrics{ sqn: 0.0, .. })` and ranks among the others. -- **`fast >= slow`** — excluded by grid construction (all points keep fast` with - distinct `r.sqn` ranks highest-SQN first. -- `rank_by_expectancy_r` and `rank_by_net_expectancy_r` — likewise. -- `rank_r_metric_sorts_none_member_last` — a mix of `r: Some` and `r: None` ranks - every `Some` above every `None`. -- `rank_pip_only_family_by_r_metric_is_ordinal` — all-`None` reports keep input - order (stable tie). -- `unknown_metric_message_lists_r_metrics` — the `UnknownMetric` Display contains - `sqn`, `expectancy_r`, `net_expectancy_r`. - -**`aura-cli` tests:** -- `stage1_r_single_run_output_golden` (added FIRST, before the refactor) — a - characterization test pinning the EXACT current stage1-r single-run output (the - emitted `RunReport` JSON, including the `r` block) as a committed golden literal, - captured against current HEAD so it is GREEN immediately. The helper extraction - must keep it green; a drift means the change was not behaviour-preserving → - bounce to `debug`. This is the guard the grounding-check found missing. -- `stage1_r_sweep_family_members_carry_r` — `stage1_r_sweep_family` returns 4 - members, each `report.metrics.r.is_some()`, and `fast` then - `aura runs family -0 rank sqn` → exit 0, 4 JSON lines, SQN non-increasing - down the output; an unknown metric → exit 2 with the extended message. -- Determinism: re-running the same sweep yields byte-identical member reports (C1). - -Commands: build `cargo build --workspace`; test `cargo test --workspace` (ONE -positional filter substring per invocation); lint -`cargo clippy --workspace --all-targets -- -D warnings`. - -## Acceptance criteria - -1. `aura sweep --strategy stage1-r --name N` writes a 4-member family to - `families.jsonl`, every member carrying `r: Some(RMetrics)`. -2. `aura runs family N-0 rank sqn` (and `expectancy_r`, `net_expectancy_r`) prints - members best-first by that metric; an unknown metric exits 2 with the extended - "known:" list. -3. The stage1-r single-run output is byte-unchanged — *verified* by a golden - characterization test added against current HEAD before the refactor and kept - green through it (not assumed; a drift bounces to `debug`). -4. A pip-only (SMA/momentum) family ranked by an R metric degrades to ordinal - order without error. -5. Full workspace suite green, clippy `-D warnings` clean; the sweep is - deterministic and reproducible (C1). - -This serves the aura research loop directly (C12 axis-2 on the C18 family store -for the R yardstick), measurably improves the research surface (the first -R-based family ranking), and reintroduces no determinism/causality violation -(members are disjoint deterministic sims).