plan: 0078 cross-instrument generalization
Three tasks for #146: (1) RunManifest.instrument serde-widened field + compat-mirror lockstep + thread all in-workspace literals + back-compat tests; (2) aura-registry FamilyKind::CrossInstrument + the R-only generalization reduction (worst-case floor + sign-agreement + breakdown) + check_r_metric + two RegistryError variants; (3) aura-cli generalize subcommand (single-cell stage1-r candidate x instrument list), data-free arity/metric refusals, a gated-skip cross-instrument E2E. fieldtests/ confirmed outside the workspace compile set; FamilyKind has no exhaustive match (serde-derive); only RegistryError's Display match is a gate. refs #146
This commit is contained in:
@@ -0,0 +1,789 @@
|
||||
# Cross-instrument generalization — Implementation Plan
|
||||
|
||||
> **Parent spec:** `docs/specs/0078-cross-instrument-generalization.md`
|
||||
>
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: use the `implement` skill to run
|
||||
> this plan. Steps use `- [ ]` checkboxes for tracking.
|
||||
|
||||
**Goal:** Add an `aura generalize` subcommand that grades how consistently a
|
||||
single candidate holds across a set of instruments (worst-case R floor +
|
||||
sign-agreement + per-instrument breakdown), as the last anti-false-discovery
|
||||
piece of the Inferential-validation milestone (#146).
|
||||
|
||||
**Architecture:** A first-class `RunManifest.instrument` lineage field
|
||||
(serde-widened, C14/C18) + a new `FamilyKind::CrossInstrument` (C12) + an R-only
|
||||
`generalization` reduction in aura-registry beside `optimize_deflated`/
|
||||
`optimize_plateau` (C9) + a CLI `generalize` subcommand that runs a single-cell
|
||||
stage1-r candidate across an instrument list, stamps each member's instrument,
|
||||
reduces, prints, and persists a CrossInstrument family. Additive — existing
|
||||
sweep/walkforward/mc/standalone paths stay byte-identical (C23).
|
||||
|
||||
**Tech Stack:** aura-engine (`RunManifest`), aura-registry (`FamilyKind`,
|
||||
`RegistryError`, `generalization`/`Generalization`/`check_r_metric`), aura-cli
|
||||
(`generalize` subcommand).
|
||||
|
||||
---
|
||||
|
||||
**Files this plan creates or modifies:**
|
||||
|
||||
- Modify: `crates/aura-engine/src/report.rs:463-482` — add `RunManifest.instrument` field + back-compat serde tests
|
||||
- Modify: `crates/aura-registry/src/compat.rs:26-80` — the `RunManifestRead` mirror + destructure + literal (lockstep with the engine field)
|
||||
- Modify: `crates/aura-registry/src/lineage.rs:22-27` — `FamilyKind::CrossInstrument` variant
|
||||
- Modify: `crates/aura-registry/src/lib.rs:429-455` — `Generalization` struct + `generalization` fn + `check_r_metric` fn + two `RegistryError` variants + Display arms + unit tests
|
||||
- Modify: `crates/aura-cli/src/main.rs` — imports (`:26`), `sim_optimal_manifest` literal (`:170`), `parse_generalize_args` + `run_generalize` + `generalize_json` (new fns near the sweep helpers), dispatch arm (`:2993`), `USAGE` (`:2956`), parser unit tests
|
||||
- Modify (thread `instrument: None`): every in-workspace `RunManifest { .. }` literal — enumerated set in Task 1, compiler-driven by `cargo check --workspace --all-targets`
|
||||
- Test: `crates/aura-cli/tests/cli_run.rs` — gated-skip success E2E + two data-free must-fail E2Es
|
||||
|
||||
**Out of scope (named in the spec):** reading-B cross-instrument *selection*;
|
||||
per-instrument walk-forward; a persisted family-aggregate record; cross-symbol
|
||||
R-pooling (#139 stays deferred); pip-metric generalization. `fieldtests/` is NOT
|
||||
a workspace member (verified: workspace members are the 7 `crates/` only), so its
|
||||
16 `RunManifest` literals are out of the compile-gate.
|
||||
|
||||
---
|
||||
|
||||
## Task 1: `RunManifest.instrument` field + compat mirror + thread all literals + back-compat tests
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/aura-engine/src/report.rs:463-482` (struct) + the `#[cfg(test)] mod tests` (new serde tests)
|
||||
- Modify: `crates/aura-registry/src/compat.rs:26-80` (mirror struct + destructure + literal — lockstep)
|
||||
- Modify: every in-workspace `RunManifest { .. }` literal (enumerated below)
|
||||
|
||||
> **RED-shape note for the implementer:** a serde-widening struct field is a
|
||||
> *compile-gated additive* change — the back-compat test that asserts
|
||||
> `m.instrument == None` / round-trips `Some` cannot exist until the field does,
|
||||
> so the field and its pin land in one diff (the #144 `selection`-field
|
||||
> precedent). The "RED" is the field's compile-absence, not a separate failing
|
||||
> commit. Do not block as spec-ambiguous: add the field, thread the literals,
|
||||
> add the tests, verify green.
|
||||
|
||||
- [ ] **Step 1: Add the `instrument` field to `RunManifest`**
|
||||
|
||||
In `crates/aura-engine/src/report.rs`, after the `selection` field (`:480-481`),
|
||||
add:
|
||||
|
||||
```rust
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub selection: Option<FamilySelection>,
|
||||
/// The instrument this run evaluated, when it is a real-data run; `None` for a
|
||||
/// synthetic run and for every pre-0078 line. The "instrument set as lineage"
|
||||
/// (C18) for a cross-instrument family is each member's stamped symbol. Same
|
||||
/// one-directional serde widening as `selection` (C14/C18).
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub instrument: Option<String>,
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Thread the field through the compat mirror (lockstep)**
|
||||
|
||||
In `crates/aura-registry/src/compat.rs`, the read-side mirror `RunManifestRead`
|
||||
(`:27-35`), its destructure (`:67`), and the `From<RunReportRead>` literal
|
||||
(`:69-76`) must all gain the field, or the registry will not compile.
|
||||
|
||||
Mirror struct — after `selection` (`:33-34`):
|
||||
```rust
|
||||
#[serde(default)]
|
||||
selection: Option<FamilySelection>,
|
||||
#[serde(default)]
|
||||
instrument: Option<String>,
|
||||
}
|
||||
```
|
||||
Destructure (`:67`):
|
||||
```rust
|
||||
let RunManifestRead { commit, params, window, seed, broker, selection, instrument } = r.manifest;
|
||||
```
|
||||
Literal (`:69-76`) — add `instrument,` after `selection,`:
|
||||
```rust
|
||||
manifest: RunManifest {
|
||||
commit,
|
||||
params: params.into_iter().map(|(name, v)| (name, v.0)).collect(),
|
||||
window,
|
||||
seed,
|
||||
broker,
|
||||
selection,
|
||||
instrument,
|
||||
},
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Thread `instrument: None` into the production CLI literal**
|
||||
|
||||
In `crates/aura-cli/src/main.rs`, `sim_optimal_manifest` (`:170-177`) — add
|
||||
`instrument: None,` after `selection: None,`:
|
||||
```rust
|
||||
broker: format!("sim-optimal(pip_size={pip_size})"),
|
||||
selection: None,
|
||||
instrument: None,
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Thread `instrument: None` into every remaining in-workspace literal**
|
||||
|
||||
A struct-field addition compile-breaks every `RunManifest { .. }` literal until
|
||||
threaded. There is **no `Default` impl / builder** for `RunManifest` (verified),
|
||||
so each literal must add the field explicitly. Add `instrument: None,` to each of
|
||||
the literals below (the value is always `None` here — the only `Some(...)` stamp
|
||||
is in Task 3's `run_generalize`). Line numbers are pre-edit anchors and will
|
||||
drift as Step 1 shifts `report.rs`; the **authoritative enumerator is the
|
||||
compiler** — run `cargo check --workspace --all-targets` and add the field to any
|
||||
site it flags as missing `instrument`.
|
||||
|
||||
Known in-workspace set (test-`src`, integration tests, examples):
|
||||
- `crates/aura-engine/src/blueprint.rs:942` (test)
|
||||
- `crates/aura-engine/src/mc.rs:239`, `:263` (test)
|
||||
- `crates/aura-engine/src/report.rs:968`, `:1344`, `:1373`, `:1394` (test — the
|
||||
pre-existing test literals; the new tests in Step 5 already include the field)
|
||||
- `crates/aura-engine/src/sweep.rs:664` (test)
|
||||
- `crates/aura-engine/src/walkforward.rs:288` (test)
|
||||
- `crates/aura-engine/tests/random_sweep_e2e.rs:112` (integration test)
|
||||
- `crates/aura-registry/src/lib.rs:474` (the `report_with` fixture — threading it
|
||||
here covers every registry test that builds a report via `report_with`/
|
||||
`report_with_r`), `:813` (test)
|
||||
- `crates/aura-registry/src/trace_store.rs:250` (test)
|
||||
- `crates/aura-ingest/tests/ger40_breakout_world.rs:65`, `real_bars.rs:82`,
|
||||
`streaming_seam.rs:82` (integration tests)
|
||||
- `crates/aura-ingest/examples/ger40_breakout_sweep.rs:63`,
|
||||
`ger40_breakout_walkforward.rs:73`, `shared/breakout_real.rs:390` (examples)
|
||||
|
||||
Plus the two pre-existing `report.rs` test literals at the top of the test module
|
||||
(`runmanifest_without_selection_serialises_without_the_key` `:771`,
|
||||
`family_selection_round_trips_on_the_manifest` `:787`) — add `instrument: None,`
|
||||
after their `selection:` field.
|
||||
|
||||
- [ ] **Step 5: Add the back-compat serde tests**
|
||||
|
||||
In `crates/aura-engine/src/report.rs`, inside `#[cfg(test)] mod tests` (beside the
|
||||
`runmanifest_without_selection_*` tests near `:762`), add:
|
||||
|
||||
```rust
|
||||
#[test]
|
||||
fn runmanifest_without_instrument_field_deserialises_to_none() {
|
||||
// A pre-0078 line carries no `instrument` key; serde `default` -> None.
|
||||
let json = r#"{"commit":"c","params":[],"window":[0,0],"seed":0,"broker":"b"}"#;
|
||||
let m: RunManifest = serde_json::from_str(json).expect("legacy manifest loads");
|
||||
assert!(m.instrument.is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn runmanifest_instrument_round_trips_and_omits_when_none() {
|
||||
let with = RunManifest {
|
||||
commit: "c".into(), params: vec![], window: (Timestamp(0), Timestamp(0)),
|
||||
seed: 0, broker: "b".into(), selection: None, instrument: Some("GER40".into()),
|
||||
};
|
||||
let json = serde_json::to_string(&with).unwrap();
|
||||
assert!(json.contains("\"instrument\":\"GER40\""), "json: {json}");
|
||||
let back: RunManifest = serde_json::from_str(&json).unwrap();
|
||||
assert_eq!(back.instrument, Some("GER40".to_string()));
|
||||
|
||||
let without = RunManifest {
|
||||
commit: "c".into(), params: vec![], window: (Timestamp(0), Timestamp(0)),
|
||||
seed: 0, broker: "b".into(), selection: None, instrument: None,
|
||||
};
|
||||
// skip_serializing_if omits the key when None -> existing manifest bytes unchanged (C23).
|
||||
assert!(!serde_json::to_string(&without).unwrap().contains("instrument"));
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 6: Verify the new tests pass and the workspace compiles**
|
||||
|
||||
Run: `cargo test -p aura-engine runmanifest_`
|
||||
Expected: PASS — `runmanifest_without_instrument_field_deserialises_to_none`,
|
||||
`runmanifest_instrument_round_trips_and_omits_when_none`, and the pre-existing
|
||||
`runmanifest_without_selection_*` all green (4+ tests run, 0 failed).
|
||||
|
||||
- [ ] **Step 7: Verify every literal is threaded (workspace gate)**
|
||||
|
||||
Run: `cargo check --workspace --all-targets`
|
||||
Expected: clean (0 errors). Any "missing field `instrument`" error names a literal
|
||||
Step 4 missed — add `instrument: None,` there and re-run.
|
||||
|
||||
- [ ] **Step 8: Verify the whole workspace suite stays green**
|
||||
|
||||
Run: `cargo test --workspace`
|
||||
Expected: PASS (0 failed) — the threaded literals compile and every existing test,
|
||||
including the registry compat-load tests and the #144/#145 serde tests, stays
|
||||
green.
|
||||
|
||||
---
|
||||
|
||||
## Task 2: aura-registry — `FamilyKind::CrossInstrument`, the `generalization` reduction, `check_r_metric`, error variants
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/aura-registry/src/lineage.rs:22-27` (`FamilyKind`)
|
||||
- Modify: `crates/aura-registry/src/lib.rs:429-455` (struct + fns + error variants + Display) + `#[cfg(test)] mod tests` (unit tests)
|
||||
|
||||
- [ ] **Step 1: Add the `CrossInstrument` family kind**
|
||||
|
||||
In `crates/aura-registry/src/lineage.rs` (`:22-27`):
|
||||
```rust
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub enum FamilyKind {
|
||||
Sweep,
|
||||
MonteCarlo,
|
||||
WalkForward,
|
||||
CrossInstrument,
|
||||
}
|
||||
```
|
||||
(No exhaustive `match` over `FamilyKind` exists workspace-wide — it serializes via
|
||||
the derive — so this breaks no call site; the `runs families` display picks up
|
||||
`"CrossInstrument"` automatically.)
|
||||
|
||||
- [ ] **Step 2: Add the two `RegistryError` variants + their Display arms**
|
||||
|
||||
In `crates/aura-registry/src/lib.rs`, the enum (`:432-440`):
|
||||
```rust
|
||||
#[derive(Debug)]
|
||||
pub enum RegistryError {
|
||||
/// An I/O error reading or writing the JSONL file.
|
||||
Io(std::io::Error),
|
||||
/// A stored line did not parse as a `RunReport` (1-based line number).
|
||||
Parse { line: usize, source: serde_json::Error },
|
||||
/// `rank_by` was given a metric name it does not know.
|
||||
UnknownMetric(String),
|
||||
/// A known metric that is not R-based — cross-instrument generalization is
|
||||
/// R-only (R is the only account/instrument-agnostic unit, C10).
|
||||
NonRMetric(String),
|
||||
/// A generalization was asked for with fewer than two instruments.
|
||||
TooFewInstruments(usize),
|
||||
}
|
||||
```
|
||||
The Display match (`:444-453`) — add two arms before the closing brace:
|
||||
```rust
|
||||
RegistryError::UnknownMetric(m) => write!(
|
||||
f,
|
||||
"unknown metric '{m}' (known: total_pips, max_drawdown, bias_sign_flips, sqn, sqn_normalized, expectancy_r, net_expectancy_r)"
|
||||
),
|
||||
RegistryError::NonRMetric(m) => write!(
|
||||
f,
|
||||
"metric '{m}' is not comparable across instruments; cross-instrument scoring is R-only (sqn, sqn_normalized, expectancy_r, net_expectancy_r)"
|
||||
),
|
||||
RegistryError::TooFewInstruments(n) => write!(
|
||||
f,
|
||||
"a generalization needs >= 2 instruments, got {n}"
|
||||
),
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Write the failing unit tests for `generalization` + `check_r_metric`**
|
||||
|
||||
In `crates/aura-registry/src/lib.rs`, inside `#[cfg(test)] mod tests` (the module
|
||||
that already defines `report_with`/`report_with_r`), add. These reuse
|
||||
`report_with_r(sqn, expectancy_r, net_expectancy_r)` (sets `metrics.r = Some` with
|
||||
the given `expectancy_r`); the default metric is `expectancy_r`.
|
||||
|
||||
```rust
|
||||
#[test]
|
||||
fn generalization_worst_case_is_the_floor_not_the_mean() {
|
||||
let a = report_with_r(0.0, 0.40, 0.0); // +0.40
|
||||
let b = report_with_r(0.0, -0.20, 0.0); // -0.20 (the floor)
|
||||
let c = report_with_r(0.0, 0.10, 0.0); // +0.10
|
||||
let pairs = vec![("A".to_string(), &a), ("B".to_string(), &b), ("C".to_string(), &c)];
|
||||
let g = generalization(&pairs, "expectancy_r").expect("R metric");
|
||||
assert_eq!(g.n_instruments, 3);
|
||||
assert!((g.worst_case - (-0.20)).abs() < 1e-12, "worst_case must be the min, got {}", g.worst_case);
|
||||
assert_eq!(g.sign_agreement, 2, "two instruments are net-positive");
|
||||
assert_eq!(g.per_instrument, vec![
|
||||
("A".to_string(), 0.40), ("B".to_string(), -0.20), ("C".to_string(), 0.10),
|
||||
]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn generalization_is_not_lifted_by_a_strong_instrument() {
|
||||
let a = report_with_r(0.0, 0.40, 0.0);
|
||||
let b = report_with_r(0.0, -0.20, 0.0);
|
||||
let strong = report_with_r(0.0, 99.0, 0.0);
|
||||
let pairs = vec![("A".to_string(), &a), ("B".to_string(), &b), ("S".to_string(), &strong)];
|
||||
let g = generalization(&pairs, "expectancy_r").expect("R metric");
|
||||
assert!((g.worst_case - (-0.20)).abs() < 1e-12, "a strong instrument must not raise the floor");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn generalization_refuses_a_non_r_metric() {
|
||||
let a = report_with_r(0.0, 0.4, 0.0);
|
||||
let b = report_with_r(0.0, 0.2, 0.0);
|
||||
let pairs = vec![("A".to_string(), &a), ("B".to_string(), &b)];
|
||||
match generalization(&pairs, "total_pips") {
|
||||
Err(RegistryError::NonRMetric(m)) => assert_eq!(m, "total_pips"),
|
||||
other => panic!("expected NonRMetric, got {other:?}"),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn generalization_refuses_fewer_than_two_instruments() {
|
||||
let a = report_with_r(0.0, 0.4, 0.0);
|
||||
let pairs = vec![("A".to_string(), &a)];
|
||||
match generalization(&pairs, "expectancy_r") {
|
||||
Err(RegistryError::TooFewInstruments(n)) => assert_eq!(n, 1),
|
||||
other => panic!("expected TooFewInstruments(1), got {other:?}"),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn generalization_is_deterministic() {
|
||||
let a = report_with_r(0.0, 0.40, 0.0);
|
||||
let b = report_with_r(0.0, -0.20, 0.0);
|
||||
let pairs = vec![("A".to_string(), &a), ("B".to_string(), &b)];
|
||||
let g1 = generalization(&pairs, "expectancy_r").expect("R metric");
|
||||
let g2 = generalization(&pairs, "expectancy_r").expect("R metric");
|
||||
assert_eq!(g1, g2, "a pure fold must be deterministic (C1)");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn check_r_metric_accepts_r_and_refuses_pip() {
|
||||
assert!(check_r_metric("expectancy_r").is_ok());
|
||||
assert!(check_r_metric("sqn_normalized").is_ok());
|
||||
match check_r_metric("total_pips") {
|
||||
Err(RegistryError::NonRMetric(m)) => assert_eq!(m, "total_pips"),
|
||||
other => panic!("expected NonRMetric, got {other:?}"),
|
||||
}
|
||||
match check_r_metric("nope") {
|
||||
Err(RegistryError::UnknownMetric(m)) => assert_eq!(m, "nope"),
|
||||
other => panic!("expected UnknownMetric, got {other:?}"),
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run the new tests to verify they fail**
|
||||
|
||||
Run: `cargo test -p aura-registry generalization`
|
||||
Expected: FAIL to compile — `cannot find function `generalization` / `check_r_metric``
|
||||
and `cannot find type `Generalization`` (the symbols do not exist yet). This is the
|
||||
RED state (compile-absence).
|
||||
|
||||
- [ ] **Step 5: Implement `Generalization` + `generalization` + `check_r_metric`**
|
||||
|
||||
In `crates/aura-registry/src/lib.rs`, after `optimize_deflated` ends (`:429`), add:
|
||||
|
||||
```rust
|
||||
/// The cross-instrument generalization aggregate (#146): a brought candidate's
|
||||
/// per-instrument R-metric, reduced to a worst-case floor + a sign-agreement count
|
||||
/// + the full breakdown. R-only (C10): `total_pips` is not comparable across
|
||||
/// instruments. Pure, deterministic (C1) — a fold over `per_instrument` in input
|
||||
/// order. `worst_case` is `min_i metric_i`; since every R-metric is
|
||||
/// higher-is-better, the min is unconditionally the conservative floor (no
|
||||
/// direction branch, unlike `PlateauMode::Worst`).
|
||||
#[derive(Clone, Debug, PartialEq)]
|
||||
pub struct Generalization {
|
||||
pub selection_metric: String,
|
||||
pub n_instruments: usize,
|
||||
pub worst_case: f64,
|
||||
pub sign_agreement: usize,
|
||||
pub per_instrument: Vec<(String, f64)>,
|
||||
}
|
||||
|
||||
/// Validate that `metric` is a known, R-based ranking key — the data-free pre-check
|
||||
/// the CLI runs before evaluating any instrument. The registry owns metric truth
|
||||
/// (C9), so the CLI never duplicates the R-set.
|
||||
pub fn check_r_metric(metric: &str) -> Result<(), RegistryError> {
|
||||
let m = resolve_metric(metric)?;
|
||||
if is_r_metric(m) {
|
||||
Ok(())
|
||||
} else {
|
||||
Err(RegistryError::NonRMetric(metric.to_string()))
|
||||
}
|
||||
}
|
||||
|
||||
/// Grade how consistently one candidate holds across instruments: read the chosen
|
||||
/// R-metric from each per-instrument report and reduce to the worst-case floor +
|
||||
/// the sign-agreement count + the per-instrument breakdown. R-only and `>= 2`
|
||||
/// instruments (the metric check precedes the arity check so a bad metric is
|
||||
/// reported even with an empty slice). Additive — reads, never re-ranks (C23).
|
||||
pub fn generalization(
|
||||
per_instrument: &[(String, &RunReport)],
|
||||
metric: &str,
|
||||
) -> Result<Generalization, RegistryError> {
|
||||
let m = resolve_metric(metric)?;
|
||||
if !is_r_metric(m) {
|
||||
return Err(RegistryError::NonRMetric(metric.to_string()));
|
||||
}
|
||||
if per_instrument.len() < 2 {
|
||||
return Err(RegistryError::TooFewInstruments(per_instrument.len()));
|
||||
}
|
||||
let vals: Vec<(String, f64)> = per_instrument
|
||||
.iter()
|
||||
.map(|(label, rep)| (label.clone(), metric_value(rep, m)))
|
||||
.collect();
|
||||
let worst_case = vals.iter().map(|(_, v)| *v).fold(f64::INFINITY, f64::min);
|
||||
let sign_agreement = vals.iter().filter(|(_, v)| *v > 0.0).count();
|
||||
Ok(Generalization {
|
||||
selection_metric: metric.to_string(),
|
||||
n_instruments: vals.len(),
|
||||
worst_case,
|
||||
sign_agreement,
|
||||
per_instrument: vals,
|
||||
})
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 6: Run the new tests to verify they pass**
|
||||
|
||||
Run: `cargo test -p aura-registry generalization`
|
||||
Expected: PASS — the five `generalization_*` tests green.
|
||||
|
||||
Run: `cargo test -p aura-registry check_r_metric`
|
||||
Expected: PASS — `check_r_metric_accepts_r_and_refuses_pip` green.
|
||||
|
||||
- [ ] **Step 7: Verify the registry suite stays green (preservation gate)**
|
||||
|
||||
Run: `cargo test -p aura-registry`
|
||||
Expected: PASS (0 failed) — the #144/#145 selector tests and the compat-load tests
|
||||
all stay green.
|
||||
|
||||
---
|
||||
|
||||
## Task 3: aura-cli — the `generalize` subcommand
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/aura-cli/src/main.rs` — imports (`:26-30`), new fns, dispatch arm (`:2993`), `USAGE` (`:2956`), parser unit tests
|
||||
- Test: `crates/aura-cli/tests/cli_run.rs` — E2Es
|
||||
|
||||
- [ ] **Step 1: Import the new registry items**
|
||||
|
||||
In `crates/aura-cli/src/main.rs`, extend the `use aura_registry::{ ... }` block
|
||||
(`:26-30`) to add `generalization, Generalization, check_r_metric`:
|
||||
```rust
|
||||
use aura_registry::{
|
||||
check_r_metric, generalization, group_families, mc_member_reports, optimize_deflated,
|
||||
optimize_plateau, rank_by, sweep_member_reports, walkforward_member_reports, FamilyKind,
|
||||
FamilyMember, Generalization, NameKind, PlateauMode, Registry, RunTraces, TraceStore, WriteKind,
|
||||
};
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Write the failing parser unit tests**
|
||||
|
||||
In `crates/aura-cli/src/main.rs`, inside `#[cfg(test)] mod tests` (where
|
||||
`parse_sweep_args` tests live), add:
|
||||
|
||||
```rust
|
||||
#[test]
|
||||
fn parse_generalize_requires_a_single_value_candidate() {
|
||||
// a candidate is one cell: a multi-value grid flag is refused
|
||||
assert!(parse_generalize_args(&[
|
||||
"--real", "GER40,USDJPY", "--fast", "2,3", "--slow", "12",
|
||||
"--stop-length", "14", "--stop-k", "2.0",
|
||||
]).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parse_generalize_requires_all_four_knobs() {
|
||||
// --slow absent -> refuse (no defaulting to the multi-value sweep grid)
|
||||
assert!(parse_generalize_args(&[
|
||||
"--real", "GER40,USDJPY", "--fast", "3",
|
||||
"--stop-length", "14", "--stop-k", "2.0",
|
||||
]).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parse_generalize_refuses_fewer_than_two_instruments() {
|
||||
assert!(parse_generalize_args(&[
|
||||
"--real", "GER40", "--fast", "3", "--slow", "12",
|
||||
"--stop-length", "14", "--stop-k", "2.0",
|
||||
]).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parse_generalize_refuses_a_duplicate_instrument() {
|
||||
assert!(parse_generalize_args(&[
|
||||
"--real", "GER40,GER40", "--fast", "3", "--slow", "12",
|
||||
"--stop-length", "14", "--stop-k", "2.0",
|
||||
]).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parse_generalize_refuses_a_non_stage1r_strategy() {
|
||||
assert!(parse_generalize_args(&[
|
||||
"--strategy", "sma", "--real", "GER40,USDJPY", "--fast", "3", "--slow", "12",
|
||||
"--stop-length", "14", "--stop-k", "2.0",
|
||||
]).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parse_generalize_defaults_name_and_metric() {
|
||||
let (name, symbols, grid, metric, _from, _to) = parse_generalize_args(&[
|
||||
"--real", "GER40,USDJPY", "--fast", "3", "--slow", "12",
|
||||
"--stop-length", "14", "--stop-k", "2.0",
|
||||
]).expect("valid generalize args");
|
||||
assert_eq!(name, "generalize");
|
||||
assert_eq!(metric, "expectancy_r");
|
||||
assert_eq!(symbols, vec!["GER40".to_string(), "USDJPY".to_string()]);
|
||||
assert_eq!(grid.fast, vec![3]);
|
||||
assert_eq!(grid.slow, vec![12]);
|
||||
assert_eq!(grid.stop_length, vec![14]);
|
||||
assert_eq!(grid.stop_k, vec![2.0]);
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Run the parser tests to verify they fail**
|
||||
|
||||
Run: `cargo test -p aura-cli parse_generalize`
|
||||
Expected: FAIL to compile — `cannot find function `parse_generalize_args``
|
||||
(does not exist yet). RED state.
|
||||
|
||||
- [ ] **Step 4: Implement `parse_generalize_args`**
|
||||
|
||||
In `crates/aura-cli/src/main.rs`, near `parse_sweep_args` (`:1622`), add. It mirrors
|
||||
`parse_sweep_args` but: defaults strategy to stage1-r and refuses any other;
|
||||
`--real` is a comma-list of `>= 2` distinct non-empty symbols; the four grid knobs
|
||||
are each required and single-valued; `--metric` defaults to `expectancy_r`;
|
||||
`--name` defaults to `generalize`.
|
||||
|
||||
```rust
|
||||
/// Parse the `generalize` tail:
|
||||
/// `[--strategy stage1-r] --real <SYM1,SYM2,...> --fast <n> --slow <n> --stop-length <n>
|
||||
/// --stop-k <f> [--from <ms>] [--to <ms>] [--metric <r-metric>] [--name <n>]`.
|
||||
/// The candidate is a single cell: each grid knob takes exactly one value and all
|
||||
/// four are required. `--real` is a comma list of >= 2 distinct non-empty symbols.
|
||||
/// Pure (no I/O / exit) so the grammar is unit-testable.
|
||||
fn parse_generalize_args(
|
||||
rest: &[&str],
|
||||
) -> Result<(String, Vec<String>, Stage1RGrid, String, Option<i64>, Option<i64>), String> {
|
||||
let usage = || "generalize [--strategy stage1-r] --real <SYM1,SYM2,...> --fast <n> --slow <n> --stop-length <n> --stop-k <f> [--from <ms>] [--to <ms>] [--metric <r-metric>] [--name <n>]".to_string();
|
||||
let mut symbols: Option<Vec<String>> = None;
|
||||
let mut from_ms: Option<i64> = None;
|
||||
let mut to_ms: Option<i64> = None;
|
||||
let mut fast: Option<i64> = None;
|
||||
let mut slow: Option<i64> = None;
|
||||
let mut stop_length: Option<i64> = None;
|
||||
let mut stop_k: Option<f64> = None;
|
||||
let mut metric = "expectancy_r".to_string();
|
||||
let mut name = "generalize".to_string();
|
||||
// a single grid knob, required and single-valued (csv with exactly one item)
|
||||
let one = |value: &str, usage: &dyn Fn() -> String| -> Result<Vec<&str>, String> {
|
||||
let items: Vec<&str> = value.split(',').collect();
|
||||
if items.len() != 1 || items[0].is_empty() { return Err(usage()); }
|
||||
Ok(items)
|
||||
};
|
||||
let mut tail = rest;
|
||||
while let Some((flag, t)) = tail.split_first() {
|
||||
let (value, t) = t.split_first().ok_or_else(usage)?;
|
||||
match *flag {
|
||||
"--strategy" => { if *value != "stage1-r" { return Err(usage()); } }
|
||||
"--real" => {
|
||||
let parts: Vec<String> = value.split(',').map(|s| s.to_string()).collect();
|
||||
if parts.iter().any(|s| s.is_empty()) { return Err(usage()); }
|
||||
symbols = Some(parts);
|
||||
}
|
||||
"--from" => from_ms = Some(value.parse().map_err(|_| usage())?),
|
||||
"--to" => to_ms = Some(value.parse().map_err(|_| usage())?),
|
||||
"--fast" => { one(value, &usage)?; fast = Some(value.parse().map_err(|_| usage())?); }
|
||||
"--slow" => { one(value, &usage)?; slow = Some(value.parse().map_err(|_| usage())?); }
|
||||
"--stop-length" => { one(value, &usage)?; stop_length = Some(value.parse().map_err(|_| usage())?); }
|
||||
"--stop-k" => { one(value, &usage)?; stop_k = Some(value.parse().map_err(|_| usage())?); }
|
||||
"--metric" => metric = (*value).to_string(),
|
||||
"--name" => name = (*value).to_string(),
|
||||
_ => return Err(usage()),
|
||||
}
|
||||
tail = t;
|
||||
}
|
||||
let symbols = symbols.ok_or_else(usage)?;
|
||||
if symbols.len() < 2 { return Err(usage()); }
|
||||
// distinct: a duplicate would double-count one instrument in the floor / sign count
|
||||
let mut seen = std::collections::HashSet::new();
|
||||
if !symbols.iter().all(|s| seen.insert(s.clone())) { return Err(usage()); }
|
||||
let grid = Stage1RGrid {
|
||||
fast: vec![fast.ok_or_else(usage)?],
|
||||
slow: vec![slow.ok_or_else(usage)?],
|
||||
stop_length: vec![stop_length.ok_or_else(usage)?],
|
||||
stop_k: vec![stop_k.ok_or_else(usage)?],
|
||||
..Stage1RGrid::default()
|
||||
};
|
||||
Ok((name, symbols, grid, metric, from_ms, to_ms))
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Run the parser tests to verify they pass**
|
||||
|
||||
Run: `cargo test -p aura-cli parse_generalize`
|
||||
Expected: PASS — all six `parse_generalize_*` tests green.
|
||||
|
||||
- [ ] **Step 6: Implement `generalize_json` and `run_generalize`**
|
||||
|
||||
In `crates/aura-cli/src/main.rs`, near `walkforward_summary_json` (`:1969`), add the
|
||||
JSON builder (mirrors the `{"walkforward": obj}` shape; renders `selection_metric`
|
||||
as `"metric"`):
|
||||
|
||||
```rust
|
||||
fn generalize_json(agg: &Generalization) -> String {
|
||||
let per: Vec<serde_json::Value> = agg
|
||||
.per_instrument
|
||||
.iter()
|
||||
.map(|(sym, v)| serde_json::json!([sym, v]))
|
||||
.collect();
|
||||
let obj = serde_json::json!({
|
||||
"metric": agg.selection_metric,
|
||||
"n_instruments": agg.n_instruments,
|
||||
"worst_case": agg.worst_case,
|
||||
"sign_agreement": agg.sign_agreement,
|
||||
"per_instrument": per,
|
||||
});
|
||||
serde_json::json!({ "generalize": obj }).to_string()
|
||||
}
|
||||
```
|
||||
|
||||
And the run handler (near `run_sweep`, `:1750`). It pre-checks the metric
|
||||
(data-free refusal), runs the single-cell candidate per instrument, stamps the
|
||||
instrument, reduces, prints, and persists a `CrossInstrument` family:
|
||||
|
||||
```rust
|
||||
fn run_generalize(
|
||||
name: &str,
|
||||
symbols: &[String],
|
||||
grid: &Stage1RGrid,
|
||||
metric: &str,
|
||||
from_ms: Option<i64>,
|
||||
to_ms: Option<i64>,
|
||||
) {
|
||||
// data-free metric pre-check: refuse a non-R / unknown metric before any run.
|
||||
if let Err(e) = check_r_metric(metric) {
|
||||
eprintln!("aura: {e}");
|
||||
std::process::exit(2);
|
||||
}
|
||||
let mut members: Vec<RunReport> = Vec::new();
|
||||
for symbol in symbols {
|
||||
let choice = DataChoice::Real { symbol: symbol.clone(), from_ms, to_ms };
|
||||
let data = DataSource::from_choice(choice); // per-instrument pip_or_refuse + has_symbol
|
||||
let family = stage1_r_sweep_family(None, &data, grid); // single-cell grid -> 1 member
|
||||
let mut report = family.points[0].report.clone();
|
||||
report.manifest.instrument = Some(symbol.clone());
|
||||
members.push(report);
|
||||
}
|
||||
let pairs: Vec<(String, &RunReport)> = symbols.iter().cloned().zip(members.iter()).collect();
|
||||
let agg = match generalization(&pairs, metric) {
|
||||
Ok(a) => a,
|
||||
Err(e) => { eprintln!("aura: {e}"); std::process::exit(2); }
|
||||
};
|
||||
println!("{}", generalize_json(&agg));
|
||||
let reg = default_registry();
|
||||
match reg.append_family(name, FamilyKind::CrossInstrument, &members) {
|
||||
Ok(id) => println!("{{\"family_id\":\"{id}\"}}"),
|
||||
Err(e) => { eprintln!("aura: failed to persist family: {e}"); std::process::exit(1); }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 7: Wire the dispatch arm and extend USAGE**
|
||||
|
||||
In `crates/aura-cli/src/main.rs`, add a `["generalize", rest @ ..]` arm beside
|
||||
`["sweep", ..]` (`:2993`):
|
||||
```rust
|
||||
["generalize", rest @ ..] => match parse_generalize_args(rest) {
|
||||
Ok((name, symbols, grid, metric, from_ms, to_ms)) => {
|
||||
run_generalize(&name, &symbols, &grid, &metric, from_ms, to_ms)
|
||||
}
|
||||
Err(msg) => {
|
||||
eprintln!("aura: {msg}");
|
||||
std::process::exit(2);
|
||||
}
|
||||
},
|
||||
```
|
||||
Append the form to the `USAGE` const (`:2956`), before the closing `aura runs ...`:
|
||||
```
|
||||
| aura generalize [--strategy stage1-r] --real <SYM1,SYM2,...> --fast <n> --slow <n> --stop-length <n> --stop-k <f> [--from <ms>] [--to <ms>] [--metric <r-metric>] [--name <n>]
|
||||
```
|
||||
|
||||
- [ ] **Step 8: Write the E2E tests**
|
||||
|
||||
In `crates/aura-cli/tests/cli_run.rs`, add (model the gated-skip on
|
||||
`run_real_sidecar_index_pip_reaches_the_emitted_manifest`, `:250`):
|
||||
|
||||
```rust
|
||||
/// Property: `aura generalize` grades a single stage1-r candidate across two
|
||||
/// instruments — its stdout carries the per-instrument breakdown + the worst-case
|
||||
/// floor + the sign-agreement, then the persisted CrossInstrument family handle.
|
||||
/// Gated on local GER40/USDJPY data (the shared Sept-2024 window), skips cleanly
|
||||
/// when the archive is absent.
|
||||
#[test]
|
||||
fn generalize_grades_a_candidate_across_two_instruments() {
|
||||
const FROM_MS: &str = "1725148800000";
|
||||
const TO_MS: &str = "1727740799999";
|
||||
let cwd = temp_cwd("generalize-two");
|
||||
let out = std::process::Command::new(env!("CARGO_BIN_EXE_aura"))
|
||||
.args([
|
||||
"generalize", "--strategy", "stage1-r", "--real", "GER40,USDJPY",
|
||||
"--fast", "3", "--slow", "12", "--stop-length", "14", "--stop-k", "2.0",
|
||||
"--from", FROM_MS, "--to", TO_MS,
|
||||
])
|
||||
.current_dir(&cwd)
|
||||
.output()
|
||||
.expect("spawn aura");
|
||||
if out.status.code() == Some(2) {
|
||||
let stderr = String::from_utf8_lossy(&out.stderr);
|
||||
assert!(
|
||||
stderr.contains("no local data") || stderr.contains("no recorded geometry"),
|
||||
"exit 2 must be a data refusal, got: {stderr}"
|
||||
);
|
||||
eprintln!("skip: no local GER40/USDJPY data");
|
||||
return;
|
||||
}
|
||||
assert_eq!(out.status.code(), Some(0), "exit: {:?}", out.status);
|
||||
let stdout = String::from_utf8(out.stdout).expect("utf-8 stdout");
|
||||
assert!(stdout.contains("\"generalize\":"), "aggregate object: {stdout}");
|
||||
assert!(stdout.contains("\"n_instruments\":2"), "two instruments: {stdout}");
|
||||
assert!(stdout.contains("\"worst_case\":"), "worst_case present: {stdout}");
|
||||
assert!(stdout.contains("\"sign_agreement\":"), "sign_agreement present: {stdout}");
|
||||
assert!(stdout.contains("GER40") && stdout.contains("USDJPY"), "per-instrument breakdown: {stdout}");
|
||||
assert!(stdout.contains("\"family_id\":\"generalize-0\""), "family handle: {stdout}");
|
||||
}
|
||||
|
||||
/// Property: a single-instrument generalize is refused at parse time (exit 2),
|
||||
/// before any data access — so it asserts the arity refusal on any machine.
|
||||
#[test]
|
||||
fn generalize_refuses_a_single_instrument() {
|
||||
let out = std::process::Command::new(env!("CARGO_BIN_EXE_aura"))
|
||||
.args([
|
||||
"generalize", "--strategy", "stage1-r", "--real", "GER40",
|
||||
"--fast", "3", "--slow", "12", "--stop-length", "14", "--stop-k", "2.0",
|
||||
])
|
||||
.output()
|
||||
.expect("spawn aura");
|
||||
assert_eq!(out.status.code(), Some(2), "single instrument must exit 2");
|
||||
}
|
||||
|
||||
/// Property: a non-R metric is refused before any instrument runs (the data-free
|
||||
/// `check_r_metric` pre-check), so this asserts the R-only refusal on any machine.
|
||||
#[test]
|
||||
fn generalize_refuses_a_non_r_metric() {
|
||||
let out = std::process::Command::new(env!("CARGO_BIN_EXE_aura"))
|
||||
.args([
|
||||
"generalize", "--strategy", "stage1-r", "--real", "GER40,USDJPY",
|
||||
"--fast", "3", "--slow", "12", "--stop-length", "14", "--stop-k", "2.0",
|
||||
"--metric", "total_pips",
|
||||
])
|
||||
.output()
|
||||
.expect("spawn aura");
|
||||
assert_eq!(out.status.code(), Some(2), "a non-R metric must exit 2");
|
||||
let stderr = String::from_utf8_lossy(&out.stderr);
|
||||
assert!(stderr.contains("R-only"), "must name the R-only refusal, got: {stderr}");
|
||||
}
|
||||
```
|
||||
|
||||
> `temp_cwd(name) -> PathBuf` (cli_run.rs:13) is the confirmed registry-isolation
|
||||
> helper (a fresh per-test cwd, so the family-store run index starts at 0 →
|
||||
> `generalize-0`); `env!("CARGO_BIN_EXE_aura")` is the binary handle (the same the
|
||||
> real-data E2Es at cli_run.rs:256 use). Both already exist in the file.
|
||||
|
||||
- [ ] **Step 9: Run the E2E tests**
|
||||
|
||||
Run: `cargo test -p aura-cli generalize`
|
||||
Expected: PASS — `generalize_refuses_a_single_instrument` and
|
||||
`generalize_refuses_a_non_r_metric` assert exit 2 (data-free);
|
||||
`generalize_grades_a_candidate_across_two_instruments` passes on a machine with
|
||||
GER40/USDJPY data or skips cleanly otherwise; plus the six `parse_generalize_*`
|
||||
unit tests green.
|
||||
|
||||
- [ ] **Step 10: Verify the CLI suite stays green (C23 preservation)**
|
||||
|
||||
Run: `cargo test -p aura-cli`
|
||||
Expected: PASS (0 failed) — every existing sweep/walkforward/mc/run/`runs` test
|
||||
stays green; the existing manifest goldens are byte-identical (instrument is
|
||||
omitted when None — `skip_serializing_if`), so no sweep/walkforward/standalone
|
||||
output changed.
|
||||
|
||||
- [ ] **Step 11: Final workspace gate**
|
||||
|
||||
Run: `cargo test --workspace`
|
||||
Expected: PASS (0 failed).
|
||||
|
||||
Run: `cargo clippy --workspace --all-targets -- -D warnings`
|
||||
Expected: clean (0 warnings).
|
||||
Reference in New Issue
Block a user