audit(0078): cycle close — drift addressed; ledger inferential-thread synced (milestone complete) + C12/C18 realization; drop ephemera
Architect drift review (d50a482..feat): drift_found — documentation only, NO contract violation. C1/C9/C10/C23 verified preserved against the diff: `generalization` is a pure deterministic fold (registry-side, C9; R-only, C10); the sole `instrument = Some` production site is `run_generalize`, every other manifest constructor threads `None` + `skip_serializing_if`, so sweep/mc/walkforward/standalone manifest bytes are byte-identical (C23); the compat read-mirror was threaded in lockstep. Drift fixed in this commit (ledger sync, docs/design/INDEX.md): - The inferential-honesty thread read "now two of three pieces built" / "#146 remains the last open piece" — false now that #146 landed. Updated to record the inferential half as structurally complete (all three built, cycle 0078) + the #146 realization, including the aggregator/validator-vs-selector distinction (#146 is a recomputable aggregate, not a FamilySelection annotation). - C18 realization block: added the cycle-0078 note — the comparison axis (C12) realized as FamilyKind::CrossInstrument + the first-class RunManifest.instrument lineage field. Regression: no regression scripts configured (project facts name only Build/Test/Lint/Doc); architect is the gate. cargo test --workspace green (0 failed) and cargo clippy --workspace --all-targets -D warnings clean, orchestrator-verified. Ephemera removed per the project lifecycle convention: docs/specs/0078-* and docs/plans/0078-* (git rm). Durable rationale lifted to the ledger above. refs #146
This commit is contained in:
+26
-4
@@ -929,6 +929,16 @@ record, and a member's window is **producer-supplied** via
|
||||
never a materialized-`Vec` scan at the call site. CLI surface: `aura mc`,
|
||||
`aura runs families`, `aura runs family <id> [rank <metric>]`; `aura sweep` /
|
||||
`walkforward` / `mc` persist via `append_family` with an optional `--name`.
|
||||
**Realization (cycle 0078 — cross-instrument family + instrument lineage, #146).**
|
||||
The comparison axis (C12) is realized as a `FamilyKind::CrossInstrument` family:
|
||||
`aura generalize` runs one candidate across an instrument list and persists the M
|
||||
per-instrument runs via `append_family`, each member self-identifying through a new
|
||||
first-class `RunManifest.instrument` lineage field (serde-widened with
|
||||
`skip_serializing_if`, so legacy lines and every non-cross-instrument path stay
|
||||
byte-identical — C14/C23). The cross-instrument *generalization score* (worst-case R
|
||||
floor + sign-agreement + per-instrument breakdown) is a **recomputable aggregate**
|
||||
over those members, not a persisted family-level record — distinct from #144/#145's
|
||||
per-winner selection annotation on `RunManifest.selection`.
|
||||
Deferred (Non-goals): content-addressed identity + replay-dedup; the "run-diff"
|
||||
depth and ranking families against each other (cross-family, vs. within-family);
|
||||
and a live producer for the flat `runs.jsonl` standalone-run path — no CLI command
|
||||
@@ -1422,8 +1432,8 @@ load-bearing in the flat graph.
|
||||
significance read (#139, landed cycle 0075). Distinct from the two threads above —
|
||||
neither orchestration *composability* (#109) nor a search *policy* (Bayesian/
|
||||
genetic), but the statistical *validity* of the selection itself. Not a
|
||||
C-invariant until built; recorded as the World's third (now two of three pieces
|
||||
built) half.
|
||||
C-invariant until built; recorded as the World's third half — **now structurally
|
||||
complete** (all three pieces built, cycle 0078).
|
||||
**Status (cycle 0076):** the trials-deflation piece (#144) landed —
|
||||
`optimize_deflated` (aura-registry) wraps `optimize` and records, additively on
|
||||
the winning member's manifest (C18, the new `RunManifest.selection`), a deflated
|
||||
@@ -1442,8 +1452,20 @@ load-bearing in the flat graph.
|
||||
byte-identical (C23). The grid lattice surfaces from the engine
|
||||
(`SweepBinder::sweep_with_lattice`); the policy stays in aura-registry with
|
||||
`walk_forward` selection-agnostic (C9). Selection is now a *pluggable objective*
|
||||
(bare argmax → trials-deflated → plateau), not a single hard-wired argmax. #146
|
||||
(cross-instrument generalization) remains the last open piece of the milestone.
|
||||
(bare argmax → trials-deflated → plateau), not a single hard-wired argmax.
|
||||
**Status (cycle 0078):** the last piece — cross-instrument generalization (#146) —
|
||||
landed, completing the inferential half. Unlike #144/#145 (which *select + annotate*
|
||||
a within-family winner on `RunManifest.selection`), #146 is an **aggregator/
|
||||
validator**, not a selector: the `aura generalize` subcommand runs a *brought*
|
||||
candidate across an instrument list and `generalization` (aura-registry) reduces the
|
||||
per-instrument R-metrics to a **worst-case floor** (min over instruments — the
|
||||
cross-instrument sibling of `PlateauMode::Worst`, R-only per C10) + a sign-agreement
|
||||
count + the per-instrument breakdown. It is a *recomputable aggregate* over a new
|
||||
`FamilyKind::CrossInstrument` family (C12's comparison axis, realized), each member
|
||||
self-identifying via the new `RunManifest.instrument` lineage field (C18) — *not*
|
||||
stamped on `FamilySelection` (there is no within-family winner to annotate). The
|
||||
inferential half is now structurally built; the per-candidate OOS bootstrap (#139,
|
||||
cycle 0075) is its adjacent significance read.
|
||||
- **`aura-std` contents** — the crate exists (doc-only); which universal blocks
|
||||
land first follows the walking-skeleton's needs.
|
||||
- **`strategies/` split** — a later split, *inside a project*, of top-level
|
||||
|
||||
@@ -1,789 +0,0 @@
|
||||
# Cross-instrument generalization — Implementation Plan
|
||||
|
||||
> **Parent spec:** `docs/specs/0078-cross-instrument-generalization.md`
|
||||
>
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: use the `implement` skill to run
|
||||
> this plan. Steps use `- [ ]` checkboxes for tracking.
|
||||
|
||||
**Goal:** Add an `aura generalize` subcommand that grades how consistently a
|
||||
single candidate holds across a set of instruments (worst-case R floor +
|
||||
sign-agreement + per-instrument breakdown), as the last anti-false-discovery
|
||||
piece of the Inferential-validation milestone (#146).
|
||||
|
||||
**Architecture:** A first-class `RunManifest.instrument` lineage field
|
||||
(serde-widened, C14/C18) + a new `FamilyKind::CrossInstrument` (C12) + an R-only
|
||||
`generalization` reduction in aura-registry beside `optimize_deflated`/
|
||||
`optimize_plateau` (C9) + a CLI `generalize` subcommand that runs a single-cell
|
||||
stage1-r candidate across an instrument list, stamps each member's instrument,
|
||||
reduces, prints, and persists a CrossInstrument family. Additive — existing
|
||||
sweep/walkforward/mc/standalone paths stay byte-identical (C23).
|
||||
|
||||
**Tech Stack:** aura-engine (`RunManifest`), aura-registry (`FamilyKind`,
|
||||
`RegistryError`, `generalization`/`Generalization`/`check_r_metric`), aura-cli
|
||||
(`generalize` subcommand).
|
||||
|
||||
---
|
||||
|
||||
**Files this plan creates or modifies:**
|
||||
|
||||
- Modify: `crates/aura-engine/src/report.rs:463-482` — add `RunManifest.instrument` field + back-compat serde tests
|
||||
- Modify: `crates/aura-registry/src/compat.rs:26-80` — the `RunManifestRead` mirror + destructure + literal (lockstep with the engine field)
|
||||
- Modify: `crates/aura-registry/src/lineage.rs:22-27` — `FamilyKind::CrossInstrument` variant
|
||||
- Modify: `crates/aura-registry/src/lib.rs:429-455` — `Generalization` struct + `generalization` fn + `check_r_metric` fn + two `RegistryError` variants + Display arms + unit tests
|
||||
- Modify: `crates/aura-cli/src/main.rs` — imports (`:26`), `sim_optimal_manifest` literal (`:170`), `parse_generalize_args` + `run_generalize` + `generalize_json` (new fns near the sweep helpers), dispatch arm (`:2993`), `USAGE` (`:2956`), parser unit tests
|
||||
- Modify (thread `instrument: None`): every in-workspace `RunManifest { .. }` literal — enumerated set in Task 1, compiler-driven by `cargo check --workspace --all-targets`
|
||||
- Test: `crates/aura-cli/tests/cli_run.rs` — gated-skip success E2E + two data-free must-fail E2Es
|
||||
|
||||
**Out of scope (named in the spec):** reading-B cross-instrument *selection*;
|
||||
per-instrument walk-forward; a persisted family-aggregate record; cross-symbol
|
||||
R-pooling (#139 stays deferred); pip-metric generalization. `fieldtests/` is NOT
|
||||
a workspace member (verified: workspace members are the 7 `crates/` only), so its
|
||||
16 `RunManifest` literals are out of the compile-gate.
|
||||
|
||||
---
|
||||
|
||||
## Task 1: `RunManifest.instrument` field + compat mirror + thread all literals + back-compat tests
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/aura-engine/src/report.rs:463-482` (struct) + the `#[cfg(test)] mod tests` (new serde tests)
|
||||
- Modify: `crates/aura-registry/src/compat.rs:26-80` (mirror struct + destructure + literal — lockstep)
|
||||
- Modify: every in-workspace `RunManifest { .. }` literal (enumerated below)
|
||||
|
||||
> **RED-shape note for the implementer:** a serde-widening struct field is a
|
||||
> *compile-gated additive* change — the back-compat test that asserts
|
||||
> `m.instrument == None` / round-trips `Some` cannot exist until the field does,
|
||||
> so the field and its pin land in one diff (the #144 `selection`-field
|
||||
> precedent). The "RED" is the field's compile-absence, not a separate failing
|
||||
> commit. Do not block as spec-ambiguous: add the field, thread the literals,
|
||||
> add the tests, verify green.
|
||||
|
||||
- [ ] **Step 1: Add the `instrument` field to `RunManifest`**
|
||||
|
||||
In `crates/aura-engine/src/report.rs`, after the `selection` field (`:480-481`),
|
||||
add:
|
||||
|
||||
```rust
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub selection: Option<FamilySelection>,
|
||||
/// The instrument this run evaluated, when it is a real-data run; `None` for a
|
||||
/// synthetic run and for every pre-0078 line. The "instrument set as lineage"
|
||||
/// (C18) for a cross-instrument family is each member's stamped symbol. Same
|
||||
/// one-directional serde widening as `selection` (C14/C18).
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub instrument: Option<String>,
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Thread the field through the compat mirror (lockstep)**
|
||||
|
||||
In `crates/aura-registry/src/compat.rs`, the read-side mirror `RunManifestRead`
|
||||
(`:27-35`), its destructure (`:67`), and the `From<RunReportRead>` literal
|
||||
(`:69-76`) must all gain the field, or the registry will not compile.
|
||||
|
||||
Mirror struct — after `selection` (`:33-34`):
|
||||
```rust
|
||||
#[serde(default)]
|
||||
selection: Option<FamilySelection>,
|
||||
#[serde(default)]
|
||||
instrument: Option<String>,
|
||||
}
|
||||
```
|
||||
Destructure (`:67`):
|
||||
```rust
|
||||
let RunManifestRead { commit, params, window, seed, broker, selection, instrument } = r.manifest;
|
||||
```
|
||||
Literal (`:69-76`) — add `instrument,` after `selection,`:
|
||||
```rust
|
||||
manifest: RunManifest {
|
||||
commit,
|
||||
params: params.into_iter().map(|(name, v)| (name, v.0)).collect(),
|
||||
window,
|
||||
seed,
|
||||
broker,
|
||||
selection,
|
||||
instrument,
|
||||
},
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Thread `instrument: None` into the production CLI literal**
|
||||
|
||||
In `crates/aura-cli/src/main.rs`, `sim_optimal_manifest` (`:170-177`) — add
|
||||
`instrument: None,` after `selection: None,`:
|
||||
```rust
|
||||
broker: format!("sim-optimal(pip_size={pip_size})"),
|
||||
selection: None,
|
||||
instrument: None,
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Thread `instrument: None` into every remaining in-workspace literal**
|
||||
|
||||
A struct-field addition compile-breaks every `RunManifest { .. }` literal until
|
||||
threaded. There is **no `Default` impl / builder** for `RunManifest` (verified),
|
||||
so each literal must add the field explicitly. Add `instrument: None,` to each of
|
||||
the literals below (the value is always `None` here — the only `Some(...)` stamp
|
||||
is in Task 3's `run_generalize`). Line numbers are pre-edit anchors and will
|
||||
drift as Step 1 shifts `report.rs`; the **authoritative enumerator is the
|
||||
compiler** — run `cargo check --workspace --all-targets` and add the field to any
|
||||
site it flags as missing `instrument`.
|
||||
|
||||
Known in-workspace set (test-`src`, integration tests, examples):
|
||||
- `crates/aura-engine/src/blueprint.rs:942` (test)
|
||||
- `crates/aura-engine/src/mc.rs:239`, `:263` (test)
|
||||
- `crates/aura-engine/src/report.rs:968`, `:1344`, `:1373`, `:1394` (test — the
|
||||
pre-existing test literals; the new tests in Step 5 already include the field)
|
||||
- `crates/aura-engine/src/sweep.rs:664` (test)
|
||||
- `crates/aura-engine/src/walkforward.rs:288` (test)
|
||||
- `crates/aura-engine/tests/random_sweep_e2e.rs:112` (integration test)
|
||||
- `crates/aura-registry/src/lib.rs:474` (the `report_with` fixture — threading it
|
||||
here covers every registry test that builds a report via `report_with`/
|
||||
`report_with_r`), `:813` (test)
|
||||
- `crates/aura-registry/src/trace_store.rs:250` (test)
|
||||
- `crates/aura-ingest/tests/ger40_breakout_world.rs:65`, `real_bars.rs:82`,
|
||||
`streaming_seam.rs:82` (integration tests)
|
||||
- `crates/aura-ingest/examples/ger40_breakout_sweep.rs:63`,
|
||||
`ger40_breakout_walkforward.rs:73`, `shared/breakout_real.rs:390` (examples)
|
||||
|
||||
Plus the two pre-existing `report.rs` test literals at the top of the test module
|
||||
(`runmanifest_without_selection_serialises_without_the_key` `:771`,
|
||||
`family_selection_round_trips_on_the_manifest` `:787`) — add `instrument: None,`
|
||||
after their `selection:` field.
|
||||
|
||||
- [ ] **Step 5: Add the back-compat serde tests**
|
||||
|
||||
In `crates/aura-engine/src/report.rs`, inside `#[cfg(test)] mod tests` (beside the
|
||||
`runmanifest_without_selection_*` tests near `:762`), add:
|
||||
|
||||
```rust
|
||||
#[test]
|
||||
fn runmanifest_without_instrument_field_deserialises_to_none() {
|
||||
// A pre-0078 line carries no `instrument` key; serde `default` -> None.
|
||||
let json = r#"{"commit":"c","params":[],"window":[0,0],"seed":0,"broker":"b"}"#;
|
||||
let m: RunManifest = serde_json::from_str(json).expect("legacy manifest loads");
|
||||
assert!(m.instrument.is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn runmanifest_instrument_round_trips_and_omits_when_none() {
|
||||
let with = RunManifest {
|
||||
commit: "c".into(), params: vec![], window: (Timestamp(0), Timestamp(0)),
|
||||
seed: 0, broker: "b".into(), selection: None, instrument: Some("GER40".into()),
|
||||
};
|
||||
let json = serde_json::to_string(&with).unwrap();
|
||||
assert!(json.contains("\"instrument\":\"GER40\""), "json: {json}");
|
||||
let back: RunManifest = serde_json::from_str(&json).unwrap();
|
||||
assert_eq!(back.instrument, Some("GER40".to_string()));
|
||||
|
||||
let without = RunManifest {
|
||||
commit: "c".into(), params: vec![], window: (Timestamp(0), Timestamp(0)),
|
||||
seed: 0, broker: "b".into(), selection: None, instrument: None,
|
||||
};
|
||||
// skip_serializing_if omits the key when None -> existing manifest bytes unchanged (C23).
|
||||
assert!(!serde_json::to_string(&without).unwrap().contains("instrument"));
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 6: Verify the new tests pass and the workspace compiles**
|
||||
|
||||
Run: `cargo test -p aura-engine runmanifest_`
|
||||
Expected: PASS — `runmanifest_without_instrument_field_deserialises_to_none`,
|
||||
`runmanifest_instrument_round_trips_and_omits_when_none`, and the pre-existing
|
||||
`runmanifest_without_selection_*` all green (4+ tests run, 0 failed).
|
||||
|
||||
- [ ] **Step 7: Verify every literal is threaded (workspace gate)**
|
||||
|
||||
Run: `cargo check --workspace --all-targets`
|
||||
Expected: clean (0 errors). Any "missing field `instrument`" error names a literal
|
||||
Step 4 missed — add `instrument: None,` there and re-run.
|
||||
|
||||
- [ ] **Step 8: Verify the whole workspace suite stays green**
|
||||
|
||||
Run: `cargo test --workspace`
|
||||
Expected: PASS (0 failed) — the threaded literals compile and every existing test,
|
||||
including the registry compat-load tests and the #144/#145 serde tests, stays
|
||||
green.
|
||||
|
||||
---
|
||||
|
||||
## Task 2: aura-registry — `FamilyKind::CrossInstrument`, the `generalization` reduction, `check_r_metric`, error variants
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/aura-registry/src/lineage.rs:22-27` (`FamilyKind`)
|
||||
- Modify: `crates/aura-registry/src/lib.rs:429-455` (struct + fns + error variants + Display) + `#[cfg(test)] mod tests` (unit tests)
|
||||
|
||||
- [ ] **Step 1: Add the `CrossInstrument` family kind**
|
||||
|
||||
In `crates/aura-registry/src/lineage.rs` (`:22-27`):
|
||||
```rust
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub enum FamilyKind {
|
||||
Sweep,
|
||||
MonteCarlo,
|
||||
WalkForward,
|
||||
CrossInstrument,
|
||||
}
|
||||
```
|
||||
(No exhaustive `match` over `FamilyKind` exists workspace-wide — it serializes via
|
||||
the derive — so this breaks no call site; the `runs families` display picks up
|
||||
`"CrossInstrument"` automatically.)
|
||||
|
||||
- [ ] **Step 2: Add the two `RegistryError` variants + their Display arms**
|
||||
|
||||
In `crates/aura-registry/src/lib.rs`, the enum (`:432-440`):
|
||||
```rust
|
||||
#[derive(Debug)]
|
||||
pub enum RegistryError {
|
||||
/// An I/O error reading or writing the JSONL file.
|
||||
Io(std::io::Error),
|
||||
/// A stored line did not parse as a `RunReport` (1-based line number).
|
||||
Parse { line: usize, source: serde_json::Error },
|
||||
/// `rank_by` was given a metric name it does not know.
|
||||
UnknownMetric(String),
|
||||
/// A known metric that is not R-based — cross-instrument generalization is
|
||||
/// R-only (R is the only account/instrument-agnostic unit, C10).
|
||||
NonRMetric(String),
|
||||
/// A generalization was asked for with fewer than two instruments.
|
||||
TooFewInstruments(usize),
|
||||
}
|
||||
```
|
||||
The Display match (`:444-453`) — add two arms before the closing brace:
|
||||
```rust
|
||||
RegistryError::UnknownMetric(m) => write!(
|
||||
f,
|
||||
"unknown metric '{m}' (known: total_pips, max_drawdown, bias_sign_flips, sqn, sqn_normalized, expectancy_r, net_expectancy_r)"
|
||||
),
|
||||
RegistryError::NonRMetric(m) => write!(
|
||||
f,
|
||||
"metric '{m}' is not comparable across instruments; cross-instrument scoring is R-only (sqn, sqn_normalized, expectancy_r, net_expectancy_r)"
|
||||
),
|
||||
RegistryError::TooFewInstruments(n) => write!(
|
||||
f,
|
||||
"a generalization needs >= 2 instruments, got {n}"
|
||||
),
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Write the failing unit tests for `generalization` + `check_r_metric`**
|
||||
|
||||
In `crates/aura-registry/src/lib.rs`, inside `#[cfg(test)] mod tests` (the module
|
||||
that already defines `report_with`/`report_with_r`), add. These reuse
|
||||
`report_with_r(sqn, expectancy_r, net_expectancy_r)` (sets `metrics.r = Some` with
|
||||
the given `expectancy_r`); the default metric is `expectancy_r`.
|
||||
|
||||
```rust
|
||||
#[test]
|
||||
fn generalization_worst_case_is_the_floor_not_the_mean() {
|
||||
let a = report_with_r(0.0, 0.40, 0.0); // +0.40
|
||||
let b = report_with_r(0.0, -0.20, 0.0); // -0.20 (the floor)
|
||||
let c = report_with_r(0.0, 0.10, 0.0); // +0.10
|
||||
let pairs = vec![("A".to_string(), &a), ("B".to_string(), &b), ("C".to_string(), &c)];
|
||||
let g = generalization(&pairs, "expectancy_r").expect("R metric");
|
||||
assert_eq!(g.n_instruments, 3);
|
||||
assert!((g.worst_case - (-0.20)).abs() < 1e-12, "worst_case must be the min, got {}", g.worst_case);
|
||||
assert_eq!(g.sign_agreement, 2, "two instruments are net-positive");
|
||||
assert_eq!(g.per_instrument, vec![
|
||||
("A".to_string(), 0.40), ("B".to_string(), -0.20), ("C".to_string(), 0.10),
|
||||
]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn generalization_is_not_lifted_by_a_strong_instrument() {
|
||||
let a = report_with_r(0.0, 0.40, 0.0);
|
||||
let b = report_with_r(0.0, -0.20, 0.0);
|
||||
let strong = report_with_r(0.0, 99.0, 0.0);
|
||||
let pairs = vec![("A".to_string(), &a), ("B".to_string(), &b), ("S".to_string(), &strong)];
|
||||
let g = generalization(&pairs, "expectancy_r").expect("R metric");
|
||||
assert!((g.worst_case - (-0.20)).abs() < 1e-12, "a strong instrument must not raise the floor");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn generalization_refuses_a_non_r_metric() {
|
||||
let a = report_with_r(0.0, 0.4, 0.0);
|
||||
let b = report_with_r(0.0, 0.2, 0.0);
|
||||
let pairs = vec![("A".to_string(), &a), ("B".to_string(), &b)];
|
||||
match generalization(&pairs, "total_pips") {
|
||||
Err(RegistryError::NonRMetric(m)) => assert_eq!(m, "total_pips"),
|
||||
other => panic!("expected NonRMetric, got {other:?}"),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn generalization_refuses_fewer_than_two_instruments() {
|
||||
let a = report_with_r(0.0, 0.4, 0.0);
|
||||
let pairs = vec![("A".to_string(), &a)];
|
||||
match generalization(&pairs, "expectancy_r") {
|
||||
Err(RegistryError::TooFewInstruments(n)) => assert_eq!(n, 1),
|
||||
other => panic!("expected TooFewInstruments(1), got {other:?}"),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn generalization_is_deterministic() {
|
||||
let a = report_with_r(0.0, 0.40, 0.0);
|
||||
let b = report_with_r(0.0, -0.20, 0.0);
|
||||
let pairs = vec![("A".to_string(), &a), ("B".to_string(), &b)];
|
||||
let g1 = generalization(&pairs, "expectancy_r").expect("R metric");
|
||||
let g2 = generalization(&pairs, "expectancy_r").expect("R metric");
|
||||
assert_eq!(g1, g2, "a pure fold must be deterministic (C1)");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn check_r_metric_accepts_r_and_refuses_pip() {
|
||||
assert!(check_r_metric("expectancy_r").is_ok());
|
||||
assert!(check_r_metric("sqn_normalized").is_ok());
|
||||
match check_r_metric("total_pips") {
|
||||
Err(RegistryError::NonRMetric(m)) => assert_eq!(m, "total_pips"),
|
||||
other => panic!("expected NonRMetric, got {other:?}"),
|
||||
}
|
||||
match check_r_metric("nope") {
|
||||
Err(RegistryError::UnknownMetric(m)) => assert_eq!(m, "nope"),
|
||||
other => panic!("expected UnknownMetric, got {other:?}"),
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run the new tests to verify they fail**
|
||||
|
||||
Run: `cargo test -p aura-registry generalization`
|
||||
Expected: FAIL to compile — `cannot find function `generalization` / `check_r_metric``
|
||||
and `cannot find type `Generalization`` (the symbols do not exist yet). This is the
|
||||
RED state (compile-absence).
|
||||
|
||||
- [ ] **Step 5: Implement `Generalization` + `generalization` + `check_r_metric`**
|
||||
|
||||
In `crates/aura-registry/src/lib.rs`, after `optimize_deflated` ends (`:429`), add:
|
||||
|
||||
```rust
|
||||
/// The cross-instrument generalization aggregate (#146): a brought candidate's
|
||||
/// per-instrument R-metric, reduced to a worst-case floor + a sign-agreement count
|
||||
/// + the full breakdown. R-only (C10): `total_pips` is not comparable across
|
||||
/// instruments. Pure, deterministic (C1) — a fold over `per_instrument` in input
|
||||
/// order. `worst_case` is `min_i metric_i`; since every R-metric is
|
||||
/// higher-is-better, the min is unconditionally the conservative floor (no
|
||||
/// direction branch, unlike `PlateauMode::Worst`).
|
||||
#[derive(Clone, Debug, PartialEq)]
|
||||
pub struct Generalization {
|
||||
pub selection_metric: String,
|
||||
pub n_instruments: usize,
|
||||
pub worst_case: f64,
|
||||
pub sign_agreement: usize,
|
||||
pub per_instrument: Vec<(String, f64)>,
|
||||
}
|
||||
|
||||
/// Validate that `metric` is a known, R-based ranking key — the data-free pre-check
|
||||
/// the CLI runs before evaluating any instrument. The registry owns metric truth
|
||||
/// (C9), so the CLI never duplicates the R-set.
|
||||
pub fn check_r_metric(metric: &str) -> Result<(), RegistryError> {
|
||||
let m = resolve_metric(metric)?;
|
||||
if is_r_metric(m) {
|
||||
Ok(())
|
||||
} else {
|
||||
Err(RegistryError::NonRMetric(metric.to_string()))
|
||||
}
|
||||
}
|
||||
|
||||
/// Grade how consistently one candidate holds across instruments: read the chosen
|
||||
/// R-metric from each per-instrument report and reduce to the worst-case floor +
|
||||
/// the sign-agreement count + the per-instrument breakdown. R-only and `>= 2`
|
||||
/// instruments (the metric check precedes the arity check so a bad metric is
|
||||
/// reported even with an empty slice). Additive — reads, never re-ranks (C23).
|
||||
pub fn generalization(
|
||||
per_instrument: &[(String, &RunReport)],
|
||||
metric: &str,
|
||||
) -> Result<Generalization, RegistryError> {
|
||||
let m = resolve_metric(metric)?;
|
||||
if !is_r_metric(m) {
|
||||
return Err(RegistryError::NonRMetric(metric.to_string()));
|
||||
}
|
||||
if per_instrument.len() < 2 {
|
||||
return Err(RegistryError::TooFewInstruments(per_instrument.len()));
|
||||
}
|
||||
let vals: Vec<(String, f64)> = per_instrument
|
||||
.iter()
|
||||
.map(|(label, rep)| (label.clone(), metric_value(rep, m)))
|
||||
.collect();
|
||||
let worst_case = vals.iter().map(|(_, v)| *v).fold(f64::INFINITY, f64::min);
|
||||
let sign_agreement = vals.iter().filter(|(_, v)| *v > 0.0).count();
|
||||
Ok(Generalization {
|
||||
selection_metric: metric.to_string(),
|
||||
n_instruments: vals.len(),
|
||||
worst_case,
|
||||
sign_agreement,
|
||||
per_instrument: vals,
|
||||
})
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 6: Run the new tests to verify they pass**
|
||||
|
||||
Run: `cargo test -p aura-registry generalization`
|
||||
Expected: PASS — the five `generalization_*` tests green.
|
||||
|
||||
Run: `cargo test -p aura-registry check_r_metric`
|
||||
Expected: PASS — `check_r_metric_accepts_r_and_refuses_pip` green.
|
||||
|
||||
- [ ] **Step 7: Verify the registry suite stays green (preservation gate)**
|
||||
|
||||
Run: `cargo test -p aura-registry`
|
||||
Expected: PASS (0 failed) — the #144/#145 selector tests and the compat-load tests
|
||||
all stay green.
|
||||
|
||||
---
|
||||
|
||||
## Task 3: aura-cli — the `generalize` subcommand
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/aura-cli/src/main.rs` — imports (`:26-30`), new fns, dispatch arm (`:2993`), `USAGE` (`:2956`), parser unit tests
|
||||
- Test: `crates/aura-cli/tests/cli_run.rs` — E2Es
|
||||
|
||||
- [ ] **Step 1: Import the new registry items**
|
||||
|
||||
In `crates/aura-cli/src/main.rs`, extend the `use aura_registry::{ ... }` block
|
||||
(`:26-30`) to add `generalization, Generalization, check_r_metric`:
|
||||
```rust
|
||||
use aura_registry::{
|
||||
check_r_metric, generalization, group_families, mc_member_reports, optimize_deflated,
|
||||
optimize_plateau, rank_by, sweep_member_reports, walkforward_member_reports, FamilyKind,
|
||||
FamilyMember, Generalization, NameKind, PlateauMode, Registry, RunTraces, TraceStore, WriteKind,
|
||||
};
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Write the failing parser unit tests**
|
||||
|
||||
In `crates/aura-cli/src/main.rs`, inside `#[cfg(test)] mod tests` (where
|
||||
`parse_sweep_args` tests live), add:
|
||||
|
||||
```rust
|
||||
#[test]
|
||||
fn parse_generalize_requires_a_single_value_candidate() {
|
||||
// a candidate is one cell: a multi-value grid flag is refused
|
||||
assert!(parse_generalize_args(&[
|
||||
"--real", "GER40,USDJPY", "--fast", "2,3", "--slow", "12",
|
||||
"--stop-length", "14", "--stop-k", "2.0",
|
||||
]).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parse_generalize_requires_all_four_knobs() {
|
||||
// --slow absent -> refuse (no defaulting to the multi-value sweep grid)
|
||||
assert!(parse_generalize_args(&[
|
||||
"--real", "GER40,USDJPY", "--fast", "3",
|
||||
"--stop-length", "14", "--stop-k", "2.0",
|
||||
]).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parse_generalize_refuses_fewer_than_two_instruments() {
|
||||
assert!(parse_generalize_args(&[
|
||||
"--real", "GER40", "--fast", "3", "--slow", "12",
|
||||
"--stop-length", "14", "--stop-k", "2.0",
|
||||
]).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parse_generalize_refuses_a_duplicate_instrument() {
|
||||
assert!(parse_generalize_args(&[
|
||||
"--real", "GER40,GER40", "--fast", "3", "--slow", "12",
|
||||
"--stop-length", "14", "--stop-k", "2.0",
|
||||
]).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parse_generalize_refuses_a_non_stage1r_strategy() {
|
||||
assert!(parse_generalize_args(&[
|
||||
"--strategy", "sma", "--real", "GER40,USDJPY", "--fast", "3", "--slow", "12",
|
||||
"--stop-length", "14", "--stop-k", "2.0",
|
||||
]).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parse_generalize_defaults_name_and_metric() {
|
||||
let (name, symbols, grid, metric, _from, _to) = parse_generalize_args(&[
|
||||
"--real", "GER40,USDJPY", "--fast", "3", "--slow", "12",
|
||||
"--stop-length", "14", "--stop-k", "2.0",
|
||||
]).expect("valid generalize args");
|
||||
assert_eq!(name, "generalize");
|
||||
assert_eq!(metric, "expectancy_r");
|
||||
assert_eq!(symbols, vec!["GER40".to_string(), "USDJPY".to_string()]);
|
||||
assert_eq!(grid.fast, vec![3]);
|
||||
assert_eq!(grid.slow, vec![12]);
|
||||
assert_eq!(grid.stop_length, vec![14]);
|
||||
assert_eq!(grid.stop_k, vec![2.0]);
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Run the parser tests to verify they fail**
|
||||
|
||||
Run: `cargo test -p aura-cli parse_generalize`
|
||||
Expected: FAIL to compile — `cannot find function `parse_generalize_args``
|
||||
(does not exist yet). RED state.
|
||||
|
||||
- [ ] **Step 4: Implement `parse_generalize_args`**
|
||||
|
||||
In `crates/aura-cli/src/main.rs`, near `parse_sweep_args` (`:1622`), add. It mirrors
|
||||
`parse_sweep_args` but: defaults strategy to stage1-r and refuses any other;
|
||||
`--real` is a comma-list of `>= 2` distinct non-empty symbols; the four grid knobs
|
||||
are each required and single-valued; `--metric` defaults to `expectancy_r`;
|
||||
`--name` defaults to `generalize`.
|
||||
|
||||
```rust
|
||||
/// Parse the `generalize` tail:
|
||||
/// `[--strategy stage1-r] --real <SYM1,SYM2,...> --fast <n> --slow <n> --stop-length <n>
|
||||
/// --stop-k <f> [--from <ms>] [--to <ms>] [--metric <r-metric>] [--name <n>]`.
|
||||
/// The candidate is a single cell: each grid knob takes exactly one value and all
|
||||
/// four are required. `--real` is a comma list of >= 2 distinct non-empty symbols.
|
||||
/// Pure (no I/O / exit) so the grammar is unit-testable.
|
||||
fn parse_generalize_args(
|
||||
rest: &[&str],
|
||||
) -> Result<(String, Vec<String>, Stage1RGrid, String, Option<i64>, Option<i64>), String> {
|
||||
let usage = || "generalize [--strategy stage1-r] --real <SYM1,SYM2,...> --fast <n> --slow <n> --stop-length <n> --stop-k <f> [--from <ms>] [--to <ms>] [--metric <r-metric>] [--name <n>]".to_string();
|
||||
let mut symbols: Option<Vec<String>> = None;
|
||||
let mut from_ms: Option<i64> = None;
|
||||
let mut to_ms: Option<i64> = None;
|
||||
let mut fast: Option<i64> = None;
|
||||
let mut slow: Option<i64> = None;
|
||||
let mut stop_length: Option<i64> = None;
|
||||
let mut stop_k: Option<f64> = None;
|
||||
let mut metric = "expectancy_r".to_string();
|
||||
let mut name = "generalize".to_string();
|
||||
// a single grid knob, required and single-valued (csv with exactly one item)
|
||||
let one = |value: &str, usage: &dyn Fn() -> String| -> Result<Vec<&str>, String> {
|
||||
let items: Vec<&str> = value.split(',').collect();
|
||||
if items.len() != 1 || items[0].is_empty() { return Err(usage()); }
|
||||
Ok(items)
|
||||
};
|
||||
let mut tail = rest;
|
||||
while let Some((flag, t)) = tail.split_first() {
|
||||
let (value, t) = t.split_first().ok_or_else(usage)?;
|
||||
match *flag {
|
||||
"--strategy" => { if *value != "stage1-r" { return Err(usage()); } }
|
||||
"--real" => {
|
||||
let parts: Vec<String> = value.split(',').map(|s| s.to_string()).collect();
|
||||
if parts.iter().any(|s| s.is_empty()) { return Err(usage()); }
|
||||
symbols = Some(parts);
|
||||
}
|
||||
"--from" => from_ms = Some(value.parse().map_err(|_| usage())?),
|
||||
"--to" => to_ms = Some(value.parse().map_err(|_| usage())?),
|
||||
"--fast" => { one(value, &usage)?; fast = Some(value.parse().map_err(|_| usage())?); }
|
||||
"--slow" => { one(value, &usage)?; slow = Some(value.parse().map_err(|_| usage())?); }
|
||||
"--stop-length" => { one(value, &usage)?; stop_length = Some(value.parse().map_err(|_| usage())?); }
|
||||
"--stop-k" => { one(value, &usage)?; stop_k = Some(value.parse().map_err(|_| usage())?); }
|
||||
"--metric" => metric = (*value).to_string(),
|
||||
"--name" => name = (*value).to_string(),
|
||||
_ => return Err(usage()),
|
||||
}
|
||||
tail = t;
|
||||
}
|
||||
let symbols = symbols.ok_or_else(usage)?;
|
||||
if symbols.len() < 2 { return Err(usage()); }
|
||||
// distinct: a duplicate would double-count one instrument in the floor / sign count
|
||||
let mut seen = std::collections::HashSet::new();
|
||||
if !symbols.iter().all(|s| seen.insert(s.clone())) { return Err(usage()); }
|
||||
let grid = Stage1RGrid {
|
||||
fast: vec![fast.ok_or_else(usage)?],
|
||||
slow: vec![slow.ok_or_else(usage)?],
|
||||
stop_length: vec![stop_length.ok_or_else(usage)?],
|
||||
stop_k: vec![stop_k.ok_or_else(usage)?],
|
||||
..Stage1RGrid::default()
|
||||
};
|
||||
Ok((name, symbols, grid, metric, from_ms, to_ms))
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Run the parser tests to verify they pass**
|
||||
|
||||
Run: `cargo test -p aura-cli parse_generalize`
|
||||
Expected: PASS — all six `parse_generalize_*` tests green.
|
||||
|
||||
- [ ] **Step 6: Implement `generalize_json` and `run_generalize`**
|
||||
|
||||
In `crates/aura-cli/src/main.rs`, near `walkforward_summary_json` (`:1969`), add the
|
||||
JSON builder (mirrors the `{"walkforward": obj}` shape; renders `selection_metric`
|
||||
as `"metric"`):
|
||||
|
||||
```rust
|
||||
fn generalize_json(agg: &Generalization) -> String {
|
||||
let per: Vec<serde_json::Value> = agg
|
||||
.per_instrument
|
||||
.iter()
|
||||
.map(|(sym, v)| serde_json::json!([sym, v]))
|
||||
.collect();
|
||||
let obj = serde_json::json!({
|
||||
"metric": agg.selection_metric,
|
||||
"n_instruments": agg.n_instruments,
|
||||
"worst_case": agg.worst_case,
|
||||
"sign_agreement": agg.sign_agreement,
|
||||
"per_instrument": per,
|
||||
});
|
||||
serde_json::json!({ "generalize": obj }).to_string()
|
||||
}
|
||||
```
|
||||
|
||||
And the run handler (near `run_sweep`, `:1750`). It pre-checks the metric
|
||||
(data-free refusal), runs the single-cell candidate per instrument, stamps the
|
||||
instrument, reduces, prints, and persists a `CrossInstrument` family:
|
||||
|
||||
```rust
|
||||
fn run_generalize(
|
||||
name: &str,
|
||||
symbols: &[String],
|
||||
grid: &Stage1RGrid,
|
||||
metric: &str,
|
||||
from_ms: Option<i64>,
|
||||
to_ms: Option<i64>,
|
||||
) {
|
||||
// data-free metric pre-check: refuse a non-R / unknown metric before any run.
|
||||
if let Err(e) = check_r_metric(metric) {
|
||||
eprintln!("aura: {e}");
|
||||
std::process::exit(2);
|
||||
}
|
||||
let mut members: Vec<RunReport> = Vec::new();
|
||||
for symbol in symbols {
|
||||
let choice = DataChoice::Real { symbol: symbol.clone(), from_ms, to_ms };
|
||||
let data = DataSource::from_choice(choice); // per-instrument pip_or_refuse + has_symbol
|
||||
let family = stage1_r_sweep_family(None, &data, grid); // single-cell grid -> 1 member
|
||||
let mut report = family.points[0].report.clone();
|
||||
report.manifest.instrument = Some(symbol.clone());
|
||||
members.push(report);
|
||||
}
|
||||
let pairs: Vec<(String, &RunReport)> = symbols.iter().cloned().zip(members.iter()).collect();
|
||||
let agg = match generalization(&pairs, metric) {
|
||||
Ok(a) => a,
|
||||
Err(e) => { eprintln!("aura: {e}"); std::process::exit(2); }
|
||||
};
|
||||
println!("{}", generalize_json(&agg));
|
||||
let reg = default_registry();
|
||||
match reg.append_family(name, FamilyKind::CrossInstrument, &members) {
|
||||
Ok(id) => println!("{{\"family_id\":\"{id}\"}}"),
|
||||
Err(e) => { eprintln!("aura: failed to persist family: {e}"); std::process::exit(1); }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 7: Wire the dispatch arm and extend USAGE**
|
||||
|
||||
In `crates/aura-cli/src/main.rs`, add a `["generalize", rest @ ..]` arm beside
|
||||
`["sweep", ..]` (`:2993`):
|
||||
```rust
|
||||
["generalize", rest @ ..] => match parse_generalize_args(rest) {
|
||||
Ok((name, symbols, grid, metric, from_ms, to_ms)) => {
|
||||
run_generalize(&name, &symbols, &grid, &metric, from_ms, to_ms)
|
||||
}
|
||||
Err(msg) => {
|
||||
eprintln!("aura: {msg}");
|
||||
std::process::exit(2);
|
||||
}
|
||||
},
|
||||
```
|
||||
Append the form to the `USAGE` const (`:2956`), before the closing `aura runs ...`:
|
||||
```
|
||||
| aura generalize [--strategy stage1-r] --real <SYM1,SYM2,...> --fast <n> --slow <n> --stop-length <n> --stop-k <f> [--from <ms>] [--to <ms>] [--metric <r-metric>] [--name <n>]
|
||||
```
|
||||
|
||||
- [ ] **Step 8: Write the E2E tests**
|
||||
|
||||
In `crates/aura-cli/tests/cli_run.rs`, add (model the gated-skip on
|
||||
`run_real_sidecar_index_pip_reaches_the_emitted_manifest`, `:250`):
|
||||
|
||||
```rust
|
||||
/// Property: `aura generalize` grades a single stage1-r candidate across two
|
||||
/// instruments — its stdout carries the per-instrument breakdown + the worst-case
|
||||
/// floor + the sign-agreement, then the persisted CrossInstrument family handle.
|
||||
/// Gated on local GER40/USDJPY data (the shared Sept-2024 window), skips cleanly
|
||||
/// when the archive is absent.
|
||||
#[test]
|
||||
fn generalize_grades_a_candidate_across_two_instruments() {
|
||||
const FROM_MS: &str = "1725148800000";
|
||||
const TO_MS: &str = "1727740799999";
|
||||
let cwd = temp_cwd("generalize-two");
|
||||
let out = std::process::Command::new(env!("CARGO_BIN_EXE_aura"))
|
||||
.args([
|
||||
"generalize", "--strategy", "stage1-r", "--real", "GER40,USDJPY",
|
||||
"--fast", "3", "--slow", "12", "--stop-length", "14", "--stop-k", "2.0",
|
||||
"--from", FROM_MS, "--to", TO_MS,
|
||||
])
|
||||
.current_dir(&cwd)
|
||||
.output()
|
||||
.expect("spawn aura");
|
||||
if out.status.code() == Some(2) {
|
||||
let stderr = String::from_utf8_lossy(&out.stderr);
|
||||
assert!(
|
||||
stderr.contains("no local data") || stderr.contains("no recorded geometry"),
|
||||
"exit 2 must be a data refusal, got: {stderr}"
|
||||
);
|
||||
eprintln!("skip: no local GER40/USDJPY data");
|
||||
return;
|
||||
}
|
||||
assert_eq!(out.status.code(), Some(0), "exit: {:?}", out.status);
|
||||
let stdout = String::from_utf8(out.stdout).expect("utf-8 stdout");
|
||||
assert!(stdout.contains("\"generalize\":"), "aggregate object: {stdout}");
|
||||
assert!(stdout.contains("\"n_instruments\":2"), "two instruments: {stdout}");
|
||||
assert!(stdout.contains("\"worst_case\":"), "worst_case present: {stdout}");
|
||||
assert!(stdout.contains("\"sign_agreement\":"), "sign_agreement present: {stdout}");
|
||||
assert!(stdout.contains("GER40") && stdout.contains("USDJPY"), "per-instrument breakdown: {stdout}");
|
||||
assert!(stdout.contains("\"family_id\":\"generalize-0\""), "family handle: {stdout}");
|
||||
}
|
||||
|
||||
/// Property: a single-instrument generalize is refused at parse time (exit 2),
|
||||
/// before any data access — so it asserts the arity refusal on any machine.
|
||||
#[test]
|
||||
fn generalize_refuses_a_single_instrument() {
|
||||
let out = std::process::Command::new(env!("CARGO_BIN_EXE_aura"))
|
||||
.args([
|
||||
"generalize", "--strategy", "stage1-r", "--real", "GER40",
|
||||
"--fast", "3", "--slow", "12", "--stop-length", "14", "--stop-k", "2.0",
|
||||
])
|
||||
.output()
|
||||
.expect("spawn aura");
|
||||
assert_eq!(out.status.code(), Some(2), "single instrument must exit 2");
|
||||
}
|
||||
|
||||
/// Property: a non-R metric is refused before any instrument runs (the data-free
|
||||
/// `check_r_metric` pre-check), so this asserts the R-only refusal on any machine.
|
||||
#[test]
|
||||
fn generalize_refuses_a_non_r_metric() {
|
||||
let out = std::process::Command::new(env!("CARGO_BIN_EXE_aura"))
|
||||
.args([
|
||||
"generalize", "--strategy", "stage1-r", "--real", "GER40,USDJPY",
|
||||
"--fast", "3", "--slow", "12", "--stop-length", "14", "--stop-k", "2.0",
|
||||
"--metric", "total_pips",
|
||||
])
|
||||
.output()
|
||||
.expect("spawn aura");
|
||||
assert_eq!(out.status.code(), Some(2), "a non-R metric must exit 2");
|
||||
let stderr = String::from_utf8_lossy(&out.stderr);
|
||||
assert!(stderr.contains("R-only"), "must name the R-only refusal, got: {stderr}");
|
||||
}
|
||||
```
|
||||
|
||||
> `temp_cwd(name) -> PathBuf` (cli_run.rs:13) is the confirmed registry-isolation
|
||||
> helper (a fresh per-test cwd, so the family-store run index starts at 0 →
|
||||
> `generalize-0`); `env!("CARGO_BIN_EXE_aura")` is the binary handle (the same the
|
||||
> real-data E2Es at cli_run.rs:256 use). Both already exist in the file.
|
||||
|
||||
- [ ] **Step 9: Run the E2E tests**
|
||||
|
||||
Run: `cargo test -p aura-cli generalize`
|
||||
Expected: PASS — `generalize_refuses_a_single_instrument` and
|
||||
`generalize_refuses_a_non_r_metric` assert exit 2 (data-free);
|
||||
`generalize_grades_a_candidate_across_two_instruments` passes on a machine with
|
||||
GER40/USDJPY data or skips cleanly otherwise; plus the six `parse_generalize_*`
|
||||
unit tests green.
|
||||
|
||||
- [ ] **Step 10: Verify the CLI suite stays green (C23 preservation)**
|
||||
|
||||
Run: `cargo test -p aura-cli`
|
||||
Expected: PASS (0 failed) — every existing sweep/walkforward/mc/run/`runs` test
|
||||
stays green; the existing manifest goldens are byte-identical (instrument is
|
||||
omitted when None — `skip_serializing_if`), so no sweep/walkforward/standalone
|
||||
output changed.
|
||||
|
||||
- [ ] **Step 11: Final workspace gate**
|
||||
|
||||
Run: `cargo test --workspace`
|
||||
Expected: PASS (0 failed).
|
||||
|
||||
Run: `cargo clippy --workspace --all-targets -- -D warnings`
|
||||
Expected: clean (0 warnings).
|
||||
@@ -1,396 +0,0 @@
|
||||
# Cross-instrument generalization as a validation step — Design Spec
|
||||
|
||||
**Date:** 2026-06-26
|
||||
**Status:** Draft — awaiting user spec review
|
||||
**Authors:** orchestrator + Claude
|
||||
|
||||
## Goal
|
||||
|
||||
Add a first-class validation step that grades how consistently a single
|
||||
**candidate** holds across a set of instruments. This is the last piece of the
|
||||
"Inferential validation" milestone and the across-instrument axis of the same
|
||||
anti-false-discovery concern as trials-deflation (#144, within one family) and
|
||||
plateau-over-peak (#145, within one instrument's surface): an edge that appears
|
||||
on one instrument can be a fit to that series' noise; an edge that survives
|
||||
across several independent instruments is far less likely to be spurious.
|
||||
|
||||
The step is an **aggregator / validator**, not a third selector. It takes a
|
||||
*brought* candidate — one already chosen (e.g. by deflation or plateau on a
|
||||
reference instrument, or authored directly) — runs it across an instrument list,
|
||||
collects per-instrument R-metrics, and reduces them to a generalization grade.
|
||||
It does **not** select a winner across instruments (that is a heavier follow-on,
|
||||
out of scope below). A generalization grade is an aggregate (like `McAggregate`),
|
||||
not a within-family selection, so it is **not** recorded on `FamilySelection`
|
||||
(which annotates a within-family winner that does not exist here).
|
||||
|
||||
## Architecture
|
||||
|
||||
A new `aura generalize` subcommand. Input is a single **candidate** (the four
|
||||
stage1-r grid knobs pinned to single values: `--fast --slow --stop-length
|
||||
--stop-k`) crossed with an **instrument list** (`--real SYM1,SYM2,...`,
|
||||
comma-separated; a shared optional `--from`/`--to` window, full available range
|
||||
per instrument if omitted). For each instrument the candidate runs as a single
|
||||
full-window stage1-r R evaluation, the M `RunReport`s are collected, reduced to a
|
||||
generalization aggregate, the per-instrument breakdown and the aggregate are
|
||||
printed, and the M runs are persisted as a `CrossInstrument` family.
|
||||
|
||||
The per-instrument run is an **honest out-of-sample test by construction**:
|
||||
because the candidate is fixed/external — not fit on that instrument — a single
|
||||
full-window run on each instrument is already OOS for it. No per-instrument
|
||||
walk-forward is needed for this first correct version.
|
||||
|
||||
The reduction is **R-only**. R is the only account- and instrument-agnostic unit
|
||||
(C10); `total_pips` is meaningless across instruments (different pip scales and
|
||||
volatility). The headline score is the **worst case across instruments** — the
|
||||
minimum of the chosen R-metric — which a strong instrument cannot lift (the
|
||||
issue's non-domination constraint, satisfied by construction). It is the
|
||||
cross-instrument sibling of `PlateauMode::Worst` (which already takes the
|
||||
most-pessimistic neighbour). Because R-metrics are all higher-is-better, the
|
||||
worst case is unconditionally the minimum (no direction branch). Alongside it the
|
||||
aggregate carries the **sign-agreement count** (how many instruments are
|
||||
net-positive) and the **full per-instrument breakdown** — together these fully
|
||||
express "consistency of sign and magnitude".
|
||||
|
||||
This honours: **C23** (additive — every existing path stays byte-identical; the
|
||||
subcommand is new surface), **C9** (the reduction lives registry-side, the engine
|
||||
stays simulation), **C12** (cross-instrument is the comparison orchestration
|
||||
axis → a new `FamilyKind`), **C14/C18** (the new manifest field serde-widens with
|
||||
the same back-compat idiom as `selection`), **C1** (a pure fold over the
|
||||
per-instrument results in input order), **C2** (each per-instrument run is an
|
||||
ordinary causal sim), **C10** (R-only).
|
||||
|
||||
## Concrete code shapes
|
||||
|
||||
### The user-facing program (the acceptance evidence)
|
||||
|
||||
The researcher has a candidate and asks "does it generalize across instruments?":
|
||||
|
||||
```
|
||||
aura generalize --strategy stage1-r --real GER40,USDJPY \
|
||||
--fast 3 --slow 12 --stop-length 14 --stop-k 2.0 \
|
||||
--from 1725148800000 --to 1727740799999
|
||||
```
|
||||
|
||||
Expected stdout (the aggregate object + the persisted family handle; numbers
|
||||
illustrative):
|
||||
|
||||
```json
|
||||
{"generalize":{"metric":"expectancy_r","n_instruments":2,"worst_case":-0.07,"sign_agreement":1,"per_instrument":[["GER40",0.31],["USDJPY",-0.07]]}}
|
||||
{"family_id":"generalize-0"}
|
||||
```
|
||||
|
||||
The aggregate is the diagnosis: the candidate makes `+0.31 R/trade` on GER40 but
|
||||
**loses** on USDJPY, so its worst-case generalization is negative and only 1 of 2
|
||||
instruments agrees in sign — a false-discovery flag the per-instrument view would
|
||||
have hidden behind GER40's strength.
|
||||
|
||||
A second `--metric` selects the R-metric (default `expectancy_r`):
|
||||
|
||||
```
|
||||
aura generalize --strategy stage1-r --real GER40,USDJPY --fast 3 --slow 12 \
|
||||
--stop-length 14 --stop-k 2.0 --metric sqn_normalized
|
||||
```
|
||||
|
||||
Must-fail surfaces (each exits 2 with a clear message):
|
||||
|
||||
```
|
||||
aura generalize --strategy stage1-r --real GER40 --fast 3 --slow 12 --stop-length 14 --stop-k 2.0
|
||||
# refuse: a generalization needs >= 2 instruments
|
||||
|
||||
aura generalize --strategy stage1-r --real GER40,USDJPY --fast 3 --slow 12 --stop-length 14 --stop-k 2.0 --metric total_pips
|
||||
# refuse: total_pips is not comparable across instruments; cross-instrument scoring is R-only
|
||||
|
||||
aura generalize --strategy stage1-r --real GER40,USDJPY --fast 2,3 --slow 12 --stop-length 14 --stop-k 2.0
|
||||
# refuse: a candidate is a single cell; --fast got 2 values
|
||||
|
||||
aura generalize --strategy sma --real GER40,USDJPY --fast 3 --slow 12 --stop-length 14 --stop-k 2.0
|
||||
# refuse: generalize requires --strategy stage1-r (the candidate must produce R)
|
||||
```
|
||||
|
||||
### Implementation shapes (before → after, secondary)
|
||||
|
||||
**1. `RunManifest` gains a first-class instrument field** — `crates/aura-engine/src/report.rs`.
|
||||
Same serde-widening idiom as `selection` (C14/C18): legacy lines without the key
|
||||
read back as `None`. Closes a latent gap (today a real run records the
|
||||
window/broker but not *which* symbol).
|
||||
|
||||
```rust
|
||||
// before
|
||||
pub struct RunManifest {
|
||||
pub commit: String,
|
||||
pub params: Vec<(String, Scalar)>,
|
||||
pub window: (Timestamp, Timestamp),
|
||||
pub seed: u64,
|
||||
pub broker: String,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub selection: Option<FamilySelection>,
|
||||
}
|
||||
|
||||
// after — add one field (every RunManifest literal threads `instrument: None`,
|
||||
// the same compile-gate the #144 `selection` field was; the planner compile-drives
|
||||
// the full literal set)
|
||||
pub struct RunManifest {
|
||||
pub commit: String,
|
||||
pub params: Vec<(String, Scalar)>,
|
||||
pub window: (Timestamp, Timestamp),
|
||||
pub seed: u64,
|
||||
pub broker: String,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub selection: Option<FamilySelection>,
|
||||
/// The instrument this run evaluated, when it is a real-data run; `None` for a
|
||||
/// synthetic run and for every pre-0078 line. The "instrument set as lineage"
|
||||
/// (C18) for a cross-instrument family is each member's stamped symbol.
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub instrument: Option<String>,
|
||||
}
|
||||
```
|
||||
|
||||
**2. `FamilyKind` gains the comparison axis** — `crates/aura-registry/src/lineage.rs`.
|
||||
A compile-gate: every `match` on `FamilyKind` (the `runs family` display, any
|
||||
kind-dispatch) must gain the arm.
|
||||
|
||||
```rust
|
||||
// before
|
||||
pub enum FamilyKind { Sweep, MonteCarlo, WalkForward }
|
||||
// after
|
||||
pub enum FamilyKind { Sweep, MonteCarlo, WalkForward, CrossInstrument }
|
||||
```
|
||||
|
||||
**3. The generalization reduction** — `crates/aura-registry/src/lib.rs`, a sibling
|
||||
of `optimize_deflated` / `optimize_plateau`, reusing `resolve_metric` /
|
||||
`metric_value` / `is_r_metric` (all already present). Note the input is **not** a
|
||||
`SweepFamily` and the output is **not** `(SweepPoint, FamilySelection)` — this is
|
||||
an aggregator, so it has its own value type.
|
||||
|
||||
```rust
|
||||
/// The cross-instrument generalization aggregate (#146): a brought candidate's
|
||||
/// per-instrument R-metric, reduced to a worst-case floor + a sign-agreement
|
||||
/// count + the full breakdown. R-only (C10): `total_pips` is not comparable across
|
||||
/// instruments. Pure, deterministic (C1) — a fold over `per_instrument` in input
|
||||
/// order. The headline `worst_case` is `min_i metric_i`; since every R-metric is
|
||||
/// higher-is-better, the min is unconditionally the conservative floor (no
|
||||
/// direction branch, unlike `PlateauMode::Worst`).
|
||||
pub struct Generalization {
|
||||
pub selection_metric: String,
|
||||
pub n_instruments: usize,
|
||||
pub worst_case: f64, // min_i metric_i
|
||||
pub sign_agreement: usize, // #{ i : metric_i > 0 }
|
||||
pub per_instrument: Vec<(String, f64)>,
|
||||
}
|
||||
|
||||
pub fn generalization(
|
||||
per_instrument: &[(String, &RunReport)],
|
||||
metric: &str,
|
||||
) -> Result<Generalization, RegistryError> {
|
||||
let m = resolve_metric(metric)?;
|
||||
if !is_r_metric(m) {
|
||||
return Err(RegistryError::NonRMetric(metric.to_string()));
|
||||
}
|
||||
if per_instrument.len() < 2 {
|
||||
return Err(RegistryError::TooFewInstruments(per_instrument.len()));
|
||||
}
|
||||
let vals: Vec<(String, f64)> = per_instrument
|
||||
.iter()
|
||||
.map(|(label, rep)| (label.clone(), metric_value(rep, m)))
|
||||
.collect();
|
||||
let worst_case = vals.iter().map(|(_, v)| *v).fold(f64::INFINITY, f64::min);
|
||||
let sign_agreement = vals.iter().filter(|(_, v)| *v > 0.0).count();
|
||||
Ok(Generalization {
|
||||
selection_metric: metric.to_string(),
|
||||
n_instruments: vals.len(),
|
||||
worst_case,
|
||||
sign_agreement,
|
||||
per_instrument: vals,
|
||||
})
|
||||
}
|
||||
```
|
||||
|
||||
Two new `RegistryError` variants (in-style with the existing typed variants):
|
||||
|
||||
```rust
|
||||
pub enum RegistryError {
|
||||
Io(std::io::Error),
|
||||
Parse { line: usize, source: serde_json::Error },
|
||||
UnknownMetric(String),
|
||||
NonRMetric(String), // a known metric that is not R-based (cross-instrument is R-only)
|
||||
TooFewInstruments(usize), // a generalization needs >= 2 instruments
|
||||
}
|
||||
```
|
||||
|
||||
**4. The CLI subcommand** — `crates/aura-cli/src/main.rs`. A new dispatch arm beside
|
||||
`sweep` / `walkforward`, a `parse_generalize_args`, and a `run_generalize` that
|
||||
loops the candidate over the instruments, stamps each manifest's `instrument`,
|
||||
reduces, prints, and persists.
|
||||
|
||||
```rust
|
||||
// dispatch (beside the existing ["sweep", ..] / ["walkforward", ..] arms)
|
||||
["generalize", rest @ ..] => match parse_generalize_args(rest) {
|
||||
Ok((name, symbols, grid, metric, from_ms, to_ms)) => {
|
||||
run_generalize(&name, &symbols, &grid, &metric, from_ms, to_ms)
|
||||
}
|
||||
Err(msg) => { eprintln!("aura: {msg}"); std::process::exit(2); }
|
||||
},
|
||||
|
||||
// run_generalize (shape): one full-window stage1-r R run per instrument, the
|
||||
// candidate Stage1RGrid pinned to single values; stamp manifest.instrument; reduce.
|
||||
fn run_generalize(name: &str, symbols: &[String], grid: &Stage1RGrid,
|
||||
metric: &str, from_ms: Option<i64>, to_ms: Option<i64>) {
|
||||
let mut members: Vec<RunReport> = Vec::new();
|
||||
for symbol in symbols {
|
||||
// the existing real-source construction (per-instrument pip_or_refuse + has_symbol)
|
||||
let choice = DataChoice::Real { symbol: symbol.clone(), from_ms, to_ms };
|
||||
let data = DataSource::from_choice(choice);
|
||||
let family = stage1_r_sweep_family(None, &data, grid); // single-cell grid -> 1 member
|
||||
let mut report = family.points[0].report.clone();
|
||||
report.manifest.instrument = Some(symbol.clone());
|
||||
members.push(report);
|
||||
}
|
||||
let pairs: Vec<(String, &RunReport)> =
|
||||
symbols.iter().cloned().zip(members.iter()).collect();
|
||||
let agg = match aura_registry::generalization(&pairs, metric) {
|
||||
Ok(a) => a,
|
||||
Err(e) => { eprintln!("aura: {e}"); std::process::exit(2); }
|
||||
};
|
||||
println!("{}", generalize_json(&agg)); // {"generalize": { ... }}
|
||||
// persist the M runs as a CrossInstrument family (same Err-handling as `sweep`)
|
||||
match reg.append_family(name, FamilyKind::CrossInstrument, &members) {
|
||||
Ok(id) => println!("{{\"family_id\":\"{id}\"}}"),
|
||||
Err(e) => { eprintln!("aura: failed to persist family: {e}"); std::process::exit(1); }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The candidate is a `Stage1RGrid` whose `fast`/`slow`/`stop_length`/`stop_k` each
|
||||
carry **exactly one** value; `parse_generalize_args` requires all four explicitly
|
||||
and refuses a multi-value flag (a candidate is one cell — it must not default to
|
||||
the multi-value sweep grid). `--real` is parsed as a comma-list into
|
||||
`Vec<String>`; the existing single-symbol `RealWindowGrammar` / `sweep` /
|
||||
`walkforward` parsers are untouched.
|
||||
|
||||
## Components
|
||||
|
||||
1. **`RunManifest.instrument: Option<String>`** (aura-engine/report.rs) — the
|
||||
instrument lineage; serde `default` / `skip_serializing_if`. Every existing
|
||||
`RunManifest` literal threads `instrument: None` (compile-gate, same shape as
|
||||
#144's `selection`). The synthetic and existing real run paths leave it `None`
|
||||
(or stamp it — a non-load-bearing nicety; this cycle only requires the
|
||||
generalize path to stamp it).
|
||||
2. **`FamilyKind::CrossInstrument`** (aura-registry/lineage.rs) — the comparison
|
||||
axis tag; persisted on each member record. Compile-gate on `FamilyKind`
|
||||
matches.
|
||||
3. **`generalization` + `Generalization` + two `RegistryError` variants**
|
||||
(aura-registry/lib.rs) — the R-only reduction beside the sibling selectors,
|
||||
reusing `resolve_metric` / `metric_value` / `is_r_metric`.
|
||||
4. **CLI: `generalize` dispatch arm, `parse_generalize_args`, `run_generalize`,
|
||||
`generalize_json`** (aura-cli/main.rs) — the candidate × instrument-list loop;
|
||||
reuses `stage1_r_sweep_family` (single-cell grid), the per-instrument
|
||||
`DataSource::Real` construction (`pip_or_refuse` + `has_symbol` refusals), and
|
||||
`append_family`. `USAGE` gains the `aura generalize ...` form.
|
||||
|
||||
## Data flow
|
||||
|
||||
```
|
||||
candidate (single-cell Stage1RGrid) + [SYM_1 .. SYM_M] + window
|
||||
-> for each SYM_i:
|
||||
DataSource::Real{SYM_i} (pip_or_refuse + has_symbol)
|
||||
-> stage1_r_sweep_family(single-cell grid) -> SweepFamily{ 1 point }
|
||||
-> report = points[0].report; report.manifest.instrument = Some(SYM_i)
|
||||
-> members: Vec<RunReport> (length M, input order)
|
||||
-> generalization(zip(symbols, members), metric)
|
||||
-> Generalization{ worst_case = min_i, sign_agreement, per_instrument }
|
||||
-> print {"generalize": agg}
|
||||
-> append_family(name, CrossInstrument, members) -> family_id
|
||||
-> print {"family_id": id}
|
||||
```
|
||||
|
||||
The aggregate is a pure function of the persisted members, so it is recomputable
|
||||
on reload (the `McAggregate` precedent) — no new persisted family-level record
|
||||
type is introduced.
|
||||
|
||||
## Error handling
|
||||
|
||||
Every refusal is stderr + `exit(2)`, the established CLI convention:
|
||||
|
||||
- **< 2 instruments** — a one-instrument "generalization" is meaningless.
|
||||
Enforced both at `parse_generalize_args` (early, on the parsed list) and in
|
||||
`generalization` (`TooFewInstruments`, the reusable-core guard).
|
||||
- **non-R / pip metric** — cross-instrument scoring is R-only;
|
||||
`generalization` returns `NonRMetric`. The CLI surfaces it as exit 2.
|
||||
- **strategy not stage1-r** — the candidate must produce R; `parse_generalize_args`
|
||||
refuses any `--strategy` other than `stage1-r`.
|
||||
- **a multi-value candidate flag** — a candidate is a single cell;
|
||||
`parse_generalize_args` refuses when any of `--fast/--slow/--stop-length/--stop-k`
|
||||
carries more than one value, or is absent.
|
||||
- **empty / duplicate symbol** in the list — refuse (a duplicate would
|
||||
double-count one instrument in the worst-case and sign-agreement).
|
||||
- **a symbol with no recorded geometry / no archive** — the existing
|
||||
`pip_or_refuse` / `no_real_data` refusals fire per instrument before any run.
|
||||
|
||||
## Testing strategy
|
||||
|
||||
RED-first where pinnable. Per-crate test command, single positional filter.
|
||||
|
||||
Registry unit tests (aura-registry):
|
||||
- **worst_case picks the floor:** a fabricated 3-member set (one strongly
|
||||
positive, one negative, one mid) → `worst_case == the negative value`, **not**
|
||||
the mean; `sign_agreement == 2`.
|
||||
- **non-domination:** appending a hugely-positive fourth instrument does **not**
|
||||
raise `worst_case` (the min is unchanged).
|
||||
- **R-only refusal:** `generalization(.., "total_pips")` → `Err(NonRMetric)`.
|
||||
- **too-few refusal:** a 1-element input → `Err(TooFewInstruments(1))`.
|
||||
- **determinism (C1):** the same inputs → an identical `Generalization` (it is a
|
||||
pure fold).
|
||||
|
||||
Engine serde tests (aura-engine):
|
||||
- **back-compat (C14/C18):** a pre-0078 manifest JSON line without `instrument`
|
||||
deserializes with `instrument == None`; a round-trip with `instrument ==
|
||||
Some("GER40")` preserves it; the existing `RunManifest`/`FamilySelection` serde
|
||||
and legacy-load tests stay green.
|
||||
|
||||
CLI E2E (aura-cli/tests/cli_run.rs), gated-skip on absent local data exactly like
|
||||
the existing `--real` tests:
|
||||
- `aura generalize --strategy stage1-r --real GER40,USDJPY --fast .. --slow ..
|
||||
--stop-length .. --stop-k .. --from .. --to ..` over the shared Sept-2024 window
|
||||
→ stdout carries a `{"generalize": ...}` object with `n_instruments == 2`, a
|
||||
`per_instrument` breakdown naming both symbols, a `worst_case`, a
|
||||
`sign_agreement`, and a following `{"family_id":"generalize-0"}`; when the
|
||||
archive is absent, the run hits the no-data refusal and the test skips cleanly
|
||||
(the established pattern).
|
||||
- a must-fail E2E: `--real GER40` (single instrument) exits 2 with the
|
||||
>=2-instruments message (no data needed — the arity refusal precedes any run).
|
||||
|
||||
C23 preservation:
|
||||
- the existing `sweep` / `walkforward` / `mc` / standalone-run goldens stay
|
||||
**byte-identical** — `generalize` is new surface and touches no existing path.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
Against aura's feature-acceptance criterion:
|
||||
|
||||
1. **The intended audience reaches for it.** The aura researcher validating a
|
||||
candidate naturally asks "does this hold across instruments?" — the last
|
||||
anti-false-discovery piece of the milestone. The worked `aura generalize`
|
||||
invocation above is the evidence: it is the command a researcher writes, and
|
||||
its output is the generalization verdict.
|
||||
2. **It measurably improves correctness.** A candidate that looks good on one
|
||||
instrument but fails the worst-case-across-instruments floor (negative
|
||||
`worst_case`, low `sign_agreement`) is now caught — a real defense the
|
||||
per-instrument view hides. It composes downstream of #144/#145.
|
||||
3. **It reintroduces no failure class the core constraints eliminate.** R-only
|
||||
(C10), behaviour-preserving (C23 — existing paths byte-identical), deterministic
|
||||
(C1), causal per-instrument sims (C2), registry-side policy (C9), serde-widened
|
||||
with full back-compat (C14/C18). No invariant is touched.
|
||||
|
||||
## Out of scope (named, deferred)
|
||||
|
||||
- **Cross-instrument selection (reading B):** running a full sweep per instrument
|
||||
and picking the cell that generalizes best across instruments. A heavier
|
||||
follow-on that conflates selection with validation; this cycle is the validator.
|
||||
- **Per-instrument walk-forward evaluation:** the first version runs a full-window
|
||||
R evaluation per instrument (honest OOS because the candidate is fixed). A
|
||||
per-instrument walk-forward knob is a later refinement.
|
||||
- **A persisted family-level aggregate record:** the aggregate is recomputable from
|
||||
the members (the `McAggregate` precedent); no new persisted record type.
|
||||
- **Cross-instrument R-pooling of trade series:** #139's deferred cross-symbol
|
||||
pooling stays deferred.
|
||||
- **Pip-metric generalization:** undefined across instruments (C10); refused.
|
||||
Reference in New Issue
Block a user