feat: the IC becomes the second metric vocabulary; rosters single-sourced
The A1 cut of #147 item 2, part 2 of 2 (the consumers), completing what part 1 (the MetricVocabulary substrate) set up: - aura-campaign's RANKABLE_METRICS hand-copy becomes a re-export of the backtest vocabulary's roster; the #190 guard test extends to pin the roster nesting (rankable == vocabulary roster, rankable within per-member, per-member within the research vocabulary) and the legacy exposure_sign_flips alias, so any drift fails a test loudly. The research and per-member rosters themselves stay hand-copies by ratified isolation (#190 accepted residual) — now oracle-pinned instead of trusted. - aura-cli gains IcMetrics + IcKey implementing MetricVocabulary: the second PRODUCTION implementor. The standalone `aura measure ic` reduction now computes its permutation p via the shared null_draw + one_sided_p_laplace building blocks (independent draws instead of chained shuffles — statistically equivalent, deterministic per seed; emitted p values may differ in the last digits from #290's). - Acceptance proven at the registry boundary: an engineered SweepFamily<IcMetrics> deflates through optimize_deflated to a LOW overfit probability and a noise family to a HIGH one (the permutation null, dispatched by the vocabulary — the deflation machinery is no longer baked-in R-only); same seed reproduces the identical FamilySelection; cross-vocabulary names are refused with the family's own roster in the prose (the C10 guard). A new cli_run E2E exercises the legacy alias through the real `aura runs family … rank` boundary. Deviation from the plan, forced by the compiler: IcMetrics additionally derives Debug (unwrap_err on the deflation Result needs it; mirrors RunMetrics). Verification: full workspace suite green (no FAILED, no compile errors), clippy -D warnings clean, C18 goldens (cli_run, measure_ic, c28_layering) green unchanged. A2 — measurement runs as sweep-family citizens — stays deliberately deferred until a concrete campaign demand exists. closes #147
This commit is contained in:
@@ -1146,6 +1146,70 @@ fn runs_families_list_and_per_family_rank_across_invocations() {
|
||||
let _ = std::fs::remove_dir_all(&cwd);
|
||||
}
|
||||
|
||||
/// #147: the pre-rename metric name `exposure_sign_flips` still resolves
|
||||
/// through the real, user-facing `aura runs family <id> rank <metric>` path —
|
||||
/// not just through a direct `MetricVocabulary::resolve` call in a unit test.
|
||||
/// Task 4 re-exports `aura_campaign::RANKABLE_METRICS` from the single-sourced
|
||||
/// backtest vocabulary instead of hand-copying it; were the vocabulary's
|
||||
/// `resolve()` match arm for the legacy alias ever dropped in that refactor,
|
||||
/// this is the only test that would catch it at the CLI's actual observable
|
||||
/// boundary (exit code + stdout), rather than inside the crate that owns the
|
||||
/// vocabulary.
|
||||
#[test]
|
||||
fn runs_family_rank_accepts_legacy_exposure_sign_flips_alias() {
|
||||
let cwd = temp_cwd("runs-rank-legacy-alias");
|
||||
let fixture = format!("{}/examples/r_sma.json", env!("CARGO_MANIFEST_DIR"));
|
||||
let sweep = Command::new(BIN)
|
||||
.args([
|
||||
"sweep", &fixture,
|
||||
"--axis", "sma_signal.fast.length=2,4",
|
||||
"--axis", "sma_signal.slow.length=8,16",
|
||||
])
|
||||
.current_dir(&cwd)
|
||||
.output()
|
||||
.expect("spawn sweep");
|
||||
assert!(sweep.status.success(), "sweep stderr: {}", String::from_utf8_lossy(&sweep.stderr));
|
||||
|
||||
let rank = Command::new(BIN)
|
||||
.args(["runs", "family", "sweep-0", "rank", "exposure_sign_flips"])
|
||||
.current_dir(&cwd)
|
||||
.output()
|
||||
.expect("spawn rank");
|
||||
assert!(
|
||||
rank.status.success(),
|
||||
"the legacy alias must still resolve, not usage-error: stderr={}",
|
||||
String::from_utf8_lossy(&rank.stderr)
|
||||
);
|
||||
let rank_out = String::from_utf8(rank.stdout).expect("utf-8");
|
||||
assert_eq!(rank_out.lines().count(), 4, "all 4 members rank: {rank_out:?}");
|
||||
// the renamed field is what actually ships in the JSON — the alias is a
|
||||
// resolve()-only back door, never a second output key.
|
||||
assert!(
|
||||
rank_out.lines().all(|l| l.contains("\"bias_sign_flips\":")),
|
||||
"output still serializes under the new key: {rank_out:?}"
|
||||
);
|
||||
assert!(
|
||||
!rank_out.contains("\"exposure_sign_flips\":"),
|
||||
"the legacy key must never appear in output: {rank_out:?}"
|
||||
);
|
||||
|
||||
let flips: Vec<i64> = rank_out
|
||||
.lines()
|
||||
.map(|l| {
|
||||
let key = "\"bias_sign_flips\":";
|
||||
let start = l.find(key).expect("line has bias_sign_flips") + key.len();
|
||||
let tail = &l[start..];
|
||||
let end = tail.find([',', '}']).expect("token end");
|
||||
tail[..end].parse().expect("bias_sign_flips is an integer")
|
||||
})
|
||||
.collect();
|
||||
// bias_sign_flips is a lower-is-better metric (fewer direction reversals),
|
||||
// so `rank` orders ascending here — unlike total_pips's descending order.
|
||||
assert!(flips.windows(2).all(|w| w[0] <= w[1]), "rank not ascending by the aliased metric: {flips:?}");
|
||||
|
||||
let _ = std::fs::remove_dir_all(&cwd);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn runs_list_and_rank_are_retired_and_exit_two() {
|
||||
// `aura runs list` / `runs rank <metric>` were retired (#73): families
|
||||
|
||||
Reference in New Issue
Block a user