diff --git a/docs/specs/0059-generic-portable-sweep-key.md b/docs/specs/0059-generic-portable-sweep-key.md new file mode 100644 index 0000000..176cf93 --- /dev/null +++ b/docs/specs/0059-generic-portable-sweep-key.md @@ -0,0 +1,431 @@ +# Generic, filesystem-portable sweep member-key + a bool-param demo strategy — Design Spec + +**Date:** 2026-06-21 +**Status:** Draft — awaiting user spec review +**Authors:** orchestrator + Claude + +> Reference issue: #105 (Brummel/Aura). Design decision log: the latest comment +> on #105 (forks: key-from-varying-axes, portable charset, demo strategy shape). + +## Goal + +Two coupled deliverables, in this order: + +1. **Generalise the family-member trace key** so it is collision-free for *any* + sweep grid, decoupled from the two hardcoded axis names — and so the on-disk + key is a **filesystem-conformant directory name on every major OS** (Linux, + Windows, macOS), including case-insensitive filesystems. +2. **A new demo strategy** with three params of three kinds — one of them a + `bool` — swept generically. It is the empirical proof that (1) is generic: the + same `aura sweep` machinery drives a strategy whose params have nothing in + common with the SMA-cross demo, and the bool axis shows up, conformant, in the + member directory name. + +The foot-gun being closed (#105): `sweep_member_key` renders `f{fast}s{slow}` +from two hardcoded axes. It is collision-free only over *today's* grid; a grid +that varies any other axis silently maps distinct points to the same directory +and overwrites traces. The fix derives the key from the axes that *actually +vary*, generically. + +## Architecture + +Three changes, smallest-blast-radius first: + +- **Engine (additive, one pure read accessor).** `SweepBinder` already + accumulates its declared axes as `Vec<(String, Vec)>` + (`blueprint.rs:379-382`). Add `SweepBinder::varying_axes(&self) -> Vec` + returning the names of axes with **> 1 value** — the axes that distinguish one + grid point from another. This is the binder's own knowledge (it holds the + value-lists); exposing it is a benign, behaviour-preserving addition. Nothing + else in the engine changes — the `sweep` HOF signature, `GridSpace`, + `SweepFamily`, and the `Fn(&[Cell]) -> RunReport + Sync` closure contract are + untouched (consistent with the #104 design: persistence stays in the CLI + closure, the engine carries no trace state). + +- **CLI (the portable key + the seam).** A new shared, pure key renderer + replaces `sweep_member_key`. The per-member closure captures the binder's + `varying_axes()` set and renders the member key from the *varying* axes of + `zip_params(&space, point)`, in param-space slot order, through a portable + sanitiser. Filesystem-portability (charset, case, length) lives here — it is a + presentation concern of the on-disk trace store, not the engine's. + +- **CLI + aura-std (the demo strategy).** A new `LongOnly` node in `aura-std` + carries the first `bool` *param*; a new `momentum` strategy composite in the + CLI wires `Ema → Sub → Exposure → LongOnly → SimBroker`; and `aura sweep` gains + a `--strategy ` selector so the new strategy is swept and traced + the same way the built-in SMA sweep is. + +### The portable key — the rules that make it conformant + +A member key is **one path component** under `runs/traces///`. It must +be valid on Linux, Windows, and macOS, and must not silently collide on +case-insensitive filesystems (NTFS / APFS default). The rules: + +- **Charset:** the rendered key contains only `[A-Za-z0-9._-]`. Every other byte + (the Windows-forbidden `<>:"/\|?*`, shell/cloud-sync-hostile `=,` `[]`, spaces, + control chars) is mapped to `_` by a single `sanitize_component` pass. This set + is also URL-path-safe (member keys appear as URL segments when served) and + survives cloud-sync clients (OneDrive/Dropbox) that reject more than the bare + filesystem does. +- **Case-less values → no case-insensitive-FS collision.** Within one family the + axis *names* are fixed (they are the grid's axes); two members differ only in + axis *values*. Values are rendered case-lessly — `i64`/`timestamp` as decimal + digits, `bool` as `true`/`false`, `f64` via Rust's `f64 Display` (decimal, with + **no** scientific notation) — so two distinct members can never differ only by + letter case. Therefore they never collapse to one directory on a + case-insensitive filesystem. +- **Collision-free by construction.** Distinct grid points differ in ≥1 varying + axis value at the same slot. Each value renders injectively per kind (Rust's + `f64 Display` is the shortest round-tripping decimal; integers/bools are + trivially injective), and the token structure (same names, same order, same + separators) is identical across a family's members — so distinct points yield + distinct keys. +- **Bounded length.** The key is capped at a fixed limit (`MAX_KEY = 200` bytes, + comfortably under the 255-byte POSIX `NAME_MAX` and inside Windows' 260-char + `MAX_PATH` once the `runs/traces//` prefix and `/.json` suffix are + added). A key within the cap is used verbatim. A key that would exceed it + degrades to a conformant, collision-resistant fallback: `h-<16-hex>` where the + hex is FNV-1a-64 of the full untruncated readable key (a fixed, version-stable, + non-cryptographic byte hash — chosen so the on-disk name never drifts across + Rust releases the way `DefaultHasher` may). The built-in demo grids stay far + under the cap, so the fallback never fires for them; it exists so the + *generic* promise holds for an arbitrarily large grid. + +### Token grammar + +For each varying axis, in param-space slot order, a token +`sanitize_component(name)` + `-` + `sanitize_component(render_value(value))`; +tokens joined by `_`. Example (the new strategy's grid): +`ema.length-5_exposure.scale-0.5_longonly.enabled-true`. + +## Concrete code shapes + +### (1) The user-facing program — the new strategy, authored and swept + +The new demo strategy a user authors (CLI-local, mirroring the existing +`sma_cross` / `macd` sample builders), and the command that sweeps + traces it: + +```rust +/// EMA-distance momentum: the signal is how far price sits above/below its own +/// EMA (a mean-reversion/trend score), sized by Exposure, then passed through a +/// long-only gate. Three swept knobs of three kinds: `ema.length` (i64), +/// `exposure.scale` (f64), `longonly.enabled` (bool). Nothing in common with the +/// SMA-cross demo — the generic-sweep proof. +fn momentum_blueprint_with_sinks() -> (Composite, Receiver<…>, Receiver<…>) { + let (tx_eq, rx_eq) = mpsc::channel(); + let (tx_ex, rx_ex) = mpsc::channel(); + let mut g = GraphBuilder::new("momentum"); + let ema = g.add(Ema::builder()); // default path segment: ema.length + let dist = g.add(Sub::builder()); // price - ema (momentum score) + let expo = g.add(Exposure::builder()); // exposure.scale + let gate = g.add(LongOnly::builder()); // longonly.enabled (the bool param) + let broker = g.add(SimBroker::builder(SYNTHETIC_PIP_SIZE)); + let eq = g.add(Recorder::builder(vec![ScalarKind::F64], Firing::Any, tx_eq)); + let ex = g.add(Recorder::builder(vec![ScalarKind::F64], Firing::Any, tx_ex)); + let price = g.source_role("price", ScalarKind::F64); + g.feed(price, [ema.input("series"), dist.input("lhs"), broker.input("price")]); + g.connect(ema.output("value"), dist.input("rhs")); // momentum = price - ema + g.connect(dist.output("value"), expo.input("signal")); + g.connect(expo.output("exposure"), gate.input("exposure")); + g.connect(gate.output("exposure"), broker.input("exposure")); // long-only exposure -> broker + g.connect(broker.output("equity"), eq.input("col[0]")); + g.connect(gate.output("exposure"), ex.input("col[0]")); // record the FINAL (gated) exposure + (g.build().expect("momentum wiring resolves"), rx_eq, rx_ex) +} +``` + +Param-space (lowering order): `[ema.length: I64, exposure.scale: F64, +longonly.enabled: Bool]` — the path segments are the nodes' default names +(lowercased builder labels), exactly as `exposure.scale` already arises in the +SMA sample. + +The invocation and its observable result: + +```console +$ aura sweep --strategy momentum --trace mom +{"family_id":"mom-0","report":{…}} # one line per grid point (8 points) +… 8 lines … +$ ls runs/traces/mom/ +ema.length-5_exposure.scale-0.5_longonly.enabled-true +ema.length-5_exposure.scale-0.5_longonly.enabled-false +ema.length-5_exposure.scale-1_longonly.enabled-true +… +ema.length-10_exposure.scale-1_longonly.enabled-false +$ aura chart 'mom/ema.length-5_exposure.scale-0.5_longonly.enabled-true' + … uPlot page … +``` + +The grid: `ema.length ∈ {5,10} × exposure.scale ∈ {0.5,1.0} × longonly.enabled ∈ +{true,false}` = 8 members, **all three axes varying** — so all three appear in +every member key, the bool among them. This is the criterion's evidence: a user +authoring a genuinely different strategy reaches for exactly the same `aura +sweep`, and gets correct, portable, self-describing member directories for free. + +### (2) The portable key renderer (replaces `sweep_member_key`) + +Before — `main.rs:186-196`, two hardcoded axes, panics on drift: + +```rust +fn sweep_member_key(named: &[(String, Scalar)]) -> String { + let val = |name: &str| named.iter().find(|(n,_)| n == name) + .unwrap_or_else(|| panic!("… missing the key axis '{name}'")).1.as_i64(); + format!("f{}s{}", val("signals.trend.fast.length"), val("signals.trend.slow.length")) +} +``` + +After — generic over the *varying* axes, portable, collision-free: + +```rust +const MAX_KEY: usize = 200; + +/// Map any byte outside the portable directory-name charset `[A-Za-z0-9._-]` to +/// `_`. The single source of filesystem-portability for an on-disk path component. +fn sanitize_component(s: &str) -> String { + s.chars().map(|c| if c.is_ascii_alphanumeric() || matches!(c, '.' | '_' | '-') { c } else { '_' }).collect() +} + +/// Render a scalar value case-lessly: integers/timestamps as decimal digits, +/// bool as `true`/`false`, f64 via Rust's decimal `Display` (no sci-notation). +fn render_value(v: &Scalar) -> String { + match v { + Scalar::I64(n) => n.to_string(), + Scalar::F64(x) => x.to_string(), // Rust f64 Display: decimal, shortest round-trip + Scalar::Bool(b) => b.to_string(), // "true" / "false" + Scalar::Timestamp(t) => t.to_string(), // epoch i64 digits, never a colon-bearing date + } +} + +/// The portable, collision-free member key for a grid point: a `name-value` token +/// per *varying* axis (slot order), joined by `_`. `varying` is the binder's +/// `varying_axes()` set; pinned singletons carry no information and are omitted. +/// Over the cap, degrade to a conformant FNV-1a fallback so the key stays one +/// valid path component for ANY grid. +fn member_key(named: &[(String, Scalar)], varying: &HashSet) -> String { + let key: String = named.iter() + .filter(|(n, _)| varying.contains(n)) + .map(|(n, v)| format!("{}-{}", sanitize_component(n), sanitize_component(&render_value(v)))) + .collect::>() + .join("_"); + let key = if key.is_empty() { "m".to_string() } else { key }; // 1-point grid: no varying axis + if key.len() <= MAX_KEY { key } else { format!("h-{:016x}", fnv1a64(key.as_bytes())) } +} +``` + +Worked examples (the unit-test corpus): + +| varying named pairs | key | +|---|---| +| `[(ema.length, I64 5), (exposure.scale, F64 0.5), (longonly.enabled, Bool true)]` | `ema.length-5_exposure.scale-0.5_longonly.enabled-true` | +| `[(longonly.enabled, Bool false)]` | `longonly.enabled-false` | +| `[(exposure.scale, F64 -0.5)]` | `exposure.scale--0.5` (leading `-` is a value sign; `-` is in the portable set) | +| `[(weird key!, F64 1.0)]` (sanitised name) | `weird_key_-1` | +| `[(t, Timestamp 1725148800000)]` | `t-1725148800000` (epoch digits, no `:`/`T`/`Z`) | + +`fnv1a64` is a fixed FNV-1a-64 over the bytes (offset `0xcbf29ce484222325`, prime +`0x100000001b3`) — a ~5-line, version-stable, non-security disambiguator for the +rare over-cap key. (Per the project dependency policy this is the per-case +judgement: a tiny fixed algorithm whose output must be stable *on disk* across +toolchain versions, where a hand-written FNV is the simplest vetted-equivalent +and `std`'s `DefaultHasher` is explicitly unstable across releases.) + +### (3) The `LongOnly` node (aura-std) — the first bool param + +New file `crates/aura-std/src/longonly.rs`, mirroring `exposure.rs` exactly; the +bool *param* drives `eval`: + +```rust +/// Long-only exposure gate. The bool param `enabled`: true clamps short +/// (negative) exposure to 0 (long-only); false passes exposure through unchanged. +/// One f64 input, one f64 output. Emits None until its input is present (C8). +pub struct LongOnly { enabled: bool, out: [Cell; 1] } + +impl LongOnly { + pub fn new(enabled: bool) -> Self { Self { enabled, out: [Cell::from_f64(0.0)] } } + + pub fn builder() -> PrimitiveBuilder { + PrimitiveBuilder::new( + "LongOnly", + NodeSchema { + inputs: vec![PortSpec { kind: ScalarKind::F64, firing: Firing::Any, name: "exposure".into() }], + output: vec![FieldSpec { name: "exposure".into(), kind: ScalarKind::F64 }], + params: vec![ParamSpec { name: "enabled".into(), kind: ScalarKind::Bool }], + }, + |p| Box::new(LongOnly::new(p[0].bool())), // first use of Cell::bool() in a builder + ) + } +} + +impl Node for LongOnly { + fn lookbacks(&self) -> Vec { vec![1] } + fn eval(&mut self, ctx: Ctx<'_>) -> Option<&[Cell]> { + let w = ctx.f64_in(0); + if w.is_empty() { return None; } + self.out[0] = Cell::from_f64(if self.enabled { w[0].max(0.0) } else { w[0] }); + Some(&self.out) + } + fn label(&self) -> String { format!("LongOnly({})", self.enabled) } +} +``` + +Exported from `aura-std/src/lib.rs` (`mod longonly; pub use longonly::LongOnly;`). + +The end-to-end bool path the cycle *proves* (it is not assumed): the type +machinery is already in place and ratified — `ScalarKind::Bool`, `Scalar::Bool` / +`as_bool`, `Cell::from_bool` / `Cell::bool`, and `GridSpace::new`'s per-value +kind-check that already classifies `Bool` (see the green +`random_space_new_non_numeric_slot`, which shows `Bool` is a known, handled kind +on the *random* side). No node has yet *used* a bool param, so this is the first; +its node test and the momentum bool-axis sweep test are what ratify the +authoring + enumeration + bootstrap path for a bool param. + +### (4) The `--strategy` selector (a pure parse fn, mirroring `parse_real_args`) + +Before — three fixed arms (`main.rs:993-995`): + +```rust +["sweep"] => run_sweep("sweep", false), +["sweep", "--name", n] => run_sweep(n, false), +["sweep", "--trace", n] => run_sweep(n, true), +``` + +After — one arm + a pure, unit-testable grammar; `run_sweep` gains a strategy: + +```rust +#[derive(Clone, Copy, PartialEq, Debug)] +enum Strategy { SmaCross, Momentum } + +/// `sweep [--strategy ] [--name | --trace ]`. Defaults: +/// SMA-cross, name "sweep", no persist (today's behaviour preserved). `--name` +/// and `--trace` are mutually exclusive. Pure (no I/O / exit) so it is testable. +fn parse_sweep_args(rest: &[&str]) -> Result<(Strategy, String, bool), String> { … } + +// dispatch: +["sweep", rest @ ..] => match parse_sweep_args(rest) { + Ok((strategy, name, persist)) => run_sweep(strategy, &name, persist), + Err(msg) => { eprintln!("aura: {msg}"); std::process::exit(2); } +}, + +fn run_sweep(strategy: Strategy, name: &str, persist: bool) { + let family = match strategy { + Strategy::SmaCross => sweep_family(persist.then_some(name)), + Strategy::Momentum => momentum_sweep_family(persist.then_some(name)), + }; + // … unchanged: append_family + print each point's record line … +} +``` + +`momentum_sweep_family` mirrors `sweep_family`: declare the 3-axis grid via the +`.axis()` chain, capture `binder.varying_axes()`, and in the closure render the +key with `member_key(&zip_params(&space, point), &varying)`. `sweep_family` +(SMA) is refactored the same way — it drops the hardcoded `sweep_member_key` and +uses the generic `member_key` with its own `varying_axes()` (so its member dirs +change from `f2s4` to `signals.trend.fast.length-2_signals.trend.slow.length-4`). + +### Seam: `SweepBinder::varying_axes` (engine, additive) + +```rust +impl SweepBinder { + /// The names of the axes that vary (more than one value) — the axes that + /// distinguish one grid point from another. Declaration order; the caller + /// derives membership, key token order comes from param-space slot order. + pub fn varying_axes(&self) -> Vec { + self.axes.iter().filter(|(_, v)| v.len() > 1).map(|(n, _)| n.clone()).collect() + } +} +``` + +The CLI captures it *before* the consuming `.sweep()`: +`let binder = bp.axis(…)….axis(…); let varying: HashSet = +binder.varying_axes().into_iter().collect(); binder.sweep(move |point| { … })`. + +## Components + +- **`aura-engine` `SweepBinder::varying_axes`** — new pure read accessor + (`blueprint.rs`). Additive; no existing signature changes. +- **`aura-std` `LongOnly`** — new node + module + export. First bool-param node. +- **`aura-cli`** — + - `sanitize_component`, `render_value`, `member_key`, `fnv1a64`, `MAX_KEY` + (replace `sweep_member_key`); + - `momentum_blueprint_with_sinks`, `momentum_sweep_family`, a momentum point + grid + (re-used) synthetic stream; + - `Strategy` enum, `parse_sweep_args`, `run_sweep(strategy, name, persist)`; + - `sweep_family` / `sweep_over` refactored to the generic key via + `varying_axes()`; + - `USAGE` updated to show `aura sweep [--strategy ] [--name |--trace ]`. + +## Data flow + +Unchanged through the sweep engine (enumerate → disjoint parallel runs → +`SweepFamily` in odometer order). The only new flow is in the per-member closure: +`point: &[Cell]` → `zip_params(&space, point)` → filter by the captured +`varying` set → `member_key` → portable dir component → `persist_traces( +"{name}/{key}", …)` → `runs/traces///.json`. The MC (`seed{seed}`) +and walk-forward (`oos{from}`) keys are already charset-conformant; they are left +as-is this cycle (single-axis, trivially portable). `sanitize_component` is the +one shared portability gate the sweep key flows through. + +## Error handling + +- A 1-point grid (no varying axis) → `member_key` returns `"m"` (a fixed valid + component) rather than the empty string. Specified, tested. +- An over-`MAX_KEY` key → FNV-1a fallback (`h-`), conformant and + collision-resistant. Specified, tested with a synthetic many-axis case. +- `parse_sweep_args` rejects an unknown flag, a flag missing its value, an + unknown strategy token, or both `--name` and `--trace` — `Err(usage)`, surfaced + as stderr + `exit(2)` (the CLI's convention), never a panic. +- The old fail-loud-on-missing-axis panic is **removed** with `sweep_member_key`: + it guarded a hardcoded-axis assumption that no longer exists. Its two unit tests + are removed with it. + +## Testing strategy + +- **`member_key` unit tests (aura-cli `--bin aura`)** — the worked-examples table: + charset conformance (`key.chars().all(|c| c.is_ascii_alphanumeric() || matches!(c, '.'|'_'|'-'))`), + the bool case, a negative-float case, a sanitised-name case, the empty→`"m"` + case, and an over-cap case (key length ≤ `MAX_KEY`, still conformant, two + distinct over-cap inputs → distinct keys). +- **Collision-freedom over a different-axis grid (the #105 regression)** — a + grid that varies an axis set *other than* the old `trend.fast`/`trend.slow` + (e.g. the momentum grid, which varies `exposure.scale` and a bool) yields N + distinct member keys for N points. The old `f…s…` key would have collapsed + them; the new key does not. +- **Determinism** — two runs of the momentum sweep produce byte-identical member + keys (the family + its keys are a pure function of the build, C1). +- **The bool axis in the key** — the momentum sweep's member dirs include both + `…_longonly.enabled-true` and `…_longonly.enabled-false`. +- **`SweepBinder::varying_axes` (aura-engine)** — a grid with mixed singleton and + multi-value axes returns exactly the multi-value names. +- **`LongOnly` node (aura-std)** — `enabled=true` clamps a negative input to 0 + and passes a positive through; `enabled=false` passes both through; `None` + until input present. Plus `enabled` is read from the bool param + (`LongOnly::builder()` schema param is `enabled: Bool`). +- **Integration (aura-cli `--test cli_run`, spawn-with-temp-cwd)** — `aura sweep + --strategy momentum --trace mom` persists 8 member dirs whose names match the + portable charset and include the bool; `aura chart 'mom/'` emits a uPlot + page for one. The existing SMA `--trace` integration test is **updated** from + the `f2s4`-style assertion to the new portable key form. +- **Backward-compat** — the existing `["sweep"]` / `--name` / `--trace` forms + still work through `parse_sweep_args` (default SMA), so the existing sweep + integration + determinism tests stay green (with the key-string assertions + updated to the new form). + +## Acceptance criteria + +The project's feature-acceptance criterion (audience reaches for it; improves +correctness; reintroduces no failure the core constraints forbid): + +- **Audience reaches for it.** A researcher authoring a new strategy with new + params (incl. a bool) runs the *same* `aura sweep` and gets correct, + self-describing, portable per-member traces — shown by the worked momentum + example. No per-strategy key code. +- **Improves correctness.** Closes #105: the key is collision-free for any grid, + not just the built-in two-axis one. Removes the latent silent-overwrite. +- **Reintroduces no forbidden failure.** Determinism (C1) preserved — keys are a + pure function of the grid; parallelism across members unchanged. No look-ahead, + no engine merge, no per-eval emission added (C2/C3/C8). The engine change is a + pure read accessor; trace state stays out of the engine (C-consistency with + #104). +- **Concrete, conformant, generic.** Member keys match `^[A-Za-z0-9._-]+$`, are + case-insensitive-FS safe, length-bounded, and carry the bool axis — proven by + the test corpus above over a grid that varies a different axis set than the old + hardcoded two.