Architect drift review (c192dfd..a295905): drift_found, medium/low only — no
ledger or hot-path contract crossed. C23-additive, C1-deterministic, C14/C18
back-compat all verified green (full workspace suite + clippy). No regression
scripts configured (test+clippy are the gate).
Resolution:
- [medium] three spec-promised optimize_deflated tests were missing (the
centring control, the all-noise high-p companion, the C2 IS-only read) — the
centring was unprotected ("removing it would pass every test"). Plus an
unguarded dispersion-floor sign assumption. FIXED in 0076.tidy (ecc9541).
- [low, RATIFY] the ledger's "Open architectural threads" inferential entry was
stale on one fact (it described optimize as having no trials penalty). Synced:
it now records that #144's trials-deflation landed this cycle (additive
optimize_deflated, winner unchanged — C23; optimize/rank_by stay bare argmax),
that #139's per-candidate OOS bootstrap landed in 0075, and that #145/#146
remain open. This ratifies the cycle-0076 intent; the gap description is
preserved for the still-open default-ranker path.
Deferred (spec 0076 Out of scope, not drift): n_eff effective-trials advisory,
reload-time recompute of the R-arm statistic, deflated-score re-ranking, CLI
flags for the resampling config.
Ephemeral spec + plan (docs/specs/0076-*, docs/plans/0076-*) removed per the
cycle-close convention. refs #144
Audit (architect, cycle 0076) found three tests the spec's testing strategy
promised but the implement-loop did not land, plus an unguarded sign assumption:
- optimize_deflated_centres_the_null_so_a_uniform_edge_is_not_called_overfit:
the load-bearing protection for the mean-subtraction. A family where every
member shares the same +R edge reads as NOT overfit only because the null is
centred; an uncentred null would give ≈0.5. Removing the centring now flips a
test (it previously passed every test — the architect's headline finding).
- optimize_deflated_all_noise_family_has_high_overfit_probability: the high-p
companion to the existing clear-edge low-p test.
- optimize_deflated_provenance_reflects_only_the_passed_family: C2 at the
registry boundary (n_trials == family size, raw == the family's own argmax),
pinning that the record encodes only in-sample information.
- A debug_assert in the dispersion-floor arm: it subtracts inflation, valid only
for higher-is-better metrics; lower-is-better (max_drawdown/bias_sign_flips)
would deflate wrong-signed. Unreached from the two shipped call sites, but
optimize_deflated is public — guard the assumption rather than leave it implicit.
cargo test -p aura-registry (33) + clippy green. refs #144
Adds optimize_deflated beside optimize in aura-registry: the in-sample
walk-forward selection now records, on each OOS winner's manifest, how much
the winner's metric is inflated by the size of the search it won — a deflated
score and (R arm) an overfit probability — without changing which member wins
(additive, C23; a regression test pins optimize_deflated's winner == optimize's).
- R arm (sqn_normalized/expectancy_r/...): a centred moving-block reality-check.
Each member's per-trade R series is mean-subtracted to impose the no-edge null,
resampled via the r_bootstrap kernel (extracted to a shared resample_block,
byte-identical), and the best-of-K null-max gives
overfit_probability = (count(null >= raw)+1)/(n+1) and
deflated_score = raw - p95(null). Deterministic given the seed (C1).
- total_pips arm: a closed-form expected-max-of-K dispersion floor
(inv_norm_cdf + expected_max_of_normals), no probability.
The record (FamilySelection on RunManifest) rides the proven
serde(default, skip_serializing_if) widening + a compat.rs mirror, so legacy
runs.jsonl/families.jsonl lines load unchanged and unstamped sweep/mc/standalone
manifests stay byte-identical (C14/C18). The SelectionMode enum reserves a
Plateau slot for #145. CLI: walk-forward stamps both arms; runs family <id> rank
surfaces a human-readable deflated/overfit line.
Quality review (re-dispatched after the implement-loop's quality gate exhausted)
caught a real defect: on the R arm with positive resamples but no member carrying
trade_rs (a family loaded from disk — trade_rs is serde-skipped), the null was
all-NEG_INFINITY, turning deflated_score into +inf. Fixed by collapsing "no usable
null" (zero resamples OR no trade_rs) into one degenerate floor (deflated == raw,
overfit == 1.0); pinned by optimize_deflated_no_trade_rs_floors_instead_of_infinity.
Verified: cargo test --workspace and cargo clippy --workspace --all-targets
-D warnings both green. Frictionless Stage-1 R; n_eff advisory and reload-time
recompute deferred (spec 0076 Out of scope).
closes#144
First cycle of the inferential-validation milestone. Adds optimize_deflated
beside optimize: an additive selection-provenance record (deflated score +
overfit probability) stamped on the OOS winner's manifest, never re-ranking
(C23). R arm = centred moving-block reality-check reusing the r_bootstrap
kernel; total_pips arm = expected-max-of-K dispersion floor. Shared
RunManifest.selection carrier with a SelectionMode::Plateau slot reserved
for #145.
refs #144
Architect drift review (c99e7b8..2e77ab1) against the design ledger +
CLAUDE.md invariants: drift-clean, debt only — no contract or spec
violation. Verified intact: C18 wire shape (trade_rs is #[serde(skip)] +
excluded from the hand-written PartialEq; runs.jsonl/families.jsonl bytes
and digests unmoved; mc_r_bootstrap is a printed line, never a record),
C1 determinism (seeded r_bootstrap + disjoint walkforward windows, both
pinned by byte-identical-rerun e2e tests), C7 frictionless Stage-1
(round_trip_cost stays 0.0), C8/C12 streaming reduction (IS sweep folds
O(trades)/member, OOS run non-reduce). The verbatim summarize_r /
r_metrics_from_rs float duplication is guarded by the cross-reducer
equality test — sound.
Tidy applied this close:
- doc-link drift: the new E[R] notation in r_bootstrap / RBootstrap /
run_mc_r_bootstrap doc-comments was parsed by rustdoc as an intra-doc
link `[R]` (4 warnings). Wrapped in backticks; doc build now clean.
- coverage micro-gap [low, architect]: added a tripwire asserting a
populated trade_rs is absent from the serialized JSON (the serde(skip)
C18-preservation property had no direct test — only Vec::new() was
serialized before).
Carry-on (benign debt, not fixed): the mc usage literal carries a `usage:`
prefix the sibling sweep/walkforward usages omit — required because the
new `["mc", rest @ ..]` arm now handles the `--real` rejection and the
preserved golden asserts stderr.contains("usage"); churning the golden to
drop the prefix is not worth it.
Removed the cycle's ephemeral spec + plans (docs/specs/0075,
docs/plans/0075, docs/plans/0076) per the lifecycle rule.
Verified: cargo test --workspace 610 passed / 0 failed, clippy -D warnings
clean, cargo doc --no-deps clean.
closes#139
Family-selection across a sweep has no false-discovery control (no
trials-deflation, plateau-over-peak, or cross-instrument generalization);
recorded as the World's third unbuilt half. refs #144#145#146#139
Add `aura mc --strategy stage1-r [--real <SYM>]`: run the stage1-r
walk-forward, pool its OOS per-trade R series, and report a moving-block
bootstrap E[R] distribution / CI (one `mc_r_bootstrap` line with an `e_r`
quantile block + P(E[R]<=0)). Completes spec 0075's Monte-Carlo half on top
of iter 1's walk-forward R-reporting.
Engine: `r_bootstrap(rs, n_resamples, block_len, seed) -> RBootstrap` — a
deterministic moving-block bootstrap (non-circular, final block truncated;
block_len=1 = i.i.d. trade-shuffle; clamped to [1,n]; empty -> all-zero),
reusing the existing MetricStats/quantile and SplitMix64.
CLI: parse_mc_args/McArgs split — bare/`--name`/`--trace` route to the
unchanged synthetic seed-resweep `run_mc` (byte-for-byte preserved); the R
path accepts ONLY `--strategy stage1-r`, so a bare `--real` stays a usage
error (mc_rejects_real_flag_with_usage_exit_2 preserved, doc-comment
refreshed). run_mc_r_bootstrap reuses walkforward_family + RMetrics.trade_rs.
Corrective fixes over the plan snippets, both verified: the mc usage string
carries a `usage:` token (the new `["mc", rest @ ..]` arm now handles the
`--real` rejection, and the preserved golden asserts `stderr.contains
("usage")`); `#[allow(clippy::large_enum_variant)]` on McArgs keeps the
unboxed Stage1RGrid field layout.
Scope: single `--real <SYM>` per invocation; cross-symbol pooling is a
deferred follow-on (needs a multi-symbol grammar). Verified: cargo build
clean, full `cargo test --workspace` green (609 passed), clippy -D warnings
clean; bare `aura mc`, SMA walkforward, stage1-r single-run, and C18
round-trip goldens all preserved.
refs #139
Iteration 2 of cycle 0075: the engine primitive r_bootstrap (moving-block
bootstrap over a flat R series, reusing MetricStats/SplitMix64); the CLI
parse_mc_args/McArgs split (synthetic seed-resweep preserved byte-for-byte;
the --strategy stage1-r real-R path reuses iter-1's walkforward_family +
RMetrics.trade_rs); run_mc_r_bootstrap prints one mc_r_bootstrap line. The
mc real-R path accepts only --strategy stage1-r, so a bare --real stays a
usage error (mc_rejects_real_flag_with_usage_exit_2 preserved).
refs #139
Make `aura walkforward --strategy stage1-r [--real <SYM>]` roll IS->OOS
windows, sweep the stage1-r grid in-sample, pick the winner by an R metric
(sqn_normalized), run it out-of-sample, and report per-window + pooled OOS
R-metrics. The bare SMA walkforward path is byte-identical (its dispatch arm
is verbatim today's body; the pooled `oos_r` block is emitted only when a
window carries an `r` block).
Engine:
- RMetrics gains an in-memory `trade_rs: Vec<f64>` (realised R per closed
trade), excluded from serde (`#[serde(skip)]`) and from a hand-written
PartialEq, so the C18 wire shape and every existing equality assertion /
round-trip stay unchanged. summarize_r now retains the per-trade R vector
it used to drop.
- New `r_metrics_from_rs(&[f64])` reduces a flat (pooled across-window) R
series to RMetrics. Its R-distribution arithmetic is copied verbatim from
summarize_r (the byte-pinned floats must not be algebraically refactored);
the two copies are guarded in lockstep by a cross-reducer equality test.
net_expectancy_r = expectancy_r (exact at the Stage-1 cost=0 invariant);
conviction_terciles_r = [0,0,0] (per-trade conviction is not pooled).
CLI:
- walkforward gains --strategy + the four stage1-r grid flags (reusing
parse_csv_list / Stage1RGrid); walkforward_family is strategy-dispatched.
- Windowed stage1-r helpers: stage1_r_sweep_over (reduce-mode folded IS
sweep, O(trades)/member) and run_oos_r (non-reduce OOS run, for the
stitched pip-equity curve), plus stage1_r_space.
Frictionless Stage-1 R (costs are Stage-2). Verified: cargo build clean,
full `cargo test --workspace` green (SMA walkforward, synthetic mc,
stage1_r_single_run_output_golden, and the C18 round-trips all preserved),
clippy -D warnings clean. Monte-Carlo R-bootstrap is iter 2.
refs #139
Make the World layer's two OOS validation verbs candidate-aware and
R-reporting instead of locked to the SMA sample blueprint:
- aura walkforward --strategy stage1-r --real <SYM>: roll IS->OOS, sweep
the stage1-r grid in-sample, pick by an R metric, run OOS, report
per-window + across-window-pooled OOS R-metrics.
- aura mc --strategy stage1-r --real <SYM>: moving-block bootstrap of the
pooled OOS per-trade R series -> E[R] distribution / CI.
Frictionless Stage-1 R (costs are Stage-2). Bare walkforward/mc paths and
their goldens preserved byte-for-byte. Grounding-check PASS; signed under
/boss (design forks recorded on the reference issue).
refs #139
Architect audit: drift-clean. The ledger and source now point at the recorded
geometry sidecar; no live reference to the removed authored floor survives in
code or any live contract. Refuse-don't-guess preserved.
Recorded debt (non-blocking, accepted):
- fieldtests/milestone-runway references the removed instrument_spec; a frozen
non-workspace snapshot, left untouched per the tracked-fieldtests convention.
- cli_run.rs test fn names retain unspecced/vetted vocabulary (plan prescribed
substring-only edits); cosmetic.
#124 stays open for the deferred override/floor tiers the Stage-2 broker needs.
Drops the ephemeral cycle spec + plan (rationale lifted to the ledger).
The data-server now ships per-symbol geometry sidecars, so aura no longer
hard-codes instrument geometry. A consumption grep showed production reads
exactly one `InstrumentSpec` field — `pip_size` — at the real-data path; the
other eight (incl. `instrument_id`) were constructed but never read.
- aura-cli resolves the real-path pip from the recorded sidecar
(`instrument_geometry` -> `InstrumentGeometry.pip_size`), refusing (exit 2)
on absent geometry. The authored table no longer gates: any symbol with a
sidecar + bars now runs (new test: USDJPY, never vetted, resolves its JPY
pip 0.01 from the sidecar).
- Removed the redundant authored floor: `InstrumentSpec`, `instrument_spec`,
`VETTED_SYMBOLS`, and the table-only tests. The `InstrumentGeometry` seam
(the surviving pip source) and its nameability guard are kept.
- `instrument_id` (the "find out"): no production consumer. The position-event
table's `instrument_id` is a decoupled caller parameter; the engine never
imports an instrument spec. Dropped with the struct.
- The example `pip_size_of` is re-sourced to the sidecar; the pure-structural
blueprint tests use a literal pip to stay archive-independent.
Speculative deploy-grade fields and the #124 override/floor tiers are deferred
to the Stage-2 broker that will consume them, read from the sidecar geometry
then. Ledger (C8/C15/#22/#106) updated; #124 collapses to "recorded sidecar
geometry -> refuse" in the meantime.
Verified: cargo test --workspace (all green), clippy -D warnings clean, cargo
doc clean, with the local archive present so the sidecar paths run.
refs #124
Resolve the real-path pip from the data-server geometry sidecar
(instrument_geometry) and remove the redundant authored floor
(InstrumentSpec / instrument_spec / VETTED_SYMBOLS) plus its table-only
tests. Consumption grep: only pip_size is read in production;
InstrumentSpec.instrument_id has no consumer (the position-event table's
id is a decoupled caller parameter). Speculative geometry fields are
deferred to the Stage-2 broker, read from the sidecar when needed.
Signed via grounding-check PASS (all assumptions tied to currently-green
tests, executed with the archive present). Decisions recorded on #124;
#140 closed as obsolete.
refs #124
Cycle-close audit: drift-clean on all load-bearing invariants (C7/C8/C15 hold; geometry stays non-scalar metadata, the Source seam still carries only (Timestamp, Scalar); scope boundary held — no resolver, quote_currency still &'static str, both #124's).
Tidy: completed the VETTED_SYMBOLS consolidation the feat commit over-claimed — migrated the two remaining inline vetted-symbol arrays in the crate-internal unit tests (instrument_spec_pip_value_matches_contract_times_pip, instrument_spec_ids_are_distinct) to the const, so #140 now extends one list.
Ledger: lifted this cycle's durable rationale into the C15 realization note (docs/design/INDEX.md) — the floor now carries tick_size/digits, and the recorded-metadata tier-2 source (instrument_geometry over symbol_meta -> neutral InstrumentGeometry, cross-checked against the authored floor) now exists at the ingestion edge; the tier-composing resolver remains #124.
Drop the ephemeral cycle spec + plan (docs/specs, docs/plans) per the lifecycle convention; durable rationale now in the ledger + the #143 thread.
refs #143
Make data-server's neutral InstrumentGeometry a first-class, broker-agnostic input at the aura-ingest edge: re-export InstrumentGeometry + a thin instrument_geometry(server, symbol) accessor over symbol_meta (the recorded dataset-metadata tier 2 of #124's hierarchy; the tier-composing resolver stays #124). Add the two deferred deploy-grade InstrumentSpec fields tick_size + digits (additive, struct stays Copy; C15), authored for the 5 vetted symbols from the verified sidecars. A new real-data cross-check asserts the authored floor agrees with provider geometry (pip_size, contract_size==lot_size, the C10 pip_value invariant, tick_size, digits, quote_currency), iterating the vetted set so #140's additions are covered for free; skip-if-archive-absent.
Scope (derived, recorded on #143): no runtime resolver and no quote_currency type change — both remain #124's. 'quote_currency beyond FX' is satisfied at the geometry tier (InstrumentGeometry.quote: String), not by widening the &'static str floor.
Beyond the plan (implementer additions, reviewed and accepted): hoisted the vetted-symbol list to one pub const VETTED_SYMBOLS, collapsing the three drift-prone copies plan-recon flagged (#140 now extends one constant); a tick_size == 10^(-digits) consistency guard (verified to hold across all 30 provider sidecars, so it will not false-trip on #140); a re-export-nameable seam test guarding the InstrumentGeometry re-export from silent removal.
Verification: cargo test --workspace all green; cargo clippy --workspace --all-targets -D warnings clean. The cross-check ran against the live Pepperstone archive (not skipped) — all 5 vetted sidecars match the authored table.
refs #143
Three tasks at the aura-ingest edge: (1) re-export InstrumentGeometry + a thin instrument_geometry accessor over symbol_meta; (2) two additive InstrumentSpec fields tick_size/digits (compile-gated literal update); (3) a real-data cross-check of the authored floor vs provider geometry, iterating the vetted set.
refs #143
data-server #3 (closed) shipped DataServer::symbol_meta -> InstrumentGeometry (the per-symbol .meta.json reader). Bump 0f5e665 -> 694f96f2 so aura-ingest can consume the neutral geometry for the #143 cross-check. The bump alone changes no behaviour (the new API is unused until the #143 code lands).
refs #143
Consume data-server's neutral InstrumentGeometry at the aura-ingest edge to cross-check the hand-authored vetted InstrumentSpec floor (#113) against provider truth, and add the two deferred deploy-grade fields (tick_size, digits) the sidecars now supply.
Scope fork derived and recorded on #143: this cycle does NOT build #124's runtime resolver and does NOT change quote_currency's &'static str type (the geometry tier carries non-FX currency as String). Auto-signed autonomously under /boss on a grounding-check PASS (6/6 assumptions ratified); reply to the run to veto.
refs #143
The EWMA Bollinger-band mean-reversion fade is refuted: no cross-index
generalization (FRA40 leans positive, GER40 negative — two correlated indices
disagreeing in sign), no OOS-significant cell on either index, and its lone
IS-significant cell is consistent with multiple-testing noise (full verdict on
#137). The machinery (--strategy stage1-meanrev, the composed EWMA Bollinger
band, zero new nodes) is permanent and kept; only the per-cycle spec/plan are
dropped per the ephemeral-artifact lifecycle convention.
refs #137
Third price-only Stage-1 R strategy: aura sweep --strategy stage1-meanrev fades
deviation from a rolling mean (price above mean+k*sigma -> short, below -> long,
latched +-1), feeding the unchanged bias -> RiskExecutor (vol-stop = R) -> flat-1R
seam. sigma = Sqrt(Ema((price-Ema(price))^2)) reuses the vol_stop deviation-
squared-then-smoothed shape, so the band is built from existing aura-std nodes
(Ema/Sub/Mul/Sqrt/LinComb/Add/Gt/Latch) -> ZERO new nodes. No Delay: the current
bar legitimately belongs to its own band (causal, C2). Swept knobs: --window (one
Ema length ganged across mean+variance) and --band-k; downstream vol-stop R knobs
unchanged.
An in-file test reads the SHIPPED graph's exposure tap to pin the fade polarity
(short above the band, long below) — a latch-set/reset copy-paste from breakout
would invert the signal yet leave the fold-vs-raw and window-grid CLI tests
green. All 588 workspace tests pass (existing R/sma/momentum/breakout goldens
byte-identical); clippy clean.
refs #137
Hand-wired EWMA Bollinger-band fade subgraph (price -> Ema mean + dev -> Mul
square -> Ema var -> Sqrt sigma -> LinComb k-band -> Add/Sub bands -> Gt/Gt ->
Latch/Latch -> Sub bias). Pins the fade direction (above mean+k*sigma -> short,
below -> long), the latch hold, and flat-series -> no fade. Green on arrival
(composes only existing aura-std nodes; no new node), so it is a characterization
pin rather than a RED->GREEN unit.
refs #137
A third price-only Stage-1 R candidate: an EWMA Bollinger-band mean-reversion
fade (price vs rolling mean +/- k*sigma, latched +-1, sign inverted vs breakout).
Composed entirely from existing aura-std primitives (the vol_stop sigma pattern)
-> zero new nodes. Screens the mean-reversion hypothesis under the same #137
lie-detector bar after the two trend candidates (momentum, breakout) were
refuted. Grounding-check PASS (9/9 load-bearing assumptions ratified).
refs #137
The Donchian channel-breakout Stage-1 R candidate is refuted: no cross-index
generalization, violent IS->OOS sign flips at long channels, no significant
positive cell (full verdict recorded on #137). The breakout machinery
(RollingMax / RollingMin nodes, --strategy stage1-breakout) is permanent and
kept; only the per-cycle spec/plan are dropped per the ephemeral-artifact
lifecycle convention.
refs #137
A second Stage-1 R strategy candidate for the edge-research milestone, a structurally
different trend mechanic than the refuted SMA-momentum: a Donchian channel breakout.
Swaps only the signal leg of the stage1-r graph —
close -> Delay(1) -> {RollingMax,RollingMin}(N) -> {Gt,Gt} -> {Latch,Latch} ->
Sub = bias in {-1,0,+1} — keeping the vol-stop R definition unchanged, so it screens
under the identical R yardstick (clean A/B vs momentum).
- aura-std: new RollingMax / RollingMin nodes (sliding-window extremum via a monotonic
deque, O(1) amortized; mirror the Sma/Delay ring-buffer shape, warm-up skip-emit,
C7/C8). The +/-1 direction latch composes from two existing Latch nodes + Sub (no new
latch type).
- aura-cli: stage1_breakout_graph + a manual-grid stage1_breakout_sweep_family
(fully-bound graph per point, sidestepping parameter-ganging since one channel length
drives both rolling nodes); --strategy stage1-breakout + --channel grid flag; usage
strings updated.
- Causality (C2): one Delay(1) on close feeds both rolling nodes, so each channel covers
close[t-N..t-1] — the current bar is excluded. Pinned by a contrastive e2e test (a
strictly rising series up-breaks every warmed bar; a window including the current bar
never could) plus a +/-1-latch-hold test.
All existing goldens byte-identical (stage1-r/sma/momentum untouched); folded-no-trace
== raw-trace metrics; workspace tests + clippy green.
refs #137
Implementation plan for the Donchian breakout candidate (parent spec
docs/specs/0071-stage1-breakout.md). Four dependency-ordered tasks: RollingMax and
RollingMin (monotonic-deque sliding extrema, mirroring the Sma/Delay node shape), a
C2-causality + ±1-latch composition test, and the CLI breakout strategy
(stage1_breakout_graph + manual-grid sweep family + --strategy/--channel wiring).
refs #137
Donchian channel-breakout as the next Stage-1 R research candidate, after the
SMA-momentum family was refuted (cross-index non-generalization + no within-symbol
significance; #137/#141). Swaps only the signal leg of stage1_r_graph:
close -> Delay(1) -> {RollingMax,RollingMin}(N) -> {Gt,Gt} -> {Latch,Latch} ->
Sub = bias in {-1,0,+1}; the vol-stop defines R unchanged (clean A/B vs momentum).
Two new aura-std nodes (RollingMax/RollingMin, monotonic-deque sliding extremum);
the +/-1 direction latch composes from two existing Latch nodes + Sub. New
--strategy stage1-breakout + --channel grid flag; the family iterates the grid
with fully-bound graphs (compile_with_params(&[]) + Harness::bootstrap), sidestepping
parameter-ganging. Delay(1) is the C2 causality guard (channel excludes current bar).
Grounding-check PASS: all 10 load-bearing assumptions ratified by named green tests.
Auto-signed under /boss (grounding-check PASS = the signature).
refs #137
Cycle 0070 (streaming sink reductions, #138) close. Durable rationale lifted to
the ledger: C8 now records the end-of-stream `finalize` lifecycle (the second node
phase beside `eval`), its `Harness::run` topo-order epilogue, and its C1/C7/C8
reconciliation — with the open `Recorder`->`Probe` / per-cycle-allocation question
(#77) cross-referenced, since `finalize` is the "non-channel exit" that issue's
accumulate-then-read option named as missing. The per-cycle spec and plan are
git-rm'd (ephemeral, valid only for their cycle).
Audit (architect) found no invariant breakage; noted debt, left as-is: the
two-mode `stage1_r_graph` (reduce vs raw sink wiring) rests on the CLI-seam
equivalence test rather than structural unification, and `SeriesReducer.last_ts`
carries the last *warm* cycle's ts (latent only if `--trace` ever folds).
refs #138
Completes the M3 fix for BLOCKER #138. The no-trace stage1-r sweep now folds its
reductions online: `stage1_r_graph(reduce=true)` wires `SeriesReducer` (equity,
exposure -> one summary row each) and `GatedRecorder` (gated on
`closed_this_cycle` -> only the trade rows + the final row), and the folded
consumer rebuilds `RunMetrics` from those compact outputs. Per-member memory drops
from O(cycles) (~2 GiB over ~5.5M one-minute bars) to O(trades), so full-history
multi-member sweeps no longer exhaust RAM+swap. The `r_equity` tap is omitted when
reducing (it is consumed only by the `--trace` persistence path).
`reduce = trace.is_none()`: the `--trace` path and the single `run_stage1_r` keep
the raw recorders + `summarize`/`summarize_r` over full rows — the byte-for-byte
reference. A new CLI-seam test
(`sweep_strategy_stage1_r_folded_no_trace_metrics_equal_raw_trace_metrics`) proves
the folded no-trace sweep reports byte-identical per-member metrics to the raw
`--trace` path.
Verified: cargo test --workspace green (42 suites); cargo clippy --workspace
--all-targets -- -D warnings clean; stage1-r goldens byte-identical.
closes#138
M3 machinery for BLOCKER #138: sink reductions can now fold online instead of
buffering every per-cycle row to an unbounded channel.
- aura-core: additive `Node::finalize` end-of-stream hook (default no-op) +
`SeriesFold` online accumulator (last / max_drawdown / sign_flips).
- aura-engine: `Harness::run` calls `finalize` once per node in topo order after
the source loop; `summarize` refactored to drive `SeriesFold` (byte-identical),
the now-dead `sign0` removed.
- aura-std: `GatedRecorder` (emits only gated rows + the final row on finalize)
and `SeriesReducer` (folds one f64 series, emits one summary row on finalize),
both emitting the `Recorder` channel type — no aura-engine dependency.
- Integration tests: folded reductions are byte-for-byte equal to the
raw-recorder path; the R-reduction's peak memory is O(trades), independent of
cycle count.
Ratified two off-plan fixes the implementer made in the tests (both plan defects,
not code defects): the GatedRecorder correctly flushes the final non-gated row on
finalize, so the retained count is K_TRADES + 1; and the heterogeneous PM record
needs kind-correct zero scalars (a blanket f64 zero is rejected by the kind-typed
AnyColumn::push).
Wiring these reducers into the stage1-r sweep (the actual O(cycles)->O(trades)
win) follows next. refs #138
M3 fix for BLOCKER #138: an end-of-stream `finalize` lifecycle on sinks
lets reductions fold online into O(trades)+O(1) state, so full-history
stage1-r sweeps run in bounded memory per member instead of OOM-killing.
Grounding-check PASS; signed autonomously under /boss.
refs #138
Enabling change for the strategy-research run: `aura sweep --strategy
stage1-r` gains four optional comma-separated grid flags `--fast`,
`--slow`, `--stop-length`, `--stop-k` that become the sweep's grid axes
over the four stage1_r_graph knobs (sma fast/slow length, vol-stop
length, vol-stop k). The family is their cartesian product. This lets a
slow time-series-momentum edge be screened across timeframes — Phase 0
showed fast (2/4-bar) signals are chop-dominated noise on M1, so any
momentum edge lives at slower timescales, and the stop timescale must be
tunable alongside the signal.
Mechanism:
- aura-composites: factor vol_stop and risk_executor into a single
shared topology body each (vol_stop_inner / risk_executor_inner), with
the vol-stop knobs either BOUND (the original constant-bind form,
byte-identical) or left OPEN and named `stop_length` / `stop_k` so
their slots surface in param_space as sweepable axes. New
risk_executor_vol_open(risk_budget) is the gridding sibling of
risk_executor(StopRule::Vol{..}, ..). One body, two arms → topology
cannot drift; a member at any (length, k) matches the bound form
byte-for-byte.
- aura-cli: parse_csv_list (generic FromStr, rejects non-numeric/empty),
pure unit-testable parse_sweep_args, all four knobs gridded via
.axis(); absent flags fall back to today's exact defaults (fast {2,3},
slow {6,12}, stop_length {3}, stop_k {2.0}).
Default behaviour byte-identical: all existing goldens/sweep tests stay
green (541 passed, 0 failed; clippy -D warnings clean). Frictionless
Stage-1 R unchanged.
refs #137
Executable-spec for the strategy-research enabling change: expose the
stage1-r sweep's hardcoded timescales as CLI grids so a slow
time-series-momentum edge can be screened across timeframes.
- sweep_strategy_stage1_r_grids_all_four_knobs_from_csv_flags (RED):
`aura sweep --strategy stage1-r --fast 240 --slow 960
--stop-length 240 --stop-k 2.0` must yield a 1-member family whose
member params are fast=240, slow=960, stop_length=240, stop_k=2.0.
Fails today: the four flags are rejected (exit 2), gridding absent.
- sweep_strategy_stage1_r_no_flags_keeps_the_default_grid (guard,
green today): a no-flags sweep still yields the {2,3}×{6,12} 4-member
family with the stop pinned at 3 / 2.0, so existing goldens stay
byte-green once the feature lands.
GREEN side follows via implement mini-mode.
refs #137
Architect audit clean (no drift): the derive is the faithful C10 book
first-difference — Close uses the stored book volume (the actual book), never an
exposure delta (the exact inversion of the abandoned 0064 exposure-integral);
post-run reduction, no in-graph node (C14), positional type-erased Scalar reads
(C7), each event's event_ts is its own cycle (C2), the >1-event-per-instant
reversal case is why it is derived not per-eval (C8/C10); aura-engine -> aura-core
only (instrument_id a caller scalar, no InstrumentSpec import).
- Ledger C10: add a "Realization (cycle 0068, #115)" note — derive_position_events
as the first difference of the executed book, the post-reframe replacement for
the rolled-back 0064 derive; fixed-fractional / currency / equity-feedback
sizing and the realistic brokers stay #116.
- Remove the ephemeral spec + plan (cycle closed; rationale in the ledger + git
history).
Accepted low debt (architect [low]): the stop-then-same-cycle-reopen producer path
emits the same closed&&open shape as a bias-flip reversal and the derive handles it
correctly (it keys on closed/open/direction/size, not exit_reason; unit-covered via
the reversal test), but is not separately asserted end-to-end. Cosmetic; the derive
is producer-agnostic.
refs #115
Add `derive_position_events(record, instrument_id) -> Vec<PositionEvent>` to
aura-engine (report.rs), the broker-independent Stage-2 audit layer of C10: the
first difference of the executed book derived from a PositionManagement dense
record. A pure post-run reduction, sibling of summarize_r — same positional,
type-erased Scalar read of the 14-column record, no in-graph node (the hot path
stays domain-free, C14). A reversal emits Close then the opposite open at one
event_ts (close first); a window-end open position emits its open with no
synthetic Close (the table records actual executed events — unlike summarize_r,
which force-closes for the R metric). instrument_id is a caller-supplied scalar
(aura-engine depends only on aura-core; it cannot import InstrumentSpec).
Two r_col indices added (DIRECTION=4, SIZE=10), the lockstep guard in
stage1_r_e2e.rs extended to name-pin `direction`, and an agreement E2E folds both
summarize_r and derive_position_events over one recorded ledger (one Close per
closed round-trip; every open closed-or-open-at-end; referential integrity).
Scope: the derive only. Fixed-fractional / currency / equity-feedback sizing is
#116 (it owns the equity->Sizer z^-1 register per C10) — deferring it here avoids
a circular dependency. Supersedes the rolled-back 0064 exposure-integral derive
(abandoned per #117): this derives from the executed book, never exposure deltas.
Verified by the orchestrator: cargo build + clippy --all-targets -D warnings
clean; full workspace suite 0 failed; aura-engine lib 214 pass (incl. the 6
derive_* unit tests); stage1_r_e2e 11 pass (incl. the new E2E + extended guard).
Fork decisions recorded on #115.
closes#115
Three tasks for spec 0068: (1) derive_position_events + r_col DIRECTION/SIZE
consts + crate-root re-export + report.rs unit tests (reversal, normal lifecycle,
short, window-end open, empty, monotonic position_id); (2) stage1_r_e2e.rs
lockstep-guard direction pin + an agreement E2E folding both summarize_r and
derive_position_events over one recorded ledger; (3) full workspace
build/test/clippy verification. Purely additive — no existing output touched.
refs #115
Derive the broker-independent position-event table (#114's PositionEvent rows)
from a completed PositionManagement run, as the first difference of the executed
book (deal = target - book - in_flight; in_flight = 0 for instant backtest
fills). A pure post-run reduction `derive_position_events(record, instrument_id)`
in aura-engine, sibling of summarize_r, reading the 14-column PM record
positionally; a reversal emits Close then the opposite open at one event_ts
(close first); a window-end open position emits its open with no synthetic Close.
Supersedes the rolled-back 0064 exposure-integral derive (abandoned per #117):
this derives from the executed book (the PM record), never from exposure deltas.
Stage-2 fixed-fractional / currency / equity-feedback sizing is out of scope
(#116 owns the equity->Sizer z^-1 register per C10).
Boss-signed: specify Step-5 grounding-check PASS (8/8 load-bearing assumptions
ratified by named green tests). Fork decisions recorded on #115.
refs #115
Architect audit clean (no drift, no debt): raw sqn verified bit-identical,
C18 serde back-compat + C4 SoA seam + C1 determinism + Part-B symmetry all hold.
- Ledger C10: add a "Realization (cycle 0067, #130 + #135)" note — SQN100 as
the turnover-robust opt-in rank key (raw sqn / default ranker byte-unchanged,
serde(default) for back-compat), and per-member --trace now symmetric across
all three sweep strategies via the shared persist_traces_r.
- Glossary: R-metrics list gains sqn_normalized; the SQN entry documents the
SQN100 variant (was "deferred #130").
- Remove the ephemeral spec + plan (cycle closed; rationale lives in the ledger
and git history).
refs #130, refs #135
Two additive follow-ups to the cycle-0065/0066 R-sweep surface.
Part A (#130) — n-normalized SQN (SQN100): RMetrics gains sqn_normalized =
(mean_R/stdev_R)·√(min(n,100)), a turnover-robust ranking objective comparable
across sweep members with different trade counts. The raw sqn is kept
byte-identical (computed sqrt(n)*mean/sd, NOT via a factored shared ratio, so
the existing metric does not drift even by 1 ULP). The field carries
#[serde(default)] for C18 back-compat (a pre-0067 r: block deserialises with
sqn_normalized = 0.0). The registry learns Metric::SqnNormalized (opt-in; the
default ranker and raw sqn are unchanged), so `runs family rank sqn_normalized`
works. The 0066 single-run golden is re-baselined (the r: block gains the
field; for n=3 ≤ 100 the cap is inactive so sqn_normalized == sqn).
Part B (#135) — wire --trace for the stage1-r sweep: stage1_r_sweep_family now
honours --trace, persisting each member's equity/exposure/r_equity under
runs/traces/<n>/<member_key>/ via the existing persist_traces_r (mirroring
momentum_sweep_family); the interim run_sweep refusal guard is dropped. --trace
is now symmetric across all three sweep strategies and a swept member is
chartable (chart <n>/<member> --tap r_equity).
Tests: SQN100 cap / below-cap / empty + serde back-compat (engine); rank-by
sqn_normalized + unknown-metric message (registry); an E2E guard that both
SQN values fold from recorded records and agree below the cap (the cross-crate
column seam); inverted stage1-r --trace persistence + a CLI rank-by-
sqn_normalized integration. Full workspace suite green (527 passed, 0 failed);
clippy --all-targets -D warnings clean.
Derived fork decisions logged on #130 and #135 (cap = fixed const 100; opt-in
metric, default ranker unchanged; serde(default); Part B mirrors
momentum_sweep_family + reuses persist_traces_r).
closes#130closes#135
5-task plan for the boss-signed spec docs/specs/0067-r-sweep-followups.md.
Task 1 adds RMetrics.sqn_normalized (+ SQN_CAP + summarize_r compute) and
threads all four RMetrics struct literals (compile-gate). Task 2 adds the
Metric::SqnNormalized rank key. Task 3 re-baselines the 0066 single-run
golden (the r: block gains the field). Task 4 wires --trace for the stage1-r
sweep (invert the refusal test → persist member dirs via persist_traces_r,
drop the run_sweep guard). Task 5 is the full-workspace gate.
Sequencing pins the golden re-pin and the unfiltered suite after Task 1.
refs #130, refs #135
Two settled, additive follow-ups to the cycle-0065/0066 R-sweep surface,
named together by the user under /boss. Signed on the grounding-check PASS
(9 load-bearing current-behaviour assumptions, each ratified by a named
green test).
Part A (refs #130) — n-normalized SQN (SQN100): a new opt-in ranking metric
sqn_normalized = (mean_R/stdev_R)·√(min(n,100)), turnover-robust across sweep
members with different trade counts. Additive RMetrics field (serde(default)
for C18 back-compat) + a new Metric::SqnNormalized rank key mirroring Sqn.
Raw sqn and the default ranker are unchanged; the 0066 single-run golden is
re-baselined (the r: block gains one field — intended, not drift).
Part B (refs #135) — wire --trace for the stage1-r sweep: stage1_r_sweep_family
honours --trace by persisting each member's equity/exposure/r_equity under
runs/traces/<n>/<member_key>/ via the existing persist_traces_r, mirroring
momentum_sweep_family; the interim refusal guard in run_sweep is dropped.
Makes --trace symmetric across all three sweep strategies.
Derived fork decisions logged on #130 and #135 (reconciliation comments).
Cycle 0066 (#133) closed. The close audit (architect) found the family ratified against C12 axis-2 (rank-by-metric over the C18 family store) and the C10 R framing, with one [high] gap (swept-member manifest omitted the fixed R-defining params) fixed forward in 499121e.
- Ledger: a C10 realization note records the durable rationale -- SQN as the single-number sweep objective, the signal-only grid (the stop defines R, so varying it breaks cross-member SQN comparability -- the SQN100 #130 motivation), the R-metric rank vocabulary, and the manifest-completeness fix.
- Glossary: new 'R metrics' and 'SQN' entries.
- The ephemeral spec + plan are removed; the ledger + git history are the durable record.
Follow-ups filed: #135 (per-member --trace for the stage1-r sweep). refs #133
Cycle-0066 close audit found a C18/C10 gap: a stage1-r sweep member's manifest recorded only the floated knobs (fast.length, slow.length) via zip_params, omitting the fixed R-defining params (stop_length, stop_k, bias_scale) that the single run records. A family member must be reproducible from its own manifest (C18), and the constant stop -- which defines 1R -- must be visible so cross-member SQN comparability (C10) is auditable from the family record, not just the swept fast/slow knobs. The sweep run_one now appends the three fixed params, matching the single run; the integration test pins their presence per member.
Verified: full workspace suite 521 passed / 0 failed; clippy -D warnings clean.
refs #133
Realizes #133 (cycle 0066): SQN becomes the single-number objective for ranking a
Stage-1 sweep family by R signal quality.
aura-registry — metric_cmp learns three higher-is-better R metrics (sqn,
expectancy_r, net_expectancy_r) that reach into RunReport.metrics.r; a member with
r:None sorts to the worst position (treated as -inf via total_cmp), so a pip-only
family ranked by an R metric degrades to ordinal order without error. The
UnknownMetric "known:" list is extended. No persistence change — the family store
already carries the r block.
aura-cli — `aura sweep --strategy stage1-r` sweeps the stage1-r harness over a
fast×slow SIGNAL grid (4 members), each folding the dense R-record via summarize_r so
its RunReport carries r:Some(..). The grid varies only the signal: the stop and
sizing are held fixed because risk_budget is R-invariant (#128) and bias.scale is
sign-only under flat-1R (both degenerate axes for the ranked metric), and because the
stop defines the R unit — varying it would break cross-member SQN comparability (the
motivation for the deferred SQN100, #130). The stage1-r topology is extracted into a
shared stage1_r_graph(.., fast_len, slow_len) helper: the single run binds
Some(2)/Some(4) (output byte-unchanged), only the sweep floats the knobs.
Per-member --trace tracing is deferred (a follow-up): rather than accept the flag and
silently write nothing (a quality-review catch), `aura sweep --strategy stage1-r
--trace` is refused explicitly (exit 2); --name records the rankable family.
Verification — a golden characterization test (stage1_r_single_run_output_golden,
added against HEAD before the refactor) verifies the helper extraction is
byte-preserving; full workspace suite 521 passed / 0 failed; clippy --all-targets
-D warnings clean. The grounding-check that signed the spec first BLOCKed a false
"already byte-pinned" claim, which is why the golden is added rather than assumed.
All load-bearing forks (signal-only grid, None-sorts-worst, --trace refusal, helper
extraction, golden shape) are recorded on the #133 decision log.
closes#133
refs #117#130
Parent spec docs/specs/0066-r-sweep-rank.md (#133). Four tasks: golden characterization of the stage1-r single run (byte-preservation guard); extract stage1_r_graph helper (float fast/slow for the sweep, single run binds Some(2)/Some(4)); metric_cmp learns sqn/expectancy_r/net_expectancy_r (None sorts worst); stage1_r_sweep_family + --strategy stage1-r + rank-by-sqn integration test.
refs #133
Cycle 0066, reference issue #133. Settled design (deferred from 0065): make SQN
the single-number objective for ranking a Stage-1 sweep family by R signal
quality.
Two halves:
- aura-registry: metric_cmp learns sqn / expectancy_r / net_expectancy_r (all
higher-is-better, reaching into RunReport.metrics.r); a member with r:None sorts
to the worst position. No persistence change (the family store already carries r).
- aura-cli: --strategy stage1-r sweeps the stage1-r harness over a fast×slow signal
grid (stop & sizing fixed — risk_budget is R-invariant, bias.scale is sign-only
for flat-1R, both degenerate axes; the stop defines the R unit so varying it
breaks cross-member SQN comparability — see the #133 decision log). The stage1-r
topology is extracted into a shared helper so the single run keeps its identical
binding (output unchanged by construction) and only the sweep floats fast/slow.
Signed under /boss via the grounding-check PASS (the autonomous signature). The
first grounding pass BLOCKed a false claim that the stage1-r single-run output was
already byte-pinned; the spec was revised so a golden characterization test is
added against current HEAD before the refactor (verifying byte-preservation rather
than assuming it), and re-grounded to PASS.
refs #133#117
Brummel