Last piece of the Inferential-validation milestone (#146): a new
`aura generalize` subcommand grades how consistently a single candidate
holds across a set of instruments — the across-instrument axis of the
same anti-false-discovery concern as trials-deflation (#144, within one
family) and plateau-over-peak (#145, within one instrument's surface).
Aggregator/validator, not a third selector (user-ratified): it runs a
brought candidate (a single-cell stage1-r grid — --fast/--slow/--stop-
length/--stop-k pinned to single values) across an instrument list
(--real SYM1,SYM2,...), collects per-instrument R-metrics, and reduces
them to a worst-case floor (min over instruments) + a sign-agreement
count + the per-instrument breakdown. R-only (C10 — R is the only
account/instrument-agnostic unit); the worst-case min is the cross-
instrument sibling of PlateauMode::Worst and satisfies the non-
domination constraint by construction (a strong instrument cannot lift
a min). Rejected: a cross-instrument t-stat mean/std*sqrt(M), which
reintroduces the pooled mean the milestone warns against.
- RunManifest gains a first-class `instrument` lineage field (serde-
widened, C14/C18 — legacy lines load as None, existing manifest bytes
unchanged); the compat read-mirror threads it in lockstep.
- FamilyKind::CrossInstrument (C12 comparison axis); the M per-instrument
runs persist as a CrossInstrument family, each member self-identifying
via its stamped instrument. The aggregate is recomputable from the
members (McAggregate precedent), printed live.
- The generalization reduction lives registry-side beside optimize_
deflated/optimize_plateau (C9), reusing resolve_metric/metric_value/
is_r_metric; check_r_metric is the data-free pre-check the CLI runs
before evaluating any instrument.
- Additive: existing sweep/walkforward/mc/standalone paths stay byte-
identical (C23 — instrument omitted when None via skip_serializing_if).
Verified by the orchestrator: cargo test --workspace green (0 failed —
the new generalization_*, check_r_metric, parse_generalize_*, and the
gated cross-instrument E2E all pass; the E2E ran the real GER40/USDJPY
path on this host, exit 0); cargo clippy --workspace --all-targets
-D warnings clean.
closes#146
Adds optimize_deflated beside optimize in aura-registry: the in-sample
walk-forward selection now records, on each OOS winner's manifest, how much
the winner's metric is inflated by the size of the search it won — a deflated
score and (R arm) an overfit probability — without changing which member wins
(additive, C23; a regression test pins optimize_deflated's winner == optimize's).
- R arm (sqn_normalized/expectancy_r/...): a centred moving-block reality-check.
Each member's per-trade R series is mean-subtracted to impose the no-edge null,
resampled via the r_bootstrap kernel (extracted to a shared resample_block,
byte-identical), and the best-of-K null-max gives
overfit_probability = (count(null >= raw)+1)/(n+1) and
deflated_score = raw - p95(null). Deterministic given the seed (C1).
- total_pips arm: a closed-form expected-max-of-K dispersion floor
(inv_norm_cdf + expected_max_of_normals), no probability.
The record (FamilySelection on RunManifest) rides the proven
serde(default, skip_serializing_if) widening + a compat.rs mirror, so legacy
runs.jsonl/families.jsonl lines load unchanged and unstamped sweep/mc/standalone
manifests stay byte-identical (C14/C18). The SelectionMode enum reserves a
Plateau slot for #145. CLI: walk-forward stamps both arms; runs family <id> rank
surfaces a human-readable deflated/overfit line.
Quality review (re-dispatched after the implement-loop's quality gate exhausted)
caught a real defect: on the R arm with positive resamples but no member carrying
trade_rs (a family loaded from disk — trade_rs is serde-skipped), the null was
all-NEG_INFINITY, turning deflated_score into +inf. Fixed by collapsing "no usable
null" (zero resamples OR no trade_rs) into one degenerate floor (deflated == raw,
overfit == 1.0); pinned by optimize_deflated_no_trade_rs_floors_instead_of_infinity.
Verified: cargo test --workspace and cargo clippy --workspace --all-targets
-D warnings both green. Frictionless Stage-1 R; n_eff advisory and reload-time
recompute deferred (spec 0076 Out of scope).
closes#144
The data-server now ships per-symbol geometry sidecars, so aura no longer
hard-codes instrument geometry. A consumption grep showed production reads
exactly one `InstrumentSpec` field — `pip_size` — at the real-data path; the
other eight (incl. `instrument_id`) were constructed but never read.
- aura-cli resolves the real-path pip from the recorded sidecar
(`instrument_geometry` -> `InstrumentGeometry.pip_size`), refusing (exit 2)
on absent geometry. The authored table no longer gates: any symbol with a
sidecar + bars now runs (new test: USDJPY, never vetted, resolves its JPY
pip 0.01 from the sidecar).
- Removed the redundant authored floor: `InstrumentSpec`, `instrument_spec`,
`VETTED_SYMBOLS`, and the table-only tests. The `InstrumentGeometry` seam
(the surviving pip source) and its nameability guard are kept.
- `instrument_id` (the "find out"): no production consumer. The position-event
table's `instrument_id` is a decoupled caller parameter; the engine never
imports an instrument spec. Dropped with the struct.
- The example `pip_size_of` is re-sourced to the sidecar; the pure-structural
blueprint tests use a literal pip to stay archive-independent.
Speculative deploy-grade fields and the #124 override/floor tiers are deferred
to the Stage-2 broker that will consume them, read from the sidecar geometry
then. Ledger (C8/C15/#22/#106) updated; #124 collapses to "recorded sidecar
geometry -> refuse" in the meantime.
Verified: cargo test --workspace (all green), clippy -D warnings clean, cargo
doc clean, with the local archive present so the sidecar paths run.
refs #124
The runnable GER40/FRA40 examples baked `const PIP_SIZE = 1.0` and a dead pip
column in the compare INSTRUMENTS table — a second source of pip truth parallel
to the vetted `aura_ingest::instrument_spec` channel the CLI `--real` path already
uses (#22). Route them all through one `pip_size_of(symbol)` helper: build_harness
sources GER40 internally, ger40_breakout_blueprint gains a `pip_size` first arg fed
from the lookup, and compare's run_cell sources each instrument's pip per cell.
Value-identical (GER40/FRA40 both resolve to 1.0) and behaviour-preserving: the
gated determinism / blueprint param-space / world tests stay green unchanged, and
the frozen `sim-optimal(pip_size=1)` broker-label pins are untouched.
closes#98
drain_trace now joins its four recording-sink taps through aura-engine's
join_on_ts instead of a hand-rolled per-tap HashMap, resolving the #93
zip-by-index panic at its source: the four taps fire at different
cardinalities (breakout one bar short via cold Delay(1); bars_since_open
filtered by Session), so a positional zip misaligns or panics. The BarTrace
mapping stays a thin layer with its caller-side defaults unchanged
(held->0.0, bars_since_open->-1, breakout->false) — the engine reports
presence, the consumer interprets absence. Behaviour is byte-preserved: the
gated GER40 tests (ger40_breakout_real, ger40_breakout_blueprint,
open_ohlc_seam) stay green against the real archive.
The Recorder doc gains a prose note on the multi-tap cadence rule (join on
ts, never by index), pointing at join_on_ts by name rather than rustdoc link
since aura-std does not depend on aura-engine.
Verified: cargo build/test/clippy --workspace --all-targets clean; the
_by_ts hand-rolled join is gone (acceptance grep empty).
closes#93
A real OHLC source now builds from aura-ingest alone, in the engine-native
epoch-ns Timestamp currency, with the one ms<->ns crossing owned by the seam.
Library surface (src/lib.rs), all additive — the ms-based `open` and
`unix_ms_to_epoch_ns` are untouched, so every existing call site is preserved
byte-for-byte:
- epoch_ns_to_unix_ms (private): the seam-owned inverse; consumers never
convert (#80).
- M1FieldSource::open_window: the Timestamp-window mirror of `open`, mapping
each bound through the private inverse and delegating to `open`.
- open_ohlc: the canonical OHLC bundle opener — four M1FieldSources in the fixed
open/high/low/close C4 merge order, sharing one Arc<DataServer> (one cache,
C12); the order lives in one vetted place (#92).
- default_data_server + `pub use data_server::{DataServer, DEFAULT_DATA_PATH}`:
a real-data source builds without naming the external data_server crate (#81).
Consumer migration: the shared breakout_real.rs helpers and every GER40
example/test move to the Timestamp-native surface — open_ohlc_sources removed,
utc_month_window_ms -> utc_month_window (Timestamp), report_from_trace retyped,
the per-consumer ns_to_ms hand-divides deleted, the direct data_server imports
replaced by the re-exports. ger40_breakout_compare.rs (which opened OHLC by its
own hand-loop, reached by neither retyped helper) is migrated by inspection.
Return type is Vec, not [_; 4]: Harness::run consumes Vec<Box<dyn Source>>, so
it feeds straight in; order-safety comes from the single helper either way.
Tests: a hermetic round-trip pinning the inverse; a gated behaviour-preservation
test (open_ohlc Timestamp path == ms-path open, bit-identical recorded series);
a hermetic absent-archive fixture pinning the file-level None contract and the
#81 re-export firewall.
Verified: cargo build --workspace --all-targets clean; cargo test --workspace
green (the gated GER40 tests ran against the local archive — open_ohlc_seam
proves byte-identity on real data, not a skip); clippy --all-targets -D warnings
clean. Acceptance greps pass: open_ohlc_sources gone, no consumer-side ns_to_ms,
no data_server import in any migrated GER40 consumer, epoch_ns_to_unix_ms private.
closes#80, #81, #92
docs/specs and docs/plans were retired (prior commit); the source/test
comments that cited them ("spec 0050 §4.1", "spec §Testing N", "per spec",
"the spec's ...") now point at nothing. Strip every such pointer while
preserving the technical substance, the design-ledger contract refs
(C1/C11/C20/C34/C12.1/...), and the Gitea issue refs (#41).
Comment/doc edits only across 20 files — no logic change; full workspace
suite green, clippy clean.
Author the shipped session-breakout as a reusable Composite blueprint via
GraphBuilder — the World-consumable form (param_space + bootstrap) — alongside
the hand-wired FlatGraph it reproduces. Applies the spec-0051 decisions:
- bar_period_minutes is a construction arg binding BOTH Resample.period_minutes
and Session's period, locked equal, so a sweep cannot desync them (#96);
- delay.lag is bound out (a structural constant, not a tuning knob — C34);
- the two EqConst targets (named entry_bar / exit_bar) are the ONLY params, so
param_space() == {entry_bar.target, exit_bar.target} — the genuine tuning
knobs, which makes walk_forward non-degenerate (#97).
The ger40_breakout_real example now bootstraps from the blueprint (still prints
-45.0 for 2024-09). New gated test ger40_breakout_blueprint.rs proves: (a) the
param_space is exactly the two targets (no lag, no period); (b) C23 — the
blueprint reproduces the FlatGraph bit-identically over real GER40 2024-09;
(c) determinism.
First increment of the milestone; the sweep / walk_forward / compare World-family
demos follow.
refs #94, #96, #97
Run the Phase-1 session-breakout node vocabulary over REAL GER40 M1 bars
from the data-server archive — the first real-data exercise of the shipped
strategy nodes (Resample/Delay/Gt/Session/EqConst/And/Latch + SimBroker).
It is a pure composition of shipped aura-std nodes (no project-specific
signal code), so it lives in the engine repo's examples/ carveout (C9), not
a separate consumer crate. The breakout FlatGraph is reused VERBATIM from
aura-engine/tests/ger40_breakout.rs; only the sources change — four real
M1FieldSource (open/high/low/close, in the fixed C4 merge order) replace the
synthetic VecSources, with pip_size = 1.0 (GER40 is an index).
- examples/ger40_breakout_real.rs — runnable demo over a Sept-2024 window:
prints the RunReport metrics plus a per-session entry/exit trace.
- examples/shared/breakout_real.rs — the single 13-node wiring (the proven
11 + two trace taps), shared by the example and the test via #[path] so it
is never duplicated.
- tests/ger40_breakout_real.rs — gated determinism test (mirrors
real_bars.rs): skips where the archive is absent; where present, asserts
finite metrics, binary Latch exposure (held in {0.0, 1.0}), and
bit-identical C1 reruns of both the report and the recorded series.
chrono / chrono-tz added as aura-ingest dev-dependencies (the Session
timezone and the UTC window bounds), test-only — never on the normal path.
Brummel