db3308fcc58b6a76de24b40925ff8018edb124b3
286 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Brummel
|
db3308fcc5 |
chore: drop redundant spec auto-sign project fact
spec auto-sign is now fixed /boss behaviour in the skills plugin (no longer a per-project toggle), so the explicit 'enabled' entry is redundant. No behaviour change — auto-sign was already on for this project. |
||
|
Brummel
|
be8d267019 |
docs(aura-engine): SweepError summary names RandomSpace::new too
The SweepError type-level rustdoc still read "A structural fault constructing a `GridSpace`" though the enum now also gates `RandomSpace::new` (the three random-only variants are each documented individually, but the type summary was stale). Broaden it to name both spaces and group the grid vs. random faults. Doc-only, behaviour-preserving — the same low-grade doc-debt class the 0049 audit fixed inline for the module doc. Surfaced by the cycle-0049 fieldtest. refs #52 |
||
|
Brummel
|
af0191884d |
fieldtest: cycle-0049 — 4 examples, 7 findings
Public-API field test of the random param-sweep surface, from a standalone downstream-consumer crate (path-deps only; the public interface = ledger + glossary + spec 0049 + cargo doc rustdoc; no crates/*/src read). Four bins, each built from HEAD and run: continuous tuning (200-point random tune ranked by total_pips), the typed validation gate (all five reachable SweepError variants pre-run), reproducibility + seed-sensitivity + the full i64::MIN..=MAX sampler edge, and Space-trait interchangeability (one tune_and_rank<S: Space> over both GridSpace and RandomSpace). Findings: 0 bugs, 4 working, 2 spec_gap, 1 friction. The four working findings confirm the cycle's acceptance criterion empirically — the headline tune reads as the code a researcher would write, the gate is precise and fires before any run, the C1 reproducibility promise is checkable in one line, and the Space trait delivers one-consumer/both-enumerations. Triage of the actionable findings: - friction (no named-axis builder for RandomSpace — positional Vec<ParamRange> must align with param_space() by hand, and a same-kind transposition passes validation silently): filed as a feature for a future cycle, refs #79 (a RandomBinder sibling to the grid's SweepBinder). - spec_gap (SweepError rustdoc summary named only GridSpace): fixed inline in a follow-up doc commit. - spec_gap (NonNumericRange / Bool-slot ranges unreachable with the shipped aura-std node roster — no node declares a Bool/Timestamp knob): RATIFIED as intentional. The variant is a forward-looking structural guard for the C16 "author your own node" path (a Bool/Timestamp param-slot a custom node may declare); its current untriggerability with the standard roster is expected, not drift. refs #79 |
||
|
Brummel
|
3fca7810d0 |
audit: cycle 0049 — drift-clean (random param-sweep)
Architect drift review over 3de00e2..HEAD (the 0049 spec/plan/feat plus the two intervening refactors that had not been audited: aura-ingest M1 transpose |
||
|
Brummel
|
e17d78f24f |
feat(aura-engine): random param-sweep — RandomSpace + a Space trait + typed ParamRange
Ship the random half of the C12.1 param-sweep axis (grid landed in 0028).
A new `Space` trait (`points`/`param_specs`) generalizes `sweep`/`sweep_with_threads`
from `&GridSpace` to `&S: Space`; `GridSpace`'s trait impl forwards to its existing
inherent methods, so the grid path is behaviour-preserving (C1, every pre-existing
grid sweep test stays green). `RandomSpace` is the second `Space` producer: N seeded
uniform points over per-slot typed `ParamRange { lo, hi }` ranges (I64 inclusive
[lo,hi], F64 half-open [lo,hi)), validated in `new` against the param-space — a
non-numeric slot, a range-kind mismatch, and an empty range are the three new typed
`SweepError` variants, all caught before any run. Sampling reuses the existing
bit-stable `SplitMix64` (promoted to `pub(crate)`) as a code-path-disjoint instance
from the data-edge seed RNG — the #52/#71 World-II source-seam firewall: they share
only the `u64` type, never a path. The sweep-layer signature stays param-only
(`Fn(&[Cell]) -> RunReport`); no `Source`/stream type enters it.
Deviation from the spec's verbatim sampler, accepted as a correctness hardening:
the I64 draw guards the full-width span ([i64::MIN, i64::MAX] computes a span of
2^64 that wraps a u64 to 0) and uses `lo.wrapping_add(draw as i64)` instead of a
plain `+`. This avoids both the `% 0` divide-by-zero at the full domain and the
signed-overflow panic the literal form hits on wide ranges in debug builds, while
preserving uniform-over-[lo,hi] draws and determinism (an extra RED test,
random_points_full_range_i64_does_not_panic, pins it). Modulo bias for spans not
dividing 2^64 remains the spec's documented, accepted simplification (param search
needs no cryptographic uniformity).
Includes a public-API E2E suite (tests/random_sweep_e2e.rs) exercising the feature
exactly as a downstream researcher would — reproducibility at the report boundary,
seed-sensitivity, in-range draws, the typed NonNumericRange gate, a well-formed
empty (count==0) family, and Grid/Random interchangeability through `Space`.
Verified: cargo test --workspace green (incl. 17 new engine tests + 6 E2E);
cargo clippy --workspace --all-targets -- -D warnings clean.
closes #52
|
||
|
Brummel
|
b99f52443d |
plan: 0049 random param-sweep
Five bite-sized tasks for the RandomSpace cut: (1) the Space trait + behaviour-preserving generalization of sweep/sweep_with_threads, (2) typed ParamRange, (3) RandomSpace::new validation + three SweepError variants, (4) SplitMix64 -> pub(crate) + the seeded sampler, (5) public exports + random-sweep integration. Each task is RED-first with single-substring test filters; the grid suite is the C1 behaviour-preservation regression guard. refs #52 |
||
|
Brummel
|
5b3d133529 |
spec: 0049 random param-sweep (boss-signed)
RandomSpace + a Space trait + typed ParamRange: the random half of the C12.1 param-sweep axis (grid shipped in 0028). N seeded uniform draws over declared continuous ranges, fed to the same enumeration-agnostic sweep core. Param-only sweep signature and a code-path-disjoint SplitMix64 instance keep the World-II source-seam firewall (refs #71) intact. Auto-signed under /boss spec auto-sign: fresh grounding-check PASS and a unanimous five-lens spec-skeptic panel (criterion, grounding, scope-fork, ambiguity, plan-readiness all SOUND). refs #52 |
||
|
Brummel
|
67c1f51cfe |
perf(aura-std): O(1) Kahan-compensated running-sum SMA (was O(length) re-sum)
Sma::eval re-summed the whole window every tick — O(length)/tick, multiplied across millions of bars and every sweep point. Replace with the industry-standard incremental running sum (ta-lib shape): the node owns the window ring and keeps a running sum, adding the new sample and subtracting the evicted one each cycle, so the input column drops to depth 1 (lookbacks() = [1], like Ema). The running sum carries Kahan/Neumaier compensation (the fix pandas' rolling mean adopted) so it does not drift over long runs; Ema needs none — its recurrence is contractive. Determinism (C1) holds: same input -> same run, reproducibly. The float output differs from the full re-sum in the last ULPs (a new, deterministic baseline); the suite needed only one arity-assumption update (Sma lookback 3 -> 1), no equity golden changed. closes #39 |
||
|
Brummel
|
6390093f93 |
refactor(aura-ingest): chunk-direct M1 transpose, drop the AoS intermediate
load_m1_window collected every bar into an intermediate Vec<M1Parsed> (a full AoS materialization, un-presized so it grew by reallocation) before transpose_m1 copied it again into the SoA columns — ~2x peak load memory plus realloc churn. Transpose each chunk straight into the columns instead, via a new shared M1Columns::extend_from_bars (reserves per chunk for amortized growth). One fewer full copy, halved peak. Behaviour-preserving: the resulting M1Columns is byte-identical and transpose_m1's / load_m1_window's signatures are unchanged. Pinned by a new parity test that needs no live DataServer. closes #78 |
||
|
Brummel
|
3de00e23e0 |
audit: cycle 0048 — drift-clean (walk-forward param plane)
cycle 0048 tidy. Architect drift review (86746e3..9febdb9, against the design ledger C7/C1/C12/C20/C23) found the walk-forward param-plane cut drift-clean: - C7 made honest: WindowRun.chosen_params is now genuinely tag-free Vec<Cell>; the param kind lives once on WalkForwardResult.space. No residual per-value tag. The deleted scalar_as_f64's per-value unreachable! became a per-slot schema cost (the coerce builder map), correctly invoking C20 only on the timestamp arm. - C1 behaviour-preserving: the i64->f64 / f64 coercions are value-identical to the deleted helper; the migrated fixtures keep mean 2.5 / p50 2.5 / mean 1.5. The Bool->0/1 arm is the only new behaviour, pinned by param_stability_reduces_bool_slot_to_fraction_true (0.75). - SweepFamily symmetry is real: the (space + tag-free Vec<Cell> points + zip_params) idiom is mirrored while the two stay distinct types. The arity debug_assert reads space.len() against windows (not the moved-out runs) — correct. walkforward_family derives space from the same blueprint sweep_over uses, so the kinds genuinely match. - No stale references: the only "Vec<Scalar>" / "self-describing record" matches are in docs/plans/0048 (a historical plan quoting the BEFORE state by design) and ledger lines describing Scalar itself (still true). No live code or current-state doc carries the old Fork-A chosen_params rationale. Regression check: no regression script configured (project facts list only build/test/lint/doc) — no-op; architect is the gate. The four gates (build/test/clippy -D warnings/doc) and the acceptance greps were verified green before the close commit. Follow-on: the out-of-scope WalkForwardResult::named_params (closing the last asymmetry vs SweepFamily::named_params) was filed as a droppable idea (#76). Not actionable drift — no consumer is blocked (zip_params already serves it). |
||
|
Brummel
|
9febdb9623 |
refactor(aura-engine): walk-forward param plane — space on family, tag-free chosen_params, schema-checked param_stability
Make the walk-forward param plane C7-clean and symmetric with the sweep param plane. WalkForwardResult gains `space: Vec<ParamSpec>` (the kinds, once, mirroring SweepFamily.space) and WindowRun.chosen_params changes Vec<Scalar> -> Vec<Cell> (the tag-free coordinate the inner sweep already produces). param_stability now resolves its numeric coercion once per slot against the schema (result.space) into a Vec<fn(Cell)->f64>, then runs a type-blind reduction loop — replacing the old per-window-value scalar_as_f64 with its per-value unreachable!. scalar_as_f64 is deleted.
This overturns cycle 0047's "Fork A" (chosen_params stays Vec<Scalar>), which was justified by Scalar's new serializability. That defense was moot: WalkForwardResult and WindowRun carry no serde derives — they are never serialized; only the embedded oos_report (a RunReport, typed via the 0047 lift) is. So chosen_params is a pure in-memory substrate for param_stability, and Vec<Scalar> made it C7-impure (carrying the kind N×M times at the value, where the kind is per-slot constant). The substantive case — kind at the schema, SweepFamily symmetry, per-value unreachable! eliminated — wins, the same argument-form that retired the {kind,cell} Scalar struct in 0047.
The seam ([A]): walk_forward / walk_forward_with_threads take `space: Vec<ParamSpec>` by value, a sibling of the run-window closure; the Fn(WindowBounds)->WindowRun+Sync bound is unchanged and run_indexed is untouched (space is never captured by the Sync closure, moved onto the result at the end). A debug_assert in walk_forward_with_threads guards the arity coupling (every window's chosen_params.len() == space.len(), guaranteed by the same blueprint). walkforward_family computes the space once before the roll and passes the tag-free winner (chosen_params: best.params) directly, dropping the from_cell+zip reconstruction.
[D]: a Bool param slot coerces to 0/1 (mean = fraction "on"), keeping the coercer total over the knob kinds; a Timestamp slot is structurally impossible (C20) so the schema pass carries one unreachable! (once per slot, never per value). param_stability keeps an `if result.windows.is_empty() { return Vec::new(); }` guard to preserve its documented empty-on-no-windows contract. WalkForwardResult and SweepFamily share the (space + tag-free Cell points + zip_params) idiom but stay distinct types — different axis (Sweep enumerates input coordinates; WF rolls time and chosen_params is the optimizer's output), and WF additionally carries bounds + oos_equity + stitched_oos_equity.
Behaviour-preserving (C1): the three migrated walk-forward tests stay green with identical MetricStats (param_stability_reduces_chosen_params_per_slot still pins mean 2.5 / p50 2.5 / mean 1.5); the i64->f64 and f64 coercions are value-identical to the deleted scalar_as_f64. The one genuinely new behaviour (Bool 0/1) is unexercised by built-in grids and pinned by the new param_stability_reduces_bool_slot_to_fraction_true (mean 0.75, 3-of-4 true).
Gates: build --workspace --all-targets, test --workspace, clippy --workspace --all-targets -D warnings, doc --workspace --no-deps — all clean.
closes #75
|
||
|
Brummel
|
6ce00ee053 |
plan: 0048 walk-forward param plane
Two-task execution plan for spec 0048: Task 1 lands the whole aura-engine/walkforward.rs cut atomically (struct shapes, walk_forward seam, param_stability rewrite, scalar_as_f64 deletion, fixture migration, new Bool-slot test) gated on a crate-only `cargo test -p aura-engine`; Task 2 threads the aura-cli caller and runs the workspace gates. Keeps a windows.is_empty() guard in param_stability to preserve the documented empty-on-no-windows contract (C1). refs #75 |
||
|
Brummel
|
791b809413 |
spec: 0048 walk-forward param plane
Settled-design spec for the walk-forward param-plane cut: WalkForwardResult gains space: Vec<ParamSpec>, WindowRun.chosen_params Vec<Scalar> -> Vec<Cell>, walk_forward takes the space by value (Fn-closure generic untouched), and param_stability schema-checks a per-slot coercer (Bool -> 0/1, Timestamp unreachable per C20) before a type-blind reduction, deleting scalar_as_f64 + its per-value unreachable. Behaviour-preserving (C1): param_stability yields identical MetricStats over the existing fixtures. Overturns 0047's Fork A (chosen_params stays Vec<Scalar>) — the WF structs carry no serde derives, so the serializability defense is moot; the substantive case is C7 purity + SweepFamily symmetry. grounding-check PASS. refs #75 |
||
|
Brummel
|
86746e3d5d |
refactor(aura-core): Scalar as a native tagged enum, disjoint from Cell; typed RunManifest.params
Scalar was `struct { kind: ScalarKind, cell: Cell }` — "a Cell wearing a kind hat." The recorded reason for that shape was migration ease, which is not a design rationale (CLAUDE.md: rationale != effort), so the struct had no substantive defense. Redefine it as the native tagged union it conceptually is:
enum Scalar { I64(i64), F64(f64), Bool(bool), Timestamp(Timestamp) }
This is substantively better on four axes: kind/bits skew becomes unrepresentable (illegal states gone); accessors panic loudly on the wrong variant instead of a release-mode silent bit reinterpret; PartialEq derives the documented IEEE-754 / cross-kind value semantics (the hand-roll, needed only because the struct inherited Cell's bitwise compare, is gone); and serde is a plain derive emitting the externally-tagged wire form ({"I64":10}/{"F64":2.5}) — the private ScalarRepr shadow enum that motivated this was never needed. It is also C7-honest: erased-on-the-hot-path (Cell) and self-describing-at-the-edge (Scalar) are two disjoint types bridged by explicit conversion.
The whole public API is preserved (Scalar's fields were private), so call sites do not churn: the enum change is contained to scalar.rs, where cell() now encodes and from_cell() decodes per kind. Cell and ScalarKind are untouched.
With Scalar serializable, lift RunManifest.params from Vec<(String, f64)> to Vec<(String, Scalar)>: the param's kind (an i64 length vs an f64 scale) now survives into the C18 record (runs.jsonl) and the CLI JSON instead of collapsing to f64. scalar_as_param_f64 is deleted; sim_optimal_manifest passes typed params through. This is a deliberate wire-shape change — params now render as tagged scalars; the JSON-asserting tests are updated to the new shape on purpose.
Hand-authored manifest params across the CLI's single-run/mc/macd sites use honest kinds (lengths -> i64, scales -> f64) so they match the sweep path (which already derives correct kinds via zip_params); their in-binary JSON assertions are re-tagged accordingly.
Walk-forward fork resolves with no code change: WindowRun.chosen_params stays Vec<Scalar> (the serializable record carrier), reduced to f64 only at the param_stability statistic boundary (scalar_as_f64 retained). Glossary 'cell' entry updated to describe Scalar as the disjoint tagged union, not 'a cell plus its kind tag'.
Gates: build --all-targets, test --workspace (incl. new scalar_serde_round_trips), clippy -D warnings, doc --no-deps — all clean.
|
||
|
Brummel
|
43716be10e |
refactor(aura-engine): Cell into the construction path — param-plane base/frontend split
Cell becomes the carrier of the construction path; Scalar narrows to the author/render boundaries. The validated/enumerated param point carries no redundant kind (it lives once, in the declared param-space); at the author edge the kind is a checksum — two independent sources, the typed value vs. the slot — so the self-describing Scalar stays there. The name->slot binding is dynamic (C23), so the check is necessarily a runtime one and the value must self-describe for it. Base/frontend split (the AnyColumn push/push_cell pattern one level up): compile_with_cells / bootstrap_with_cells are the kind-check-free base; compile_with_params / bootstrap_with_params are the frontend that adds only the per-value kind checksum, strips to cells (new Scalar::cell(), the partner of Scalar::from_cell), and delegates. lower_items loses its per-primitive kind-check; PrimitiveBuilder::build and the std node builders (sma/ema/exposure/lincomb) read cells. Boundary: construction -> Cell (PrimitiveBuilder::build, lower_items, GridSpace.axes/points, SweepPoint.params, the sweep closure). Author edges stay Scalar (GridSpace::new, bind, compile_with_params/bootstrap_with_params). walkforward chosen_params stays a self-describing Scalar report record (Option A — WalkForwardResult carries no space); the cell winner is reconstructed once at the WindowRun site via from_cell. injective-check moved ahead of the arity-check in the frontend to preserve the pre-split error order (DuplicateParamPath before ParamArity). The lossy i64->f64 param projection (scalar_as_f64 / scalar_as_param_f64) is deliberately untouched — a separate follow-up. Behaviour-preserving (C1): build --all-targets / test / clippy -D warnings / doc all clean across the workspace. |
||
|
Brummel
|
3598cb9dde |
audit: cycle 0047 — drift-clean (cell hot-path carrier)
cycle 0047 tidy. Architect drift review (b188773..82635fa, against the design ledger C7/C8/C1) found the carrier swap drift-clean: - Node::eval -> Option<&[Cell]>, all 8 aura-std out-buffers [Cell; N], the inter-node forward via the branch-free push_cell (harness.rs:426); Scalar survives only on the three keep-set boundaries (param plane, AnyColumn::get, source ingestion at harness.rs:402). The convert/keep partition and the three-role test rule are honoured site-for-site. - The dual API is coherent: fallible push (ingestion + kind-guard tests) vs infallible push_cell (bootstrap-verified inter-node forward) are each load-bearing and tested. No dead code, no C7 violation (no tag on the hot path; scratch: Vec<Cell> reused via clear(), no per-cycle alloc). - Both C7 realization notes + the C8 prose accurately describe the landed state. 285 tests green; build/clippy/test/doc clean. Tidy fix (pre-existing, surfaced by the review): - docs/glossary.md `node` entry said a node implements `schema()`, but Node::schema() was removed in cycle 0024 (the signature is declared pre-build on PrimitiveBuilder). Corrected to `lookbacks()` + `eval(ctx)`, the methods the Node trait actually carries. Record-reality. |
||
|
Brummel
|
82635fa259 |
refactor: Cell becomes the hot-path value carrier (C7/C8 realization)
Motivation ---------- The C7 realization note ( |
||
|
Brummel
|
1a33b4f252 |
plan: 0047 cell hot-path carrier
Four tasks for the behaviour-preserving carrier swap: (1) add the branch-free AnyColumn::push_cell + round-trip test (additive); (2) the atomic lib-code flip (Node::eval -> Option<&[Cell]>, 8 aura-std nodes, recorder, harness forward loop) gated by a lib-only build; (3) the cfg(test) migration (fixtures + eval-output assertions per the spec's three-role rule) gated by the full four-gate run; (4) the C7/C8 ledger realization note. refs #74 |
||
|
Brummel
|
73e1558383 |
spec: 0047 cell hot-path carrier (boss-signed)
Auto-signed under /boss spec auto-sign: all objective gates green (Step-1.5 precondition, Step-4 self-review, Step-5 grounding-check PASS) and a unanimous five-lens spec-skeptic panel SOUND, after one editorial round on the ambiguity lens (the test-migration rule was sharpened to a three-role classification: eval-output -> Cell, column-write -> Scalar, channel-send -> Scalar). Spec for the deferred C7/C8 carrier swap flagged by the C7 realization note: Cell becomes the hot-path value carrier (Node::eval -> Option<&[Cell]>, node out-buffers, inter-node forward via a new AnyColumn::push_cell), and Scalar narrows to the self-describing dynamic boundaries (the param path, AnyColumn::get, source ingestion). Behaviour-preserving (C1): only the carrier type narrows. refs #74 |
||
|
Brummel
|
8eff6398ca | docs(glossary): add the cell entry | ||
|
Brummel
|
049f22a401 |
docs(design): C7 realization note for the Cell carrier split
Record the Scalar -> {kind, cell} representation change under C7 (
|
||
|
Brummel
|
cd3d1ca9ed |
refactor(aura-core): split Scalar into a tag-free Cell + ScalarKind
Motivation
----------
`Scalar` was a tagged enum (I64/F64/Bool/Ts), so every scalar value
physically carried its own kind tag. But the kind is already known from
the schema/port/column the value flows through (C7: the type is a
property of the column, not of the value — the hot path is already
columnar `Column<T>`, and `AnyColumn::get` *reconstructs* the tag from
the column on the way out). The per-value tag was therefore redundant
with the kind the surrounding context already holds.
That redundancy had three costs:
* It baked an implicit `match` (a branch) into every function that read
a Scalar payload — even where the caller statically knew the type.
The tag could never be exploited away.
* Size: a tagged enum is tag + payload = 16 bytes (f64/i64 alignment),
twice the 8 bytes the value needs. A `Column<Scalar>` would be double
the memory and half the cache utilisation.
* It is the shared root of several downstream papercuts we keep hitting
— the lossy f64 manifest field, the `unreachable!` panic on a
non-numeric param, the serde-tag question — all symptoms of "the type
is baked into the value".
Change
------
Introduce `Cell`: a type-erased 64-bit word (`struct Cell(u64)`) that is
not readable without external type context. It is constructed per base
type (`from_i64/from_f64/from_bool/from_ts`) and read only by naming the
type at the call site (`i64()/f64()/bool()/ts()`) — each a branch-free
bit-cast. The hot path resolves the kind once at the boundary (from the
schema) and then reads natively, with no per-value branch. `Cell` knows
nothing of `Scalar` or `ScalarKind`; the dependency is strictly one-way,
and it lives in its own `cell.rs` (more is planned on top of it).
`Scalar` becomes `struct { kind: ScalarKind, cell: Cell }` — the
self-describing form for the dynamic boundaries (builder binding,
serialization, rendering), built on top of `Cell`. Its `as_*` accessors
now `debug_assert` the kind and return the native value (free in
release); calling the wrong accessor is a caller bug, not a checked
`Option`. The variant constructors `Scalar::I64(..)` become associated
fns `Scalar::i64(..)`.
`PartialEq` is hand-written (not derived) to preserve the former enum's
value semantics: kinds must match, then native payloads compare, so f64
keeps IEEE-754 behaviour (`NaN != NaN`, `+0.0 == -0.0`) and a kind
mismatch is never equal even when the raw words coincide. A fixture
(`scalar_eq_is_value_not_bitwise`) pins exactly the cases where bit- and
value-equality diverge, so it can't silently regress. `Cell`'s own
`Eq`/`Hash` stay bitwise — correct for a raw word.
The change is behaviour-preserving: Scalar's observable behaviour is
identical to the pre-Cell enum (the value-equality fixture proves it);
only the internal representation changed. The ~440 call sites across the
workspace are a mechanical constructor rename plus ~12 destructuring
sites (match-arms / `let`-patterns) rewritten to `kind()` + `as_*`.
Verified: cargo build --workspace --all-targets, cargo clippy --workspace
--all-targets -- -D warnings, cargo test --workspace — all green.
|
||
|
Brummel
|
b188773fb8 |
audit: cycle 0046 — drift-clean (sweep named-binding)
cycle 0046 tidy (clean). Architect drift review against the design ledger (C8/C12/C23 + the #71 firewall) found no drift: - The named view is a derived, non-load-bearing projection (C23 — slot is identity); no name reaches the flat graph, no new per-point state, no new identity mechanism. SweepFamily.space is param_space() carried once, not minted. - Enumeration and determinism (C1) are untouched (GridSpace::points / run_indexed unchanged; the new SweepFamily.space is equal across thread counts). - The run-closure signature Fn(&[Scalar]) -> RunReport is byte-unchanged, so #52's RandomSpace plugs into the same sweep() layer with zero reconciliation (#71 firewall held). - All three CLI hand-zips collapsed to one zip_params call each; no un-collapsed duplicate remains. zip_params sits in aura-core (a pure projection over aura-core types) — correct home. - The f64 manifest field stays Vec<(String,f64)> (the typed-param-space precursor honestly left deferred, not entrenched). Regression gate green: cargo test --workspace (all crates, 0 failed; 4 new tests this cycle), cargo clippy --workspace --all-targets -D warnings clean. Forward-consistency item (NOT drift, out of #57 scope): McDraw/McFamily and WindowRun still carry bare Vec<Scalar> with no named view — a future item if a consumer demands it. closes #57 |
||
|
Brummel
|
fb8cabf13f |
feat(aura-engine): sweep named-binding — zip_params + SweepFamily.named_params (C12 #57)
Thread a derived named view of a sweep point so consumers stop hand-zipping param_space() names onto the bare &[Scalar]. The free function zip_params(space, point) -> Vec<(String, Scalar)> in aura-core is the one shared projection; GridSpace retains the ParamSpec list it already receives in new() (it was validated then discarded); SweepFamily carries it (stamped once in sweep_with_threads) and exposes named_params(i). The three CLI hand-zip sites collapse to one zip_params call each. The run-closure signature `Fn(&[Scalar]) -> RunReport` is byte-unchanged — the named view is a derived projection (C23: slot is identity, name is derived), not a closure-currency change — so #52's RandomSpace plugs into the same sweep() execution layer with zero signature reconciliation (#71 firewall held). sim_optimal_manifest now takes typed Vec<(String, Scalar)> and does the lossy f64 collapse internally (one place; the manifest's f64-precursor field owns its own lossiness — the typed-manifest upgrade stays the deferred typed-param-space item, a non-goal here). All eight call sites migrated: three sweep closures pass zip_params, five hand-listing callers (run_sample, run_sample_real, mc_family, run_macd, run_sample_seeded) pass Scalar::F64 literals. Behaviour-preserving: the collapse output is identical to the old per-site zip, so aura sweep / walkforward / run output is byte-identical; SweepFamily.space threaded into the aura-registry optimize test literal. Verified by the orchestrator: cargo test --workspace green (4 new tests: zip_params x2, sweep_family_carries_param_space, family_named_params_round_trips), cargo clippy --workspace --all-targets -D warnings clean. (rust-analyzer emitted stale mid-edit diagnostics; the real cargo build/test is clean.) refs #57 |
||
|
Brummel
|
cd9f7e4ea3 |
plan: 0046 sweep named-binding
Three tasks for spec 0046. T1: zip_params (aura-core), RED-first. T2: GridSpace retains its ParamSpec list + SweepFamily carries it + named_params (aura-engine), RED-first, with the aura-registry optimize-test SweepFamily literal threaded in the same task. T3: sim_optimal_manifest typed-Scalar migration + all eight call sites (aura-cli), behaviour-preserving. Each signature/field change threads every one of its sites within its own task, before that task's workspace compile gate. refs #57 |
||
|
Brummel
|
119221c8bb |
spec: 0046 sweep named-binding (C12 #57)
Thread a derived named view of a sweep point so consumers stop re-zipping param_space() names onto the bare &[Scalar]. A free function zip_params over (ParamSpec names ⊗ positional point) in aura-core; GridSpace retains the ParamSpec list it already receives in new(); SweepFamily carries it and exposes named_params(i). The run-closure signature `F: Fn(&[Scalar]) -> RunReport + Sync` stays byte-for-byte (honours #52's #71-firewall constraint, so RandomSpace plugs in with zero reconciliation). The lossy f64 collapse moves into the manifest constructor; the view stays typed. Behaviour-preserving; SweepPoint and enumeration untouched. Design ratified in-context (brainstorm, approach C over A, free function over a NamedPoint type); a reconciliation comment on #57 records the resolved forks with provenance. Human sign-off after the auto-sign panel's grounding lens caught a blast-radius undercount in the first draft — corrected to all eight sim_optimal_manifest call sites plus the aura-registry SweepFamily struct-literal. refs #57 |
||
|
Brummel
|
c2756732d4 |
refactor(aura-registry): split FamilyRunRecord family_id into {family, run} + derived id
Behaviour-preserving normalisation of the lineage record. The fused
`family_id: String` ("{name}-{counter}") is split into its two factors —
`family: String` + `run: usize` — and the user-facing handle becomes a derived
`family_id()` method ("{family}-{run}"), never stored or parsed.
Why: the fused field forced the counter logic to generate-and-check candidate
strings (to stay robust to '-' inside a name) rather than read it; with `run` a
plain int, `next_run` is a clean numeric max+1 over the field. The split is more
normalised (one fact per field) and the only property the fused token bought — a
single paste-able CLI handle — is recovered by deriving it. Storage and all
user-visible output stay byte-identical: append_family still returns
"{name}-{run}", Family.id is still "{name}-{run}", and the CLI prints the same
"family_id" lines (the json! key + the returned String, both unchanged).
Internal-only: FamilyRunRecord and group_families (now keyed on the (family,
run) tuple, first-seen order preserved) change; lib.rs re-exports, the
extractors, and all of aura-cli are untouched (everything downstream goes
through Family.id + append_family's returned String). The on-disk
families.jsonl key changes ({"family","run"} vs {"family_id"}) — a gitignored
runtime artifact with no committed fixture, and tests round-trip within a temp
dir, so no migration is owed. Doc reconciliation: the lineage/lib module headers
and the C18 ledger realization note now describe the split shape.
Verification: a workflow ran the refactor in an implementer agent, then three
adversarial lenses (behaviour-preservation, new-logic correctness,
completeness/doc-lag) tried to refute it — the first two found nothing, the
third flagged the four doc-lag sites now fixed here. Self-run gates:
cargo test --workspace green (registry 11, incl. a strengthened round-trip test
pinning the split fields + derived handle), cargo clippy --workspace
--all-targets -D warnings clean, cargo doc clean.
|
||
|
Brummel
|
a168f07cbb |
audit: cycle 0045 — registry lineage (C18 ledger note; flat-store follow-up #73)
Architect drift review (eeba218..HEAD) + regression gate (the project's cargo test --workspace is the gate; no separate regression script). What holds (architect): C9 layering intact — aura-engine gains no registry dependency, lineage lives entirely in aura-registry. #71 firewall held — RunManifest/aura-core byte-untouched, no blob/path/payload field on FamilyRunRecord, the member window is producer-supplied via Source::bounds()/window_of, never a Vec scan. C2 clean — bounds() is cursor-independent (no look-ahead). Family store is a disjoint sibling of the flat store (pinned by a disjointness test). Regression gate green: 228 tests, clippy --workspace --all-targets -D warnings clean. Drift resolved: - [fix, this commit] C18 ledger gap: docs/design/INDEX.md C18 carried no realization note for the run registry. Added the cycle-0029 (flat registry) and cycle-0045 (lineage as related records / family store) realization notes, in the ledger's established per-cycle style. - [forward-queue] Flat-store CLI dead-end: after this cycle no CLI command writes the flat runs.jsonl (sweep/walkforward moved to the family store, aura run never persisted), so aura runs list / rank read an unfed store. This is a spec-accepted state (0045 kept runs list "for standalone runs only") and a genuine design fork (give aura run a persisting path vs. retire the flat-store CLI surface) with no clear default — filed as #73 rather than decided here. Recorded as a deferred follow-up in the C18 ledger note. - [carry-on] Low/cosmetic: family-wrapped CLI stdout alphabetizes nested manifest keys (serde_json::json!) vs. declaration order (to_json()); both valid JSON, the stored families.jsonl uses declaration order, round-trip unaffected. Noted as a sub-item on #73. refs #70 |
||
|
Brummel
|
bcd1072ca8 |
feat(aura-cli): aura mc + family-aware runs + window_of migration (0045 iter 2)
Iteration 2 of spec 0045 — the CLI surface for orchestration families. window_of migration: every manifest-window scan (run_sample, sweep_family, walkforward_family, sweep_over, run_oos, run_macd) now reads the window from Source::bounds() via window_of(&sources) instead of prices.first()/.last(). Behaviour-preserving (the walk-forward roller boundaries are bar-aligned, so the producer-supplied window equals the old first/last) — pinned by the unchanged run_sample (1,7) and walk-forward determinism tests. aura mc: new built-in Monte-Carlo family on the CLI (monte_carlo over a built-in seed set + the sample harness re-seeded per draw), persisted via append_family and rendered as one member line per seed (carrying the family_id) plus an mc_aggregate line. McAggregate is not Serialize, so the aggregate line is built from its three MetricStats blocks. family-aware runs: aura sweep / walkforward / mc now persist to the family store via append_family with an optional --name (per-kind default), printing the assigned family_id per member. aura runs families lists family headers (id, kind, member count); aura runs family <id> [rank <metric>] lists/ranks one family's members as a unit. An unknown family id is an empty family (exit 0); an unknown metric is a usage error (exit 2). Two pre-existing cli_run.rs E2E tests were adapted to the new family contract (a required consequence of sweep/walkforward switching from the flat store to the family store — a plan gap caught at implement time): the sweep-output test now asserts the family-wrapper, and the runs list/rank flow became the family list / per-family-rank flow (which also exercises the per-name counter sweep-0/sweep-1 end-to-end). A new E2E test drives aura mc through the binary. Known cosmetic detail: the family-wrapped sweep/mc/walkforward stdout nests the report via serde_json::json! (to_value -> alphabetical keys, leading "broker"), while runs family / runs list print via RunReport::to_json() (declaration order, leading "commit"). Both valid JSON; key order is not load-bearing and the stored families.jsonl uses to_string (declaration order), so the round-trip is unaffected. Verification: cargo test --workspace green; cargo clippy --workspace --all-targets -D warnings clean. refs #70 |
||
|
Brummel
|
1d9555c468 |
feat(aura-engine,aura-registry): producer-supplied window + registry lineage (0045 iter 1)
Iteration 1 of spec 0045 — the engine + registry core for storing orchestration families as linked records. aura-engine: add Source::bounds() — the inclusive (from, to) data extent a producer will stream, known without materialization (#71 firewall) — and a free window_of() that folds the union extent across a run's sources. This is the producer-supplied replacement for the prices.first()/.last() Vec scans at the CLI call sites (those migrate in iteration 2); a lazy producer now reports its window so stored lineage stays byte-identical whether a source is eager or streamed. aura-ingest: M1FieldSource stores its requested [from_ms, to_ms] window at open and reports it via bounds() (normalized to epoch-ns; None when open-ended — the archive-extent query is a deferred non-goal). aura-registry: new lineage module — FamilyKind / FamilyRunRecord (a RunReport stamped with family_id + kind + ordinal) / Family, plus Registry::append_family (assigns family_id = "{name}-{counter}" via a per-name generate-and-check counter, writes members to a sibling families.jsonl) and load_family_members; group_families re-derives the families (the round-trip), and the three *_member_reports extractors pull the per-kind member reports. The flat runs.jsonl store and its append/load/rank_by/optimize API are byte-for-byte unchanged (a test pins the two stores' disjointness). C9 preserved: aura-engine gains no registry dependency. New dep: serde derive on aura-registry (per-case policy, same basis as serde_json already is). Verification: cargo build --workspace; cargo test --workspace (228 green, incl. 3 new engine bounds/window_of, 1 data-gated ingest bounds run against real archive data, 5 new registry lineage round-trip/counter/disjointness tests); cargo clippy --workspace --all-targets -D warnings clean. The CLI surface (aura mc, family-aware aura runs, the window_of migration) is iteration 2. refs #70 |
||
|
Brummel
|
d6daaef009 |
plan: 0045 registry lineage for orchestration families
Decompose spec 0045 into two iterations. Iteration 1 (engine + registry core): Source::bounds() + window_of, M1FieldSource::bounds(), and the aura-registry lineage module (FamilyKind / FamilyRunRecord / Family, group_families, next_family_id, the three *_member_reports extractors, Registry::append_family / load_family_members) with round-trip and per-name-counter tests. Iteration 2 (CLI surface): aura mc, family-aware aura runs (families / family [rank]), sweep/walkforward family persistence via append_family with --name, and the window_of migration of the manifest-window scans. refs #70 |
||
|
Brummel
|
bf829f605b |
spec: 0045 registry lineage for orchestration families (boss-signed)
Persist sweep / Monte-Carlo / walk-forward runs as named, linked families
(C18 lineage / C21): a FamilyRunRecord stamps each member with a shared
family_id = "{name}-{counter}" in a sibling families.jsonl store, group_families
re-derives a family as a unit, and the CLI gains aura mc + family-aware aura runs.
Producer-supplied window via Source::bounds()/window_of (the #71 firewall: no Vec
scan, no input artifact). family_id form (name + counter, not a content-hash)
resolved by the user in-session; reconciliation comment on the issue records it.
Auto-signed under the boss spec-auto-sign gate: all objective gates green
(precondition, self-review, grounding-check PASS) and a unanimous five-lens
spec-skeptic panel (criterion, grounding, scope-fork, ambiguity, plan-readiness).
refs #70
|
||
|
Brummel
|
eeba2182d7 |
audit: cycle 0044 — drift-clean (walk-forward family)
Architect drift review over 61a8436..HEAD (the 0044 walk-forward cycle: spec |
||
|
Brummel
|
4764656062 |
feat(aura-engine): walk-forward family — WindowRoller + walk_forward (C12 axis 3)
The third C12 orchestration axis: walk-forward varies the data window. A
WindowRoller is a pure iterator of WindowBounds { is, oos } — bounds only, never
tick data (#71 firewall). walk_forward runs a caller closure per split disjointly
over the shared run_indexed core (C1), and stitches the OOS pip-equity segments
into one continuous curve (each segment offset by the running sum of prior
segments' finals; an empty segment contributes 0.0, mirroring summarize). C2
no-look-ahead is structural: the roller only ever emits oos.0 == is.1 + 1, pinned
by a zero-tick bounds test. The in-sample optimize (axis 2) is closure-supplied,
not called by the engine: aura-engine has no aura-registry dependency (C9), and
C12 forbids baking search policy into the primitive — the CLI bridges both crates.
Param-stability is on-demand, not stored (the R2 decision, recorded with
provenance on #69): WalkForwardResult stores only the raw per-window outcomes +
the stitched curve, and param_stability(&result) -> Vec<MetricStats> is a public
reduction over the retained per-window chosen params. A stored summary would be
recomputable from the windows (redundant) and would force one statistic canonical
when "stability" admits several; this mirrors SweepFamily (raw points + on-demand
optimize), not McFamily (stored aggregate). MetricStats::from_values is extracted
from McAggregate::from_draws (behaviour-preserving — the 7 mc tests stay green)
and gains serde so the CLI summary renders it and #70 lineage can persist it.
`aura walkforward` runs a built-in rolling walk-forward over a synthetic windowed
source: per window it sweeps the built-in grid in-sample, optimizes by total_pips,
runs the chosen params out-of-sample, persists each OOS RunReport (C18), and prints
a stitched summary line.
Two plan deviations, both compiler-forced and consistent with siblings: added
#[derive(Debug)] to WindowRoller (the RED tests' unwrap_err needs Ok: Debug; the
sibling types already derive Debug); removed a now-redundant test-module
SyntheticSpec import after Step 2 hoisted it to the top-level use.
Verification: cargo test --workspace green (aura-engine 152 incl. walkforward 9 +
mc 9, aura-cli 14); clippy --workspace --all-targets -D warnings exit 0; cargo doc
--workspace --no-deps clean.
refs #69
|
||
|
Brummel
|
7d52eff41f |
plan: 0044 walk-forward family (C12 axis 3)
Bite-sized, placeholder-free plan for the walk-forward axis, in three tasks: (1) extract MetricStats::from_values from McAggregate::from_draws + serde derive (behaviour-preserving, the 7 mc tests guard); (2) the walkforward module — WindowRoller (bounds-only iterator), walk_forward over the shared run_indexed core, continuous OOS stitch (empty segment -> +0.0), on-demand param_stability, + crate exports + module-header doc-lag fix; (3) the aura walkforward CLI demo bridging engine+registry via optimize, with serde_json for the summary line. RED-first tests 1-11 (incl. 7b empty-segment stitch) per the spec. refs #69 |
||
|
Brummel
|
a4712eb336 |
spec: 0044 walk-forward family (C12 axis 3)
Walk-forward orchestration: a WindowRoller rolls (in-sample, out-of-sample) splits over a time span, walk_forward runs a disjoint harness per split via the shared run_indexed core (C1), and stitches the OOS pip-equity segments into one continuous curve. The varying dimension is the data window (axis 3), realized eager-agnostically (#71): the public surface carries WindowBounds + a per-window closure, never a materialized stream Vec; C2 no-look-ahead is a pure bounds invariant (oos.0 > is.1), checkable with zero ticks. The in-sample optimize (axis 2) is closure-supplied, not called by the engine: aura-engine cannot depend on aura-registry (C9), and C12 forbids baking search policy into the primitive. The CLI bridges both crates in run_walkforward. Param-stability shape (the load-bearing fork) resolved with the user as R2: on-demand, not stored. WalkForwardResult stores only raw per-window outcomes + the stitched curve; param_stability(&result) -> Vec<MetricStats> is a public on-demand reduction over the retained per-window chosen params (sweep-precedent: summaries computed on demand, not cached, vs McFamily's stored aggregate). A stored summary would be recomputable from the raw windows (redundant) and would force one statistic canonical when "stability" admits several. Recorded with provenance in the #69 reconciliation comment. MetricStats::from_values is extracted from McAggregate::from_draws (behaviour-preserving; the 7 mc tests guard) so the MC aggregate and the helper share one reduction. Empty OOS segment contributes 0.0 to the stitch offset (mirrors summarize's unwrap_or(0.0)). Gates: Step-1.5 precondition clean (no fork silently picked), self-review clean, grounding-check PASS (twice — re-run after each edit). Auto-sign panel escalated on the scope-fork design lens (param-stability shape) to human sign-off; the user resolved it (R2), so this is a user-signed spec, not boss-signed. refs #69 |
||
|
Brummel
|
61a8436c2c |
audit: cycle 0043 — drift-clean (monte-carlo family)
Architect drift review over b920362..HEAD (the 0043 monte-carlo cycle). Cycle is sound: C12 axis 4 faithfully realized (monte_carlo varies the seed, the 0042 seeded source genuinely perturbs each run, so distinct_seeds_produce_distinct_draws is a real property); C1 honoured (one run_indexed core, output sorted by job index = input order independent of completion, 1-vs-8-thread determinism pinned, parallelism across draws never within); the run_indexed extraction is a genuine behaviour-preserving refactor (C11/C23 — the three pre-existing sweep tests stay green, and both family executors are thin adapters over the single core; the points.len() -> n.max(1) clamp is equivalent since GridSpace rejects empty axes); the #71 eager-agnostic firewall is respected (API takes seeds + a per-draw closure, no materialized stream Vec crosses the boundary). One medium doc-lag found and fixed here: aura-engine/src/lib.rs module header listed Monte-Carlo (and, already stale, the grid sweep) under "Still to come"; both orchestration axes ship now, so the header records the grid axis (sweep / SweepFamily) and the seed axis (monte_carlo / McFamily) as delivered, with walk-forward, random param-sweep, registry lineage, and the C10 position-event / broker layer remaining future. Regression gate (test suite — no dedicated regression scripts in project facts): cargo test --workspace green (incl. 7 new mc::tests + the sweep guard tests); clippy --workspace --all-targets -D warnings clean; cargo doc -p aura-engine clean (the new intra-doc links resolve). closes #68 |
||
|
Brummel
|
e2c550f2f9 |
feat(aura-engine): monte_carlo family — McFamily over a seed set (C12 axis 4)
`monte_carlo(base_point, seeds, run_one) -> McFamily`, the Monte-Carlo
orchestration family: a fixed base point run over a seed set, each seed a
disjoint C1 realization. C12 axis 4 — Monte-Carlo IS a sweep over seeds, so it
reuses the disjoint-parallel executor rather than forking a run loop.
The reuse is realized by extracting the index-parallel core out of
`sweep_with_threads` into a `pub(crate) run_indexed<T>` (generic over the
per-job result, over a 0..n job range); `sweep_with_threads` becomes a thin
adapter over it. Behaviour-preserving (C1/C11/C23): the three pre-existing sweep
tests stay green — they ARE the proof the refactor preserved behaviour. A small
incidental win: the adapter consumes `points` via zip, dropping the old
per-point clone.
mc.rs adds McDraw{seed,report} / McFamily{draws,aggregate} (draws in seed-INPUT
order, independent of thread completion) / McAggregate (mean + p5/p25/p50/p75/p95
of all three run metrics — the V1 decision: covers every metric, not a "chosen"
one) / MetricStats, a private type-7 linear-interpolation `quantile`, and
`McAggregate::from_draws` (a pure post-run reduction recomputable from the
retained raw draws). Eager-agnostic firewall (#71): the API takes seeds + a
per-draw closure `Fn(u64,&[Scalar])->RunReport`, never a materialized stream Vec;
seed->Source construction stays a closure-body concern. An empty seed set is a
debug-asserted precondition, preserving the no-Result `-> McFamily` signature.
7 RED-first mc::tests pin: one draw per seed in input order; determinism across
1 vs 8 threads (C1); draw == an independent seeded run; distinct seeds perturb
the family; aggregate == from_draws(draws); type-7 quantile/mean math on a known
fixture; quantile endpoints + singleton. Workspace green, clippy -D warnings
clean, cargo doc clean.
refs #68
|
||
|
Brummel
|
5181a7132c |
plan: 0043 monte-carlo family — McFamily over a seed set
Three tasks: (1) extract the disjoint-parallel core out of sweep_with_threads into a shared `pub(crate) run_indexed<T>` (behaviour-preserving; the three pre-existing sweep tests are the green guard), (2) add the `mc.rs` module (McDraw/McFamily/McAggregate/MetricStats, type-7 `quantile`, `McAggregate::from_draws`, `monte_carlo`/`monte_carlo_with_threads` driving run_indexed) with 7 RED tests, (3) wire `mod mc;` + the five public exports into lib.rs. RED-first; module-declaration ordering keeps mc.rs's tests dark until the final wire-up task. refs #68 |
||
|
Brummel
|
0bbc8190b8 |
spec: 0043 monte-carlo family — McFamily over a seed set (boss-signed)
C12 axis 4: Monte-Carlo as a sweep over seeds. `monte_carlo(base_point, seeds, closure) -> McFamily`, the analog of SweepFamily, reusing the disjoint-parallel executor (extracted into a shared `run_indexed` core) rather than forking a new run loop. Each realization is a disjoint C1 unit; parallelism is across draws. The aggregate (V1) covers all three run metrics with mean + p5/p25/p50/p75/p95, a pure post-run reduction over the retained raw draws. Eager-agnostic firewall (#71): the API takes seeds + a per-draw closure, never a materialized stream Vec; seed->Source construction stays a closure-body concern. Signed under the /boss spec auto-sign gate: objective gates green (precondition, self-review, grounding-check PASS) and a unanimous five-lens spec-skeptic panel (criterion, grounding, scope-fork, ambiguity, plan-readiness) returned SOUND. The aggregate fork was settled to V1 by the user (issue #68 reconciliation comment, verbatim "ja, wir machen v1", 2026-06-15). refs #68 |
||
|
Brummel
|
b9203624d3 |
audit: cycle 0042 — drift-clean (seed-as-input)
Architect drift review over 682e459..HEAD (the seed-as-input cycle plus the prior unaudited `aura run --real` work). Cycle is sound: Fork B respected (seed forks at the data-generation edge into the source and the manifest; the engine loop sees only a Source — C12/C1/C3 intact), the contract names Source never Vec (precondition for #68/#52 delivered as specced), and the four seed-free callers thread 0 in lockstep. One medium doc-lag item found and fixed here: aura-engine/src/lib.rs module-doc listed the seed axis under "still to come"; seed-as-input shipped this cycle and RunManifest.seed is now live, so the headline doc now records SyntheticSpec as a delivered seeded Source producer (params + orchestration families remain future). One low carry-on note (not drift): run_sample_seeded exists only under #[cfg(test)] — intended this cycle (no --seed CLI flag per spec non-goals); the only non-test seed value remains 0. A scaling marker for #68, not debt. Regression gate (test suite — no dedicated regression scripts in project facts): cargo test --workspace green; clippy --workspace --all-targets -D warnings clean; cargo doc -p aura-engine clean (the new [`SyntheticSpec`] intra-doc link resolves). |
||
|
Brummel
|
c9f0e438e1 |
feat(aura-engine): seeded source + live RunManifest.seed (C12 seed-as-input)
A seeded source whose stream is fully determined by a u64 seed, making RunManifest.seed a live captured input instead of the dead 0 it was. This is C12's reconciliation of stochastic runs with C1: a run is bit-identical for a fixed seed (the seed is a captured input, not hidden nondeterminism), and a different seed gives a different-but-reproducible run. Precondition for the Monte-Carlo family (#68) and random param-sweep (#52). Engine (aura-engine): - SplitMix64: a ~10-line private, dependency-free PRNG. Chosen over a rand-family crate for guaranteed bit-stability across toolchains/crate versions — the whole value of seed-as-input is reproducibility, and a crate whose algorithm is not version-stable would silently break a recorded seed on a dep bump. No new direct dep. - SyntheticSpec + source(seed) -> impl Source: the Fn(u64) -> impl Source contract (Fork B — seed at the data-generation edge, outside the engine graph). Returns a Source, never a Vec, so the MC family can re-seed N times without materializing N streams. First cut collects to a VecSource internally; the signature does not name Vec, so a lazy generator is a drop-in replacement. Re-exported from lib.rs beside VecSource. CLI (aura-cli): - sim_optimal_manifest gains a seed: u64 param recorded into the manifest (the single Fork B capture site). All four existing seed-free callers (run_sample, run_sample_real, the sweep grid-report, run_macd) pass 0 unchanged — their determinism tests stay green. - A test-only run_sample_seeded + SeededTrace exercise a live seed and expose the drained sink trace, so the three e2e tests assert bit-identity at the trace level (strictly stronger than the folded 3-field metrics). Tests (RED-first): 2 engine producer tests (same-seed identical / different-seed differs) + 3 cli e2e tests (bit-identical trace, different-seed metrics differ, seed recorded in manifest). Full workspace green; clippy -D warnings clean. Errata vs plan 0042 (two compile-forced deviations, neither alters the spec contract): - source returns `impl Source + use<>`: under edition 2024 a bare `impl Source` captures the &self lifetime, which blocks boxing as Box<dyn Source + 'static> for run() (E0597). The source owns its data, so use<> (captures nothing) is correct. - SyntheticSpec is imported inside `#[cfg(test)] mod tests` in main.rs: it is used only by the seeded test vehicle, so a top-level import is unused in the non-test build under clippy -D warnings. closes #66 |
||
|
Brummel
|
f7230a5068 |
plan: 0042 seed-as-input — seeded source + live RunManifest.seed
Two tasks: (1) aura-engine SplitMix64 PRNG + pub SyntheticSpec::source(seed) -> impl Source beside VecSource, with producer RED tests; (2) aura-cli sim_optimal_manifest gains a seed param (four callers pass 0), plus a test-only run_sample_seeded + SeededTrace and three e2e RED tests pinning the #66 acceptance. RED-first throughout. refs #66 |
||
|
Brummel
|
5604c420c5 |
spec: 0042 seed-as-input — seeded source + live RunManifest.seed (boss-signed)
A seeded source whose stream is fully determined by a u64 seed, wired so the seed reaches RunManifest.seed at the manifest-construction site (Fork B: seed at the data-generation edge, outside the engine graph). Contract is a producer Fn(u64) -> impl Source, never seed -> Vec, so the Monte-Carlo family can re-seed N times without materializing N streams. Deterministic dependency-free PRNG (SplitMix64) for C1 bit-stability. Precondition for #68 (Monte-Carlo) and #52 (random sweep). Auto-signed under /boss spec auto-sign: objective gates green (precondition, self-review, grounding-check PASS) and a unanimous five-lens spec-skeptic panel (criterion, grounding, scope-fork, ambiguity, plan-readiness all SOUND) after one editorial-repair round. refs #66 |
||
|
Brummel
|
0a2e2022a5 |
docs(aura-cli): note the --real real-data path in the module header
The header described `aura run` as synthetic-only; --real now also streams real M1 bars through the #71 Source seam. Keep the entry-point doc honest. |
||
|
Brummel
|
0ce843bada |
feat(aura-cli): aura run --real <SYMBOL> backtests the sample on real M1 bars
The first real-data backtest from the CLI: `aura run --real <SYMBOL> [--from <ms>] [--to <ms>]` runs the existing built-in sample signal-quality harness over real M1 *close* bars instead of synthetic prices, printing the same RunReport JSON as `aura run`. Bars are streamed lazily through aura_ingest::M1FieldSource (a Box<dyn Source>) — dogfooding the #71 Source seam, O(one chunk) resident, never eager. aura-ingest + data-server enter aura-cli's deps (data-server git line mirrored from aura-ingest). M1 only this cycle, deliberately. The TickSource is deferred: raw ticks are the wrong input for the bar-semantic SMA sample (an SMA over ticks is microstructure noise until a resampler node exists, C2), and tick's real driver is the realistic broker's bid/ask execution (C10) — it should arrive with that consumer, not speculatively here. Tick data is also local to only EURUSD/XAUUSD/GER40, whereas M1 covers the test symbol AAPL.US. Shape: run_sample_real builds sample_harness(), guards has_symbol / M1FieldSource::open (unknown symbol or empty window -> 'aura: no local data' + exit 2, never a panic), and reuses sim_optimal_manifest + the run_sample fold. parse_real_args is a pure, unit-tested grammar (mandatory symbol, then --from/--to in any order; bad/missing/duplicate flag -> Err). Data path is data_server::DEFAULT_DATA_PATH (no --data-path flag this cycle). Known simplification: the manifest window is derived by draining a *separate* single-pass probe source for first/last ts, so an unbounded window streams the archive twice (once to bound, once to run). Acceptable and O(1)-resident for now; a future Harness::run could surface the first/last cycle ts to drop the probe pass. Verified: cargo build/test/clippy --workspace all green; the gated run_sample_real_streams_real_close_bars_deterministically RAN (not skipped) over AAPL.US 2006-08; manual CLI smoke of the success + two error paths. |
||
|
Brummel
|
682e459554 |
audit: cycle 0041 — drift-clean (Source ingestion seam)
Architect drift review over 8b330e3..HEAD (the Source-seam cycle #71 plus the interim #67 optimize work and refactors). No semantic or contract drift: the seam faithfully preserves C3 (ms→epoch-ns at the one ingestion boundary), C4 (tie-by-source-index, byte-for-byte), and C7 (field-per-source SoA); the full suite is byte-identical green and the gated streaming tests pass on real data. Regression gate: `cargo test --workspace` green (the project declares no dedicated regression script; the suite is the gate). Tidied stale prose the cycle left behind (no behaviour change): - docs/design/INDEX.md (C12): replaced the cycle-0011 "deliberate eager gap" status note with a cycle-0041 realization note — the producer `Source` seam + streaming `M1FieldSource` now deliver per-source O(one-chunk) streaming; precisely scopes what remains open (cross-*sim* Arc<[T]> window sharing, still unbuilt until the orchestration families #66/#68/#69 consume it). - aura-core/src/lib.rs: dropped the now-shipped "Source trait + data-server ingestion" from the module doc's "still to come" list (aura-engine's twin line was fixed in the cycle; aura-core's was missed). - aura-ingest/src/lib.rs: module header now documents both coexisting paths (eager load_m1_window/M1Columns vs lazy streaming M1FieldSource). - aura-registry/src/lib.rs: rank_by/optimize doc comments linked to the private `metric_cmp` (a public→private intra-doc-link warning from the #67 commit); demoted to plain code spans, so `cargo doc --workspace` is now warning-clean. |
||
|
Brummel
|
8158e97dc6 |
feat(aura-ingest): streaming M1FieldSource — lazy data-server window through the seam
Completes the Source ingestion seam (plan Tasks 3-4): a real, lazily-streamed
data-server source proving the producer seam against the actual data source,
and closing the cycle-0011 deliberate eager-materialization gap.
- M1Field + a free decode(field, bar) — a pure projection of one M1 bar into
(epoch-ns ts, Scalar), reusing unix_ms_to_epoch_ns so ms->ns stays the one
C3 normalization point. Pure ⇒ hermetically unit-tested without an iterator.
- M1FieldSource: a streaming Source over a data-server M1 window. Holds one
Arc<[M1Parsed]> chunk (a zero-copy clone of the cache's chunk) + a cursor +
a pre-decoded head (so peek(&self) is cheap over a &mut next_chunk refill);
constructs each Scalar per-pull, refills when the chunk drains. Resident
footprint is O(one chunk), independent of window length.
- aura-engine moved [dev-dependencies] -> [dependencies] (M1FieldSource is a
library item implementing aura_engine::Source, not a test-only type).
- A gated streaming integration test (skips where local data is absent):
* end-to-end backtest through the seam to a RunReport, C1-deterministic
(two runs bit-identical);
* residency predicate — driving the source the way run() does (peek/next)
over the *unbounded* archive (~1.77M bars locally), peak resident_records
stays <= one data-server chunk (CHUNK_SIZE) while total >> CHUNK_SIZE:
O(one chunk), NOT O(window). A O(window) source would peak at `total`;
* two-field sharing (close + volume over one window, same ts axis).
The eager load_m1_window / close_stream path is kept (still valid for bounded
loads); the gap is closed by a streaming path now existing, not by deleting the
eager one. The residency test drives the full archive rather than the plan's
narrow 2006-08 window, which holds only ~21 bars locally — too few for the
> CHUNK_SIZE non-vacuity precondition; the unbounded archive is the stronger
"independent of window length" proof.
Verified: cargo build/test/clippy --workspace clean; the 3 gated tests ran
against real local data and passed; aura-ingest/aura-engine docs clean (two
pre-existing aura-registry doc-link warnings are from an earlier commit, not
this cycle).
closes #71
|
||
|
Brummel
|
eec129ee51 |
feat(aura-engine): Source ingestion seam — run() takes Vec<Box<dyn Source>>
The engine's ingestion input was the only type encoding an eager-dataflow
assumption: `Harness::run(Vec<Vec<(Timestamp, Scalar)>>)` — one fully
materialized stream per source. At realistic research scale (20y, 3 tick
streams, ~7.9e9 ticks) that layout is ~190 GB resident, non-residable. Yet
the merge loop never needs the whole stream: it only ever peeks the head
timestamp of each live source and pops one record. So the eager Vec is
gratuitous.
This lands the first half of the Source seam (plan Tasks 1-2):
- A `Source` trait (`peek(&self) -> Option<Timestamp>`,
`next(&mut) -> Option<(Timestamp, Scalar)>`), object-safe so the merge
holds `Vec<Box<dyn Source>>`. The peek/next split is the producer contract
the k-way merge drives.
- `VecSource`, an adapter over the old `Vec<(Timestamp, Scalar)>` — the
cycle-0011 eager shape, named. It IS the old behaviour.
- `Harness::run` re-typed to `Vec<Box<dyn Source>>`; the merge loop's index
arithmetic becomes peek/next. C4 tie-by-source-index is preserved (the
strictly-`<` replace + source-order scan is byte-for-byte the old
semantics). The forward+eval body is unchanged.
- All 53 call sites threaded VecSource-wrapped in the same change, so the
workspace compiles atomically and run output is byte-identical (C1).
Behaviour-preserving is the load-bearing guarantee: the full suite is green
and unchanged (the two-run C1 determinism pairs — real_bars r1/r2, blueprint
sweep-equality, harness h/h2 — stay byte-identical); aura-engine unit tests
129 -> 132 (+3 VecSource unit tests). No residency change yet: VecSource holds
the same Vec as before. The lazy data-server-backed streaming source and its
end-to-end residency proof are the second half (plan Tasks 3-4).
refs #71
|
||
|
Brummel
|
9337a8585d |
plan: 0041 source ingestion seam
Decomposes the boss-signed spec into four tasks: (1) the `Source` trait + `VecSource` adapter in aura-engine; (2) the atomic `Harness::run` re-type to `Vec<Box<dyn Source>>` with all 53 call sites threaded behaviour-preserving; (3) the streaming `M1FieldSource` over a data-server window; (4) a gated streaming e2e + the measured residency predicate. refs #71 |