Milestone-close gate for "The World, part II — orchestration families".
Four end-to-end scenarios derived top-down from the milestone promise,
each built from HEAD and run as a downstream consumer through the public
API + CLI only (no crates/*/src read):
- mw_3 (headline): real EURUSD-M1 walk-forward composing all four axes
(grid sweep + optimize in-sample, OOS stitch) over the M1FieldSource
seam + lineage — 5 windows, 145 200-pt stitched curve, C2 held.
- mw_2: monte_carlo over 128 seeds, reproducible McFamily (C1) + aggregate.
- mw_1: random sweep -> optimize -> 64-member lineage round-trip.
- mw_4: family-lineage CLI transcript.
Findings: 4 working (promise empirically delivered), 1 bug, 3 friction.
- bug F5: `aura runs list`/`rank` cannot read a pre-0047 runs.jsonl — the
cycle-0047 typed-Scalar params migration (86746e3) left the persisted
flat store unmigrated; load() fails at the first bare float (+ exits 0).
- friction F6: no epoch_ns_to_unix_ms inverse for real-data walk-forward.
- friction F7: DataServer/DEFAULT_DATA_PATH not re-exported by aura-ingest.
- friction F8: family store is directory-keyed, undocumented on Registry::open.
Spec: docs/specs/fieldtest-milestone-the-world-orchestration-families.md
Add the typed/named convenience view on `WalkForwardResult`, closing the
symmetry cycle 0048 left open: it shares the `(space, tag-free Vec<Cell>)`
idiom with `SweepFamily` but had no named-params accessor, so a consumer
hand-wrote `zip_params(&result.space, &result.windows[i].run.chosen_params)`.
`named_params(window)` pairs each param-space name with that window's chosen
value in slot order, delegating to `zip_params` — the direct mirror of
`SweepFamily::named_params`. A derived view, no new stored state.
closes#76
Add `RandomBinder`, the by-name sibling to `SweepBinder`, so a random
sweep (`RandomSpace`, C12.1) can be built by name against `param_space()`
instead of by positional `Vec<ParamRange>` slot order. By-name resolution
makes a same-kind transposition (e.g. swapping the I64 ranges for
`fast.length` and `slow.length`) structurally impossible — the failure
class `RandomSpace::new`'s positional validation passes silently.
Direct structural mirror of `SweepBinder`: `Composite::range` opens the
binder, `.range(name, ParamRange)` accumulates, `.sweep(count, seed, run)`
resolves the named ranges via the shared `resolve_into` (new `resolve_ranges`
caller) and runs the disjoint sweep. New `BindError::EmptyRange` mirrors
`EmptyAxis`; `ParamRange::is_empty` homes the non-empty invariant the named
layer pre-checks so `RandomSpace::new` cannot fail.
closes#79
spec auto-sign is now fixed /boss behaviour in the skills plugin (no longer a per-project toggle), so the explicit 'enabled' entry is redundant. No behaviour change — auto-sign was already on for this project.
The SweepError type-level rustdoc still read "A structural fault constructing a
`GridSpace`" though the enum now also gates `RandomSpace::new` (the three
random-only variants are each documented individually, but the type summary was
stale). Broaden it to name both spaces and group the grid vs. random faults.
Doc-only, behaviour-preserving — the same low-grade doc-debt class the 0049
audit fixed inline for the module doc. Surfaced by the cycle-0049 fieldtest.
refs #52
Public-API field test of the random param-sweep surface, from a standalone
downstream-consumer crate (path-deps only; the public interface = ledger +
glossary + spec 0049 + cargo doc rustdoc; no crates/*/src read). Four bins,
each built from HEAD and run: continuous tuning (200-point random tune ranked
by total_pips), the typed validation gate (all five reachable SweepError
variants pre-run), reproducibility + seed-sensitivity + the full i64::MIN..=MAX
sampler edge, and Space-trait interchangeability (one tune_and_rank<S: Space>
over both GridSpace and RandomSpace).
Findings: 0 bugs, 4 working, 2 spec_gap, 1 friction. The four working findings
confirm the cycle's acceptance criterion empirically — the headline tune reads
as the code a researcher would write, the gate is precise and fires before any
run, the C1 reproducibility promise is checkable in one line, and the Space
trait delivers one-consumer/both-enumerations.
Triage of the actionable findings:
- friction (no named-axis builder for RandomSpace — positional Vec<ParamRange>
must align with param_space() by hand, and a same-kind transposition passes
validation silently): filed as a feature for a future cycle, refs #79
(a RandomBinder sibling to the grid's SweepBinder).
- spec_gap (SweepError rustdoc summary named only GridSpace): fixed inline in a
follow-up doc commit.
- spec_gap (NonNumericRange / Bool-slot ranges unreachable with the shipped
aura-std node roster — no node declares a Bool/Timestamp knob): RATIFIED as
intentional. The variant is a forward-looking structural guard for the C16
"author your own node" path (a Bool/Timestamp param-slot a custom node may
declare); its current untriggerability with the standard roster is expected,
not drift.
refs #79
Architect drift review over 3de00e2..HEAD (the 0049 spec/plan/feat plus the
two intervening refactors that had not been audited: aura-ingest M1 transpose
6390093, aura-std SMA perf 67c1f51). Verdict: feature clean — C1 determinism
preserved (RandomSpace points seed-determined before any run; grid path
behaviour-preserving via trait-forwards-to-inherent), the #52/#71 source-seam
firewall honoured in code (sweep stays &S: Space + Fn(&[Cell]) -> RunReport, no
Source type enters the sweep layer; SplitMix64 a code-path-disjoint instance),
and both intervening refactors behaviour-preserving (pinned by
incremental_matches_full_resum_within_tolerance and
chunked_accumulation_equals_single_transpose).
Two low-grade doc-debt items found and fixed inline (doc-only, behaviour-
preserving):
- sweep.rs module doc named only the grid axis; refreshed to name RandomSpace
+ the Space trait the module now also owns.
- C12 ledger had no realization note for axis-1 (param-sweep); added one
recording the grid (0028) + random (0049) landing and the Space-trait
unification.
Regression gate: the project configures no regression scripts, so the test
suite + lint are the gate. Independently verified: cargo test --workspace green
(incl. the 17 new engine tests + 6 public-API E2E), cargo clippy --workspace
--all-targets -- -D warnings clean.
Drift-clean, not a milestone close (no milestone fieldtest run this cycle).
Ship the random half of the C12.1 param-sweep axis (grid landed in 0028).
A new `Space` trait (`points`/`param_specs`) generalizes `sweep`/`sweep_with_threads`
from `&GridSpace` to `&S: Space`; `GridSpace`'s trait impl forwards to its existing
inherent methods, so the grid path is behaviour-preserving (C1, every pre-existing
grid sweep test stays green). `RandomSpace` is the second `Space` producer: N seeded
uniform points over per-slot typed `ParamRange { lo, hi }` ranges (I64 inclusive
[lo,hi], F64 half-open [lo,hi)), validated in `new` against the param-space — a
non-numeric slot, a range-kind mismatch, and an empty range are the three new typed
`SweepError` variants, all caught before any run. Sampling reuses the existing
bit-stable `SplitMix64` (promoted to `pub(crate)`) as a code-path-disjoint instance
from the data-edge seed RNG — the #52/#71 World-II source-seam firewall: they share
only the `u64` type, never a path. The sweep-layer signature stays param-only
(`Fn(&[Cell]) -> RunReport`); no `Source`/stream type enters it.
Deviation from the spec's verbatim sampler, accepted as a correctness hardening:
the I64 draw guards the full-width span ([i64::MIN, i64::MAX] computes a span of
2^64 that wraps a u64 to 0) and uses `lo.wrapping_add(draw as i64)` instead of a
plain `+`. This avoids both the `% 0` divide-by-zero at the full domain and the
signed-overflow panic the literal form hits on wide ranges in debug builds, while
preserving uniform-over-[lo,hi] draws and determinism (an extra RED test,
random_points_full_range_i64_does_not_panic, pins it). Modulo bias for spans not
dividing 2^64 remains the spec's documented, accepted simplification (param search
needs no cryptographic uniformity).
Includes a public-API E2E suite (tests/random_sweep_e2e.rs) exercising the feature
exactly as a downstream researcher would — reproducibility at the report boundary,
seed-sensitivity, in-range draws, the typed NonNumericRange gate, a well-formed
empty (count==0) family, and Grid/Random interchangeability through `Space`.
Verified: cargo test --workspace green (incl. 17 new engine tests + 6 E2E);
cargo clippy --workspace --all-targets -- -D warnings clean.
closes#52
Five bite-sized tasks for the RandomSpace cut: (1) the Space trait +
behaviour-preserving generalization of sweep/sweep_with_threads, (2) typed
ParamRange, (3) RandomSpace::new validation + three SweepError variants,
(4) SplitMix64 -> pub(crate) + the seeded sampler, (5) public exports +
random-sweep integration. Each task is RED-first with single-substring test
filters; the grid suite is the C1 behaviour-preservation regression guard.
refs #52
RandomSpace + a Space trait + typed ParamRange: the random half of the
C12.1 param-sweep axis (grid shipped in 0028). N seeded uniform draws over
declared continuous ranges, fed to the same enumeration-agnostic sweep core.
Param-only sweep signature and a code-path-disjoint SplitMix64 instance keep
the World-II source-seam firewall (refs #71) intact.
Auto-signed under /boss spec auto-sign: fresh grounding-check PASS and a
unanimous five-lens spec-skeptic panel (criterion, grounding, scope-fork,
ambiguity, plan-readiness all SOUND).
refs #52
Sma::eval re-summed the whole window every tick — O(length)/tick, multiplied
across millions of bars and every sweep point. Replace with the industry-standard
incremental running sum (ta-lib shape): the node owns the window ring and keeps a
running sum, adding the new sample and subtracting the evicted one each cycle, so
the input column drops to depth 1 (lookbacks() = [1], like Ema). The running sum
carries Kahan/Neumaier compensation (the fix pandas' rolling mean adopted) so it
does not drift over long runs; Ema needs none — its recurrence is contractive.
Determinism (C1) holds: same input -> same run, reproducibly. The float output
differs from the full re-sum in the last ULPs (a new, deterministic baseline); the
suite needed only one arity-assumption update (Sma lookback 3 -> 1), no equity
golden changed.
closes#39
load_m1_window collected every bar into an intermediate Vec<M1Parsed> (a full
AoS materialization, un-presized so it grew by reallocation) before transpose_m1
copied it again into the SoA columns — ~2x peak load memory plus realloc churn.
Transpose each chunk straight into the columns instead, via a new shared
M1Columns::extend_from_bars (reserves per chunk for amortized growth). One fewer
full copy, halved peak. Behaviour-preserving: the resulting M1Columns is
byte-identical and transpose_m1's / load_m1_window's signatures are unchanged.
Pinned by a new parity test that needs no live DataServer.
closes#78
cycle 0048 tidy. Architect drift review (86746e3..9febdb9, against the design
ledger C7/C1/C12/C20/C23) found the walk-forward param-plane cut drift-clean:
- C7 made honest: WindowRun.chosen_params is now genuinely tag-free Vec<Cell>;
the param kind lives once on WalkForwardResult.space. No residual per-value
tag. The deleted scalar_as_f64's per-value unreachable! became a per-slot
schema cost (the coerce builder map), correctly invoking C20 only on the
timestamp arm.
- C1 behaviour-preserving: the i64->f64 / f64 coercions are value-identical to
the deleted helper; the migrated fixtures keep mean 2.5 / p50 2.5 / mean 1.5.
The Bool->0/1 arm is the only new behaviour, pinned by
param_stability_reduces_bool_slot_to_fraction_true (0.75).
- SweepFamily symmetry is real: the (space + tag-free Vec<Cell> points +
zip_params) idiom is mirrored while the two stay distinct types. The arity
debug_assert reads space.len() against windows (not the moved-out runs) —
correct. walkforward_family derives space from the same blueprint sweep_over
uses, so the kinds genuinely match.
- No stale references: the only "Vec<Scalar>" / "self-describing record"
matches are in docs/plans/0048 (a historical plan quoting the BEFORE state by
design) and ledger lines describing Scalar itself (still true). No live code
or current-state doc carries the old Fork-A chosen_params rationale.
Regression check: no regression script configured (project facts list only
build/test/lint/doc) — no-op; architect is the gate. The four gates
(build/test/clippy -D warnings/doc) and the acceptance greps were verified
green before the close commit.
Follow-on: the out-of-scope WalkForwardResult::named_params (closing the last
asymmetry vs SweepFamily::named_params) was filed as a droppable idea (#76).
Not actionable drift — no consumer is blocked (zip_params already serves it).
Make the walk-forward param plane C7-clean and symmetric with the sweep param plane. WalkForwardResult gains `space: Vec<ParamSpec>` (the kinds, once, mirroring SweepFamily.space) and WindowRun.chosen_params changes Vec<Scalar> -> Vec<Cell> (the tag-free coordinate the inner sweep already produces). param_stability now resolves its numeric coercion once per slot against the schema (result.space) into a Vec<fn(Cell)->f64>, then runs a type-blind reduction loop — replacing the old per-window-value scalar_as_f64 with its per-value unreachable!. scalar_as_f64 is deleted.
This overturns cycle 0047's "Fork A" (chosen_params stays Vec<Scalar>), which was justified by Scalar's new serializability. That defense was moot: WalkForwardResult and WindowRun carry no serde derives — they are never serialized; only the embedded oos_report (a RunReport, typed via the 0047 lift) is. So chosen_params is a pure in-memory substrate for param_stability, and Vec<Scalar> made it C7-impure (carrying the kind N×M times at the value, where the kind is per-slot constant). The substantive case — kind at the schema, SweepFamily symmetry, per-value unreachable! eliminated — wins, the same argument-form that retired the {kind,cell} Scalar struct in 0047.
The seam ([A]): walk_forward / walk_forward_with_threads take `space: Vec<ParamSpec>` by value, a sibling of the run-window closure; the Fn(WindowBounds)->WindowRun+Sync bound is unchanged and run_indexed is untouched (space is never captured by the Sync closure, moved onto the result at the end). A debug_assert in walk_forward_with_threads guards the arity coupling (every window's chosen_params.len() == space.len(), guaranteed by the same blueprint). walkforward_family computes the space once before the roll and passes the tag-free winner (chosen_params: best.params) directly, dropping the from_cell+zip reconstruction.
[D]: a Bool param slot coerces to 0/1 (mean = fraction "on"), keeping the coercer total over the knob kinds; a Timestamp slot is structurally impossible (C20) so the schema pass carries one unreachable! (once per slot, never per value). param_stability keeps an `if result.windows.is_empty() { return Vec::new(); }` guard to preserve its documented empty-on-no-windows contract. WalkForwardResult and SweepFamily share the (space + tag-free Cell points + zip_params) idiom but stay distinct types — different axis (Sweep enumerates input coordinates; WF rolls time and chosen_params is the optimizer's output), and WF additionally carries bounds + oos_equity + stitched_oos_equity.
Behaviour-preserving (C1): the three migrated walk-forward tests stay green with identical MetricStats (param_stability_reduces_chosen_params_per_slot still pins mean 2.5 / p50 2.5 / mean 1.5); the i64->f64 and f64 coercions are value-identical to the deleted scalar_as_f64. The one genuinely new behaviour (Bool 0/1) is unexercised by built-in grids and pinned by the new param_stability_reduces_bool_slot_to_fraction_true (mean 0.75, 3-of-4 true).
Gates: build --workspace --all-targets, test --workspace, clippy --workspace --all-targets -D warnings, doc --workspace --no-deps — all clean.
closes#75
Two-task execution plan for spec 0048: Task 1 lands the whole aura-engine/walkforward.rs cut atomically (struct shapes, walk_forward seam, param_stability rewrite, scalar_as_f64 deletion, fixture migration, new Bool-slot test) gated on a crate-only `cargo test -p aura-engine`; Task 2 threads the aura-cli caller and runs the workspace gates. Keeps a windows.is_empty() guard in param_stability to preserve the documented empty-on-no-windows contract (C1).
refs #75
Settled-design spec for the walk-forward param-plane cut: WalkForwardResult gains space: Vec<ParamSpec>, WindowRun.chosen_params Vec<Scalar> -> Vec<Cell>, walk_forward takes the space by value (Fn-closure generic untouched), and param_stability schema-checks a per-slot coercer (Bool -> 0/1, Timestamp unreachable per C20) before a type-blind reduction, deleting scalar_as_f64 + its per-value unreachable.
Behaviour-preserving (C1): param_stability yields identical MetricStats over the existing fixtures. Overturns 0047's Fork A (chosen_params stays Vec<Scalar>) — the WF structs carry no serde derives, so the serializability defense is moot; the substantive case is C7 purity + SweepFamily symmetry. grounding-check PASS.
refs #75
Scalar was `struct { kind: ScalarKind, cell: Cell }` — "a Cell wearing a kind hat." The recorded reason for that shape was migration ease, which is not a design rationale (CLAUDE.md: rationale != effort), so the struct had no substantive defense. Redefine it as the native tagged union it conceptually is:
enum Scalar { I64(i64), F64(f64), Bool(bool), Timestamp(Timestamp) }
This is substantively better on four axes: kind/bits skew becomes unrepresentable (illegal states gone); accessors panic loudly on the wrong variant instead of a release-mode silent bit reinterpret; PartialEq derives the documented IEEE-754 / cross-kind value semantics (the hand-roll, needed only because the struct inherited Cell's bitwise compare, is gone); and serde is a plain derive emitting the externally-tagged wire form ({"I64":10}/{"F64":2.5}) — the private ScalarRepr shadow enum that motivated this was never needed. It is also C7-honest: erased-on-the-hot-path (Cell) and self-describing-at-the-edge (Scalar) are two disjoint types bridged by explicit conversion.
The whole public API is preserved (Scalar's fields were private), so call sites do not churn: the enum change is contained to scalar.rs, where cell() now encodes and from_cell() decodes per kind. Cell and ScalarKind are untouched.
With Scalar serializable, lift RunManifest.params from Vec<(String, f64)> to Vec<(String, Scalar)>: the param's kind (an i64 length vs an f64 scale) now survives into the C18 record (runs.jsonl) and the CLI JSON instead of collapsing to f64. scalar_as_param_f64 is deleted; sim_optimal_manifest passes typed params through. This is a deliberate wire-shape change — params now render as tagged scalars; the JSON-asserting tests are updated to the new shape on purpose.
Hand-authored manifest params across the CLI's single-run/mc/macd sites use honest kinds (lengths -> i64, scales -> f64) so they match the sweep path (which already derives correct kinds via zip_params); their in-binary JSON assertions are re-tagged accordingly.
Walk-forward fork resolves with no code change: WindowRun.chosen_params stays Vec<Scalar> (the serializable record carrier), reduced to f64 only at the param_stability statistic boundary (scalar_as_f64 retained). Glossary 'cell' entry updated to describe Scalar as the disjoint tagged union, not 'a cell plus its kind tag'.
Gates: build --all-targets, test --workspace (incl. new scalar_serde_round_trips), clippy -D warnings, doc --no-deps — all clean.
Cell becomes the carrier of the construction path; Scalar narrows to the author/render boundaries. The validated/enumerated param point carries no redundant kind (it lives once, in the declared param-space); at the author edge the kind is a checksum — two independent sources, the typed value vs. the slot — so the self-describing Scalar stays there. The name->slot binding is dynamic (C23), so the check is necessarily a runtime one and the value must self-describe for it.
Base/frontend split (the AnyColumn push/push_cell pattern one level up): compile_with_cells / bootstrap_with_cells are the kind-check-free base; compile_with_params / bootstrap_with_params are the frontend that adds only the per-value kind checksum, strips to cells (new Scalar::cell(), the partner of Scalar::from_cell), and delegates. lower_items loses its per-primitive kind-check; PrimitiveBuilder::build and the std node builders (sma/ema/exposure/lincomb) read cells.
Boundary: construction -> Cell (PrimitiveBuilder::build, lower_items, GridSpace.axes/points, SweepPoint.params, the sweep closure). Author edges stay Scalar (GridSpace::new, bind, compile_with_params/bootstrap_with_params). walkforward chosen_params stays a self-describing Scalar report record (Option A — WalkForwardResult carries no space); the cell winner is reconstructed once at the WindowRun site via from_cell.
injective-check moved ahead of the arity-check in the frontend to preserve the pre-split error order (DuplicateParamPath before ParamArity). The lossy i64->f64 param projection (scalar_as_f64 / scalar_as_param_f64) is deliberately untouched — a separate follow-up.
Behaviour-preserving (C1): build --all-targets / test / clippy -D warnings / doc all clean across the workspace.
cycle 0047 tidy. Architect drift review (b188773..82635fa, against the
design ledger C7/C8/C1) found the carrier swap drift-clean:
- Node::eval -> Option<&[Cell]>, all 8 aura-std out-buffers [Cell; N], the
inter-node forward via the branch-free push_cell (harness.rs:426); Scalar
survives only on the three keep-set boundaries (param plane,
AnyColumn::get, source ingestion at harness.rs:402). The convert/keep
partition and the three-role test rule are honoured site-for-site.
- The dual API is coherent: fallible push (ingestion + kind-guard tests)
vs infallible push_cell (bootstrap-verified inter-node forward) are each
load-bearing and tested. No dead code, no C7 violation (no tag on the hot
path; scratch: Vec<Cell> reused via clear(), no per-cycle alloc).
- Both C7 realization notes + the C8 prose accurately describe the landed
state. 285 tests green; build/clippy/test/doc clean.
Tidy fix (pre-existing, surfaced by the review):
- docs/glossary.md `node` entry said a node implements `schema()`, but
Node::schema() was removed in cycle 0024 (the signature is declared
pre-build on PrimitiveBuilder). Corrected to `lookbacks()` + `eval(ctx)`,
the methods the Node trait actually carries. Record-reality.
Motivation
----------
The C7 realization note (cd3d1ca / 049f22a) deferred this step: with
Scalar split into { kind, cell }, the tag-free Cell could become the
hot-path carrier, but eval and the edges still flowed the fatter
self-describing Scalar. This lands the swap.
Change
------
- Node::eval now returns Option<&[Cell]>; every node out-buffer is
[Cell; N]. The inter-node forward copies the row into a Vec<Cell> and
pushes each field via a new branch-free, infallible AnyColumn::push_cell
— the edge from_field->slot kind match is verified once at bootstrap
(C7), so the runtime push needs no per-value kind check. It only removes
a Result the forward path already .expect()ed.
- Scalar stays on the self-describing dynamic boundaries: the param plane
(build / bind / compile_with_params / sweep points / RunManifest),
AnyColumn::get (the type-erased read for sinks / serde), and source
ingestion (Source::next, the heterogeneous C3 merge). The fallible
AnyColumn::push is retained for ingestion + the kind-guard tests.
Trade
-----
The per-value runtime kind check on node OUTPUT is removed: a node that
builds a wrong-kind cell pushes raw bits with no panic. This is the same
authoring-bug class C8 already leaves to a debug_assert (output width);
node-output-kind correctness is the node's declared FieldSpec contract,
caught by each node's own eval test. Cross-node kind agreement still rests
on the bootstrap kind-check (bootstrap_rejects_*_kind_mismatch, green).
Tests
-----
Behaviour-preserving (C1): 285 green, 0 red. The only test-expectation
edits are carrier re-spellings (Scalar::f64(x) -> Cell::from_f64(x)) of
identical values; test column writes and out-of-graph channel rows stay
Scalar (the spec's three-role rule). Adds an AnyColumn::push_cell
round-trip test plus a cross-kind end-to-end forward test (a mixed
f64/i64 record whose bit patterns diverge across kinds — pins the i64
forward path the f64-only tests cannot catch). Ledger C7/C8 notes updated.
Gates: cargo build / test / clippy --all-targets -D warnings / doc
--workspace all clean.
closes#74
Four tasks for the behaviour-preserving carrier swap: (1) add the
branch-free AnyColumn::push_cell + round-trip test (additive); (2) the
atomic lib-code flip (Node::eval -> Option<&[Cell]>, 8 aura-std nodes,
recorder, harness forward loop) gated by a lib-only build; (3) the
cfg(test) migration (fixtures + eval-output assertions per the spec's
three-role rule) gated by the full four-gate run; (4) the C7/C8 ledger
realization note.
refs #74
Auto-signed under /boss spec auto-sign: all objective gates green
(Step-1.5 precondition, Step-4 self-review, Step-5 grounding-check PASS)
and a unanimous five-lens spec-skeptic panel SOUND, after one editorial
round on the ambiguity lens (the test-migration rule was sharpened to a
three-role classification: eval-output -> Cell, column-write -> Scalar,
channel-send -> Scalar).
Spec for the deferred C7/C8 carrier swap flagged by the C7 realization
note: Cell becomes the hot-path value carrier (Node::eval ->
Option<&[Cell]>, node out-buffers, inter-node forward via a new
AnyColumn::push_cell), and Scalar narrows to the self-describing dynamic
boundaries (the param path, AnyColumn::get, source ingestion).
Behaviour-preserving (C1): only the carrier type narrows.
refs #74
Record the Scalar -> {kind, cell} representation change under C7 (cd3d1ca):
Cell as the tag-free single-value carrier, with the hot-path carrier swap
(eval -> Option<&[Scalar]> and the edges) explicitly flagged as the deferred
next step. Doc-only; no code change.
Motivation
----------
`Scalar` was a tagged enum (I64/F64/Bool/Ts), so every scalar value
physically carried its own kind tag. But the kind is already known from
the schema/port/column the value flows through (C7: the type is a
property of the column, not of the value — the hot path is already
columnar `Column<T>`, and `AnyColumn::get` *reconstructs* the tag from
the column on the way out). The per-value tag was therefore redundant
with the kind the surrounding context already holds.
That redundancy had three costs:
* It baked an implicit `match` (a branch) into every function that read
a Scalar payload — even where the caller statically knew the type.
The tag could never be exploited away.
* Size: a tagged enum is tag + payload = 16 bytes (f64/i64 alignment),
twice the 8 bytes the value needs. A `Column<Scalar>` would be double
the memory and half the cache utilisation.
* It is the shared root of several downstream papercuts we keep hitting
— the lossy f64 manifest field, the `unreachable!` panic on a
non-numeric param, the serde-tag question — all symptoms of "the type
is baked into the value".
Change
------
Introduce `Cell`: a type-erased 64-bit word (`struct Cell(u64)`) that is
not readable without external type context. It is constructed per base
type (`from_i64/from_f64/from_bool/from_ts`) and read only by naming the
type at the call site (`i64()/f64()/bool()/ts()`) — each a branch-free
bit-cast. The hot path resolves the kind once at the boundary (from the
schema) and then reads natively, with no per-value branch. `Cell` knows
nothing of `Scalar` or `ScalarKind`; the dependency is strictly one-way,
and it lives in its own `cell.rs` (more is planned on top of it).
`Scalar` becomes `struct { kind: ScalarKind, cell: Cell }` — the
self-describing form for the dynamic boundaries (builder binding,
serialization, rendering), built on top of `Cell`. Its `as_*` accessors
now `debug_assert` the kind and return the native value (free in
release); calling the wrong accessor is a caller bug, not a checked
`Option`. The variant constructors `Scalar::I64(..)` become associated
fns `Scalar::i64(..)`.
`PartialEq` is hand-written (not derived) to preserve the former enum's
value semantics: kinds must match, then native payloads compare, so f64
keeps IEEE-754 behaviour (`NaN != NaN`, `+0.0 == -0.0`) and a kind
mismatch is never equal even when the raw words coincide. A fixture
(`scalar_eq_is_value_not_bitwise`) pins exactly the cases where bit- and
value-equality diverge, so it can't silently regress. `Cell`'s own
`Eq`/`Hash` stay bitwise — correct for a raw word.
The change is behaviour-preserving: Scalar's observable behaviour is
identical to the pre-Cell enum (the value-equality fixture proves it);
only the internal representation changed. The ~440 call sites across the
workspace are a mechanical constructor rename plus ~12 destructuring
sites (match-arms / `let`-patterns) rewritten to `kind()` + `as_*`.
Verified: cargo build --workspace --all-targets, cargo clippy --workspace
--all-targets -- -D warnings, cargo test --workspace — all green.
cycle 0046 tidy (clean). Architect drift review against the design ledger
(C8/C12/C23 + the #71 firewall) found no drift:
- The named view is a derived, non-load-bearing projection (C23 — slot is
identity); no name reaches the flat graph, no new per-point state, no new
identity mechanism. SweepFamily.space is param_space() carried once, not minted.
- Enumeration and determinism (C1) are untouched (GridSpace::points / run_indexed
unchanged; the new SweepFamily.space is equal across thread counts).
- The run-closure signature Fn(&[Scalar]) -> RunReport is byte-unchanged, so
#52's RandomSpace plugs into the same sweep() layer with zero reconciliation
(#71 firewall held).
- All three CLI hand-zips collapsed to one zip_params call each; no un-collapsed
duplicate remains. zip_params sits in aura-core (a pure projection over
aura-core types) — correct home.
- The f64 manifest field stays Vec<(String,f64)> (the typed-param-space precursor
honestly left deferred, not entrenched).
Regression gate green: cargo test --workspace (all crates, 0 failed; 4 new tests
this cycle), cargo clippy --workspace --all-targets -D warnings clean.
Forward-consistency item (NOT drift, out of #57 scope): McDraw/McFamily and
WindowRun still carry bare Vec<Scalar> with no named view — a future item if a
consumer demands it.
closes#57
Thread a derived named view of a sweep point so consumers stop hand-zipping
param_space() names onto the bare &[Scalar]. The free function
zip_params(space, point) -> Vec<(String, Scalar)> in aura-core is the one
shared projection; GridSpace retains the ParamSpec list it already receives in
new() (it was validated then discarded); SweepFamily carries it (stamped once
in sweep_with_threads) and exposes named_params(i). The three CLI hand-zip
sites collapse to one zip_params call each.
The run-closure signature `Fn(&[Scalar]) -> RunReport` is byte-unchanged — the
named view is a derived projection (C23: slot is identity, name is derived),
not a closure-currency change — so #52's RandomSpace plugs into the same
sweep() execution layer with zero signature reconciliation (#71 firewall held).
sim_optimal_manifest now takes typed Vec<(String, Scalar)> and does the lossy
f64 collapse internally (one place; the manifest's f64-precursor field owns its
own lossiness — the typed-manifest upgrade stays the deferred typed-param-space
item, a non-goal here). All eight call sites migrated: three sweep closures
pass zip_params, five hand-listing callers (run_sample, run_sample_real,
mc_family, run_macd, run_sample_seeded) pass Scalar::F64 literals.
Behaviour-preserving: the collapse output is identical to the old per-site zip,
so aura sweep / walkforward / run output is byte-identical; SweepFamily.space
threaded into the aura-registry optimize test literal.
Verified by the orchestrator: cargo test --workspace green (4 new tests:
zip_params x2, sweep_family_carries_param_space, family_named_params_round_trips),
cargo clippy --workspace --all-targets -D warnings clean. (rust-analyzer
emitted stale mid-edit diagnostics; the real cargo build/test is clean.)
refs #57
Three tasks for spec 0046. T1: zip_params (aura-core), RED-first. T2:
GridSpace retains its ParamSpec list + SweepFamily carries it + named_params
(aura-engine), RED-first, with the aura-registry optimize-test SweepFamily
literal threaded in the same task. T3: sim_optimal_manifest typed-Scalar
migration + all eight call sites (aura-cli), behaviour-preserving. Each
signature/field change threads every one of its sites within its own task,
before that task's workspace compile gate.
refs #57
Thread a derived named view of a sweep point so consumers stop re-zipping
param_space() names onto the bare &[Scalar]. A free function zip_params over
(ParamSpec names ⊗ positional point) in aura-core; GridSpace retains the
ParamSpec list it already receives in new(); SweepFamily carries it and
exposes named_params(i). The run-closure signature
`F: Fn(&[Scalar]) -> RunReport + Sync` stays byte-for-byte (honours #52's
#71-firewall constraint, so RandomSpace plugs in with zero reconciliation).
The lossy f64 collapse moves into the manifest constructor; the view stays
typed. Behaviour-preserving; SweepPoint and enumeration untouched.
Design ratified in-context (brainstorm, approach C over A, free function
over a NamedPoint type); a reconciliation comment on #57 records the
resolved forks with provenance. Human sign-off after the auto-sign panel's
grounding lens caught a blast-radius undercount in the first draft —
corrected to all eight sim_optimal_manifest call sites plus the
aura-registry SweepFamily struct-literal.
refs #57
Behaviour-preserving normalisation of the lineage record. The fused
`family_id: String` ("{name}-{counter}") is split into its two factors —
`family: String` + `run: usize` — and the user-facing handle becomes a derived
`family_id()` method ("{family}-{run}"), never stored or parsed.
Why: the fused field forced the counter logic to generate-and-check candidate
strings (to stay robust to '-' inside a name) rather than read it; with `run` a
plain int, `next_run` is a clean numeric max+1 over the field. The split is more
normalised (one fact per field) and the only property the fused token bought — a
single paste-able CLI handle — is recovered by deriving it. Storage and all
user-visible output stay byte-identical: append_family still returns
"{name}-{run}", Family.id is still "{name}-{run}", and the CLI prints the same
"family_id" lines (the json! key + the returned String, both unchanged).
Internal-only: FamilyRunRecord and group_families (now keyed on the (family,
run) tuple, first-seen order preserved) change; lib.rs re-exports, the
extractors, and all of aura-cli are untouched (everything downstream goes
through Family.id + append_family's returned String). The on-disk
families.jsonl key changes ({"family","run"} vs {"family_id"}) — a gitignored
runtime artifact with no committed fixture, and tests round-trip within a temp
dir, so no migration is owed. Doc reconciliation: the lineage/lib module headers
and the C18 ledger realization note now describe the split shape.
Verification: a workflow ran the refactor in an implementer agent, then three
adversarial lenses (behaviour-preservation, new-logic correctness,
completeness/doc-lag) tried to refute it — the first two found nothing, the
third flagged the four doc-lag sites now fixed here. Self-run gates:
cargo test --workspace green (registry 11, incl. a strengthened round-trip test
pinning the split fields + derived handle), cargo clippy --workspace
--all-targets -D warnings clean, cargo doc clean.
Architect drift review (eeba218..HEAD) + regression gate (the project's
cargo test --workspace is the gate; no separate regression script).
What holds (architect): C9 layering intact — aura-engine gains no registry
dependency, lineage lives entirely in aura-registry. #71 firewall held —
RunManifest/aura-core byte-untouched, no blob/path/payload field on
FamilyRunRecord, the member window is producer-supplied via
Source::bounds()/window_of, never a Vec scan. C2 clean — bounds() is
cursor-independent (no look-ahead). Family store is a disjoint sibling of the
flat store (pinned by a disjointness test). Regression gate green: 228 tests,
clippy --workspace --all-targets -D warnings clean.
Drift resolved:
- [fix, this commit] C18 ledger gap: docs/design/INDEX.md C18 carried no
realization note for the run registry. Added the cycle-0029 (flat registry)
and cycle-0045 (lineage as related records / family store) realization
notes, in the ledger's established per-cycle style.
- [forward-queue] Flat-store CLI dead-end: after this cycle no CLI command
writes the flat runs.jsonl (sweep/walkforward moved to the family store,
aura run never persisted), so aura runs list / rank read an unfed store.
This is a spec-accepted state (0045 kept runs list "for standalone runs
only") and a genuine design fork (give aura run a persisting path vs. retire
the flat-store CLI surface) with no clear default — filed as #73 rather than
decided here. Recorded as a deferred follow-up in the C18 ledger note.
- [carry-on] Low/cosmetic: family-wrapped CLI stdout alphabetizes nested
manifest keys (serde_json::json!) vs. declaration order (to_json()); both
valid JSON, the stored families.jsonl uses declaration order, round-trip
unaffected. Noted as a sub-item on #73.
refs #70
Iteration 2 of spec 0045 — the CLI surface for orchestration families.
window_of migration: every manifest-window scan (run_sample, sweep_family,
walkforward_family, sweep_over, run_oos, run_macd) now reads the window from
Source::bounds() via window_of(&sources) instead of prices.first()/.last().
Behaviour-preserving (the walk-forward roller boundaries are bar-aligned, so
the producer-supplied window equals the old first/last) — pinned by the
unchanged run_sample (1,7) and walk-forward determinism tests.
aura mc: new built-in Monte-Carlo family on the CLI (monte_carlo over a
built-in seed set + the sample harness re-seeded per draw), persisted via
append_family and rendered as one member line per seed (carrying the
family_id) plus an mc_aggregate line. McAggregate is not Serialize, so the
aggregate line is built from its three MetricStats blocks.
family-aware runs: aura sweep / walkforward / mc now persist to the family
store via append_family with an optional --name (per-kind default), printing
the assigned family_id per member. aura runs families lists family headers
(id, kind, member count); aura runs family <id> [rank <metric>] lists/ranks
one family's members as a unit. An unknown family id is an empty family
(exit 0); an unknown metric is a usage error (exit 2).
Two pre-existing cli_run.rs E2E tests were adapted to the new family
contract (a required consequence of sweep/walkforward switching from the
flat store to the family store — a plan gap caught at implement time): the
sweep-output test now asserts the family-wrapper, and the runs list/rank
flow became the family list / per-family-rank flow (which also exercises the
per-name counter sweep-0/sweep-1 end-to-end). A new E2E test drives aura mc
through the binary.
Known cosmetic detail: the family-wrapped sweep/mc/walkforward stdout nests
the report via serde_json::json! (to_value -> alphabetical keys, leading
"broker"), while runs family / runs list print via RunReport::to_json()
(declaration order, leading "commit"). Both valid JSON; key order is not
load-bearing and the stored families.jsonl uses to_string (declaration
order), so the round-trip is unaffected.
Verification: cargo test --workspace green; cargo clippy --workspace
--all-targets -D warnings clean.
refs #70
Iteration 1 of spec 0045 — the engine + registry core for storing
orchestration families as linked records.
aura-engine: add Source::bounds() — the inclusive (from, to) data extent a
producer will stream, known without materialization (#71 firewall) — and a
free window_of() that folds the union extent across a run's sources. This is
the producer-supplied replacement for the prices.first()/.last() Vec scans
at the CLI call sites (those migrate in iteration 2); a lazy producer now
reports its window so stored lineage stays byte-identical whether a source
is eager or streamed.
aura-ingest: M1FieldSource stores its requested [from_ms, to_ms] window at
open and reports it via bounds() (normalized to epoch-ns; None when
open-ended — the archive-extent query is a deferred non-goal).
aura-registry: new lineage module — FamilyKind / FamilyRunRecord (a
RunReport stamped with family_id + kind + ordinal) / Family, plus
Registry::append_family (assigns family_id = "{name}-{counter}" via a
per-name generate-and-check counter, writes members to a sibling
families.jsonl) and load_family_members; group_families re-derives the
families (the round-trip), and the three *_member_reports extractors pull
the per-kind member reports. The flat runs.jsonl store and its
append/load/rank_by/optimize API are byte-for-byte unchanged (a test pins
the two stores' disjointness). C9 preserved: aura-engine gains no registry
dependency. New dep: serde derive on aura-registry (per-case policy, same
basis as serde_json already is).
Verification: cargo build --workspace; cargo test --workspace (228 green,
incl. 3 new engine bounds/window_of, 1 data-gated ingest bounds run against
real archive data, 5 new registry lineage round-trip/counter/disjointness
tests); cargo clippy --workspace --all-targets -D warnings clean. The CLI
surface (aura mc, family-aware aura runs, the window_of migration) is
iteration 2.
refs #70
Decompose spec 0045 into two iterations. Iteration 1 (engine + registry
core): Source::bounds() + window_of, M1FieldSource::bounds(), and the
aura-registry lineage module (FamilyKind / FamilyRunRecord / Family,
group_families, next_family_id, the three *_member_reports extractors,
Registry::append_family / load_family_members) with round-trip and
per-name-counter tests. Iteration 2 (CLI surface): aura mc, family-aware
aura runs (families / family [rank]), sweep/walkforward family persistence
via append_family with --name, and the window_of migration of the
manifest-window scans.
refs #70
Persist sweep / Monte-Carlo / walk-forward runs as named, linked families
(C18 lineage / C21): a FamilyRunRecord stamps each member with a shared
family_id = "{name}-{counter}" in a sibling families.jsonl store, group_families
re-derives a family as a unit, and the CLI gains aura mc + family-aware aura runs.
Producer-supplied window via Source::bounds()/window_of (the #71 firewall: no Vec
scan, no input artifact). family_id form (name + counter, not a content-hash)
resolved by the user in-session; reconciliation comment on the issue records it.
Auto-signed under the boss spec-auto-sign gate: all objective gates green
(precondition, self-review, grounding-check PASS) and a unanimous five-lens
spec-skeptic panel (criterion, grounding, scope-fork, ambiguity, plan-readiness).
refs #70
Architect drift review over 61a8436..HEAD (the 0044 walk-forward cycle: spec
a4712eb, plan 7d52eff, feat 4764656). Cycle is sound, no drift:
- C1 — walk_forward reuses the shared sweep::run_indexed core; output is roll
order (index-sorted), not completion order; walk_forward_is_deterministic_across_thread_counts
pins 1-vs-8-thread equality. Parallelism is across windows, never within.
- C2 — no look-ahead is structural: WindowRoller::next emits is.1 = oos_start-1,
so oos.0 == is.1+1 by construction; roller_every_split_has_no_lookahead asserts
it with zero ticks materialized (a pure bounds invariant).
- C9 — aura-engine has no aura-registry dependency (grep clean; Cargo.toml lists
only aura-core/aura-std); the in-sample optimize is closure-supplied, the CLI
bridges engine+registry. The orchestration primitive bakes in no search policy.
- C12 axis 3 realized as a sibling of sweep/mc; the atomic unit is unchanged.
- #71 firewall — the walk_forward API carries WindowBounds + a per-window closure,
never a materialized stream Vec; per-window data is a bounds-keyed closure concern.
- R2 honored — WalkForwardResult stores only windows + stitched_oos_equity;
param_stability is a pure on-demand reduction (sweep-precedent, non-redundant).
The #69 reconciliation-comment provenance (R2, 2026-06-15) matches the spec and
the feat commit body. MetricStats::from_values extraction is behaviour-preserving
(the 7 mc tests stay green). The lib.rs module header was updated in the feat
commit to list walk-forward as delivered, so no doc-lag remains for the audit.
One non-load-bearing observation (not fixed — demo-only, crosses no contract):
the CLI walkforward_window_source regenerates the full synthetic seed-stream per
window-source construction. Confined to the built-in demo; bounds-keyed, no #71 line.
Regression gate (test suite — no dedicated regression scripts in project facts):
cargo test --workspace green (aura-engine 152 incl. walkforward 9 + mc 9, aura-cli
14, all crates 0 red); clippy --workspace --all-targets -D warnings exit 0; cargo
doc --workspace --no-deps clean.
closes#69
The third C12 orchestration axis: walk-forward varies the data window. A
WindowRoller is a pure iterator of WindowBounds { is, oos } — bounds only, never
tick data (#71 firewall). walk_forward runs a caller closure per split disjointly
over the shared run_indexed core (C1), and stitches the OOS pip-equity segments
into one continuous curve (each segment offset by the running sum of prior
segments' finals; an empty segment contributes 0.0, mirroring summarize). C2
no-look-ahead is structural: the roller only ever emits oos.0 == is.1 + 1, pinned
by a zero-tick bounds test. The in-sample optimize (axis 2) is closure-supplied,
not called by the engine: aura-engine has no aura-registry dependency (C9), and
C12 forbids baking search policy into the primitive — the CLI bridges both crates.
Param-stability is on-demand, not stored (the R2 decision, recorded with
provenance on #69): WalkForwardResult stores only the raw per-window outcomes +
the stitched curve, and param_stability(&result) -> Vec<MetricStats> is a public
reduction over the retained per-window chosen params. A stored summary would be
recomputable from the windows (redundant) and would force one statistic canonical
when "stability" admits several; this mirrors SweepFamily (raw points + on-demand
optimize), not McFamily (stored aggregate). MetricStats::from_values is extracted
from McAggregate::from_draws (behaviour-preserving — the 7 mc tests stay green)
and gains serde so the CLI summary renders it and #70 lineage can persist it.
`aura walkforward` runs a built-in rolling walk-forward over a synthetic windowed
source: per window it sweeps the built-in grid in-sample, optimizes by total_pips,
runs the chosen params out-of-sample, persists each OOS RunReport (C18), and prints
a stitched summary line.
Two plan deviations, both compiler-forced and consistent with siblings: added
#[derive(Debug)] to WindowRoller (the RED tests' unwrap_err needs Ok: Debug; the
sibling types already derive Debug); removed a now-redundant test-module
SyntheticSpec import after Step 2 hoisted it to the top-level use.
Verification: cargo test --workspace green (aura-engine 152 incl. walkforward 9 +
mc 9, aura-cli 14); clippy --workspace --all-targets -D warnings exit 0; cargo doc
--workspace --no-deps clean.
refs #69
Bite-sized, placeholder-free plan for the walk-forward axis, in three tasks:
(1) extract MetricStats::from_values from McAggregate::from_draws + serde derive
(behaviour-preserving, the 7 mc tests guard); (2) the walkforward module —
WindowRoller (bounds-only iterator), walk_forward over the shared run_indexed
core, continuous OOS stitch (empty segment -> +0.0), on-demand param_stability,
+ crate exports + module-header doc-lag fix; (3) the aura walkforward CLI demo
bridging engine+registry via optimize, with serde_json for the summary line.
RED-first tests 1-11 (incl. 7b empty-segment stitch) per the spec.
refs #69
Walk-forward orchestration: a WindowRoller rolls (in-sample, out-of-sample)
splits over a time span, walk_forward runs a disjoint harness per split via the
shared run_indexed core (C1), and stitches the OOS pip-equity segments into one
continuous curve. The varying dimension is the data window (axis 3), realized
eager-agnostically (#71): the public surface carries WindowBounds + a per-window
closure, never a materialized stream Vec; C2 no-look-ahead is a pure bounds
invariant (oos.0 > is.1), checkable with zero ticks.
The in-sample optimize (axis 2) is closure-supplied, not called by the engine:
aura-engine cannot depend on aura-registry (C9), and C12 forbids baking search
policy into the primitive. The CLI bridges both crates in run_walkforward.
Param-stability shape (the load-bearing fork) resolved with the user as R2:
on-demand, not stored. WalkForwardResult stores only raw per-window outcomes +
the stitched curve; param_stability(&result) -> Vec<MetricStats> is a public
on-demand reduction over the retained per-window chosen params (sweep-precedent:
summaries computed on demand, not cached, vs McFamily's stored aggregate). A
stored summary would be recomputable from the raw windows (redundant) and would
force one statistic canonical when "stability" admits several. Recorded with
provenance in the #69 reconciliation comment. MetricStats::from_values is
extracted from McAggregate::from_draws (behaviour-preserving; the 7 mc tests
guard) so the MC aggregate and the helper share one reduction. Empty OOS segment
contributes 0.0 to the stitch offset (mirrors summarize's unwrap_or(0.0)).
Gates: Step-1.5 precondition clean (no fork silently picked), self-review clean,
grounding-check PASS (twice — re-run after each edit). Auto-sign panel escalated
on the scope-fork design lens (param-stability shape) to human sign-off; the user
resolved it (R2), so this is a user-signed spec, not boss-signed.
refs #69
Architect drift review over b920362..HEAD (the 0043 monte-carlo cycle). Cycle is
sound: C12 axis 4 faithfully realized (monte_carlo varies the seed, the 0042
seeded source genuinely perturbs each run, so distinct_seeds_produce_distinct_draws
is a real property); C1 honoured (one run_indexed core, output sorted by job index
= input order independent of completion, 1-vs-8-thread determinism pinned,
parallelism across draws never within); the run_indexed extraction is a genuine
behaviour-preserving refactor (C11/C23 — the three pre-existing sweep tests stay
green, and both family executors are thin adapters over the single core; the
points.len() -> n.max(1) clamp is equivalent since GridSpace rejects empty axes);
the #71 eager-agnostic firewall is respected (API takes seeds + a per-draw
closure, no materialized stream Vec crosses the boundary).
One medium doc-lag found and fixed here: aura-engine/src/lib.rs module header
listed Monte-Carlo (and, already stale, the grid sweep) under "Still to come";
both orchestration axes ship now, so the header records the grid axis (sweep /
SweepFamily) and the seed axis (monte_carlo / McFamily) as delivered, with
walk-forward, random param-sweep, registry lineage, and the C10 position-event /
broker layer remaining future.
Regression gate (test suite — no dedicated regression scripts in project facts):
cargo test --workspace green (incl. 7 new mc::tests + the sweep guard tests);
clippy --workspace --all-targets -D warnings clean; cargo doc -p aura-engine
clean (the new intra-doc links resolve).
closes#68
`monte_carlo(base_point, seeds, run_one) -> McFamily`, the Monte-Carlo
orchestration family: a fixed base point run over a seed set, each seed a
disjoint C1 realization. C12 axis 4 — Monte-Carlo IS a sweep over seeds, so it
reuses the disjoint-parallel executor rather than forking a run loop.
The reuse is realized by extracting the index-parallel core out of
`sweep_with_threads` into a `pub(crate) run_indexed<T>` (generic over the
per-job result, over a 0..n job range); `sweep_with_threads` becomes a thin
adapter over it. Behaviour-preserving (C1/C11/C23): the three pre-existing sweep
tests stay green — they ARE the proof the refactor preserved behaviour. A small
incidental win: the adapter consumes `points` via zip, dropping the old
per-point clone.
mc.rs adds McDraw{seed,report} / McFamily{draws,aggregate} (draws in seed-INPUT
order, independent of thread completion) / McAggregate (mean + p5/p25/p50/p75/p95
of all three run metrics — the V1 decision: covers every metric, not a "chosen"
one) / MetricStats, a private type-7 linear-interpolation `quantile`, and
`McAggregate::from_draws` (a pure post-run reduction recomputable from the
retained raw draws). Eager-agnostic firewall (#71): the API takes seeds + a
per-draw closure `Fn(u64,&[Scalar])->RunReport`, never a materialized stream Vec;
seed->Source construction stays a closure-body concern. An empty seed set is a
debug-asserted precondition, preserving the no-Result `-> McFamily` signature.
7 RED-first mc::tests pin: one draw per seed in input order; determinism across
1 vs 8 threads (C1); draw == an independent seeded run; distinct seeds perturb
the family; aggregate == from_draws(draws); type-7 quantile/mean math on a known
fixture; quantile endpoints + singleton. Workspace green, clippy -D warnings
clean, cargo doc clean.
refs #68
Three tasks: (1) extract the disjoint-parallel core out of sweep_with_threads
into a shared `pub(crate) run_indexed<T>` (behaviour-preserving; the three
pre-existing sweep tests are the green guard), (2) add the `mc.rs` module
(McDraw/McFamily/McAggregate/MetricStats, type-7 `quantile`,
`McAggregate::from_draws`, `monte_carlo`/`monte_carlo_with_threads` driving
run_indexed) with 7 RED tests, (3) wire `mod mc;` + the five public exports into
lib.rs. RED-first; module-declaration ordering keeps mc.rs's tests dark until the
final wire-up task.
refs #68
C12 axis 4: Monte-Carlo as a sweep over seeds. `monte_carlo(base_point, seeds,
closure) -> McFamily`, the analog of SweepFamily, reusing the disjoint-parallel
executor (extracted into a shared `run_indexed` core) rather than forking a new
run loop. Each realization is a disjoint C1 unit; parallelism is across draws.
The aggregate (V1) covers all three run metrics with mean + p5/p25/p50/p75/p95,
a pure post-run reduction over the retained raw draws.
Eager-agnostic firewall (#71): the API takes seeds + a per-draw closure, never a
materialized stream Vec; seed->Source construction stays a closure-body concern.
Signed under the /boss spec auto-sign gate: objective gates green (precondition,
self-review, grounding-check PASS) and a unanimous five-lens spec-skeptic panel
(criterion, grounding, scope-fork, ambiguity, plan-readiness) returned SOUND.
The aggregate fork was settled to V1 by the user (issue #68 reconciliation
comment, verbatim "ja, wir machen v1", 2026-06-15).
refs #68
Architect drift review over 682e459..HEAD (the seed-as-input cycle plus the
prior unaudited `aura run --real` work). Cycle is sound: Fork B respected (seed
forks at the data-generation edge into the source and the manifest; the engine
loop sees only a Source — C12/C1/C3 intact), the contract names Source never Vec
(precondition for #68/#52 delivered as specced), and the four seed-free callers
thread 0 in lockstep.
One medium doc-lag item found and fixed here: aura-engine/src/lib.rs module-doc
listed the seed axis under "still to come"; seed-as-input shipped this cycle and
RunManifest.seed is now live, so the headline doc now records SyntheticSpec as a
delivered seeded Source producer (params + orchestration families remain future).
One low carry-on note (not drift): run_sample_seeded exists only under
#[cfg(test)] — intended this cycle (no --seed CLI flag per spec non-goals); the
only non-test seed value remains 0. A scaling marker for #68, not debt.
Regression gate (test suite — no dedicated regression scripts in project facts):
cargo test --workspace green; clippy --workspace --all-targets -D warnings clean;
cargo doc -p aura-engine clean (the new [`SyntheticSpec`] intra-doc link resolves).
A seeded source whose stream is fully determined by a u64 seed, making
RunManifest.seed a live captured input instead of the dead 0 it was. This is
C12's reconciliation of stochastic runs with C1: a run is bit-identical for a
fixed seed (the seed is a captured input, not hidden nondeterminism), and a
different seed gives a different-but-reproducible run. Precondition for the
Monte-Carlo family (#68) and random param-sweep (#52).
Engine (aura-engine):
- SplitMix64: a ~10-line private, dependency-free PRNG. Chosen over a rand-family
crate for guaranteed bit-stability across toolchains/crate versions — the whole
value of seed-as-input is reproducibility, and a crate whose algorithm is not
version-stable would silently break a recorded seed on a dep bump. No new direct
dep.
- SyntheticSpec + source(seed) -> impl Source: the Fn(u64) -> impl Source contract
(Fork B — seed at the data-generation edge, outside the engine graph). Returns a
Source, never a Vec, so the MC family can re-seed N times without materializing
N streams. First cut collects to a VecSource internally; the signature does not
name Vec, so a lazy generator is a drop-in replacement. Re-exported from lib.rs
beside VecSource.
CLI (aura-cli):
- sim_optimal_manifest gains a seed: u64 param recorded into the manifest (the
single Fork B capture site). All four existing seed-free callers (run_sample,
run_sample_real, the sweep grid-report, run_macd) pass 0 unchanged — their
determinism tests stay green.
- A test-only run_sample_seeded + SeededTrace exercise a live seed and expose the
drained sink trace, so the three e2e tests assert bit-identity at the trace
level (strictly stronger than the folded 3-field metrics).
Tests (RED-first): 2 engine producer tests (same-seed identical / different-seed
differs) + 3 cli e2e tests (bit-identical trace, different-seed metrics differ,
seed recorded in manifest). Full workspace green; clippy -D warnings clean.
Errata vs plan 0042 (two compile-forced deviations, neither alters the spec
contract):
- source returns `impl Source + use<>`: under edition 2024 a bare `impl Source`
captures the &self lifetime, which blocks boxing as Box<dyn Source + 'static>
for run() (E0597). The source owns its data, so use<> (captures nothing) is
correct.
- SyntheticSpec is imported inside `#[cfg(test)] mod tests` in main.rs: it is
used only by the seeded test vehicle, so a top-level import is unused in the
non-test build under clippy -D warnings.
closes#66
Two tasks: (1) aura-engine SplitMix64 PRNG + pub SyntheticSpec::source(seed) ->
impl Source beside VecSource, with producer RED tests; (2) aura-cli
sim_optimal_manifest gains a seed param (four callers pass 0), plus a test-only
run_sample_seeded + SeededTrace and three e2e RED tests pinning the #66
acceptance. RED-first throughout.
refs #66
A seeded source whose stream is fully determined by a u64 seed, wired so the
seed reaches RunManifest.seed at the manifest-construction site (Fork B: seed at
the data-generation edge, outside the engine graph). Contract is a producer
Fn(u64) -> impl Source, never seed -> Vec, so the Monte-Carlo family can re-seed
N times without materializing N streams. Deterministic dependency-free PRNG
(SplitMix64) for C1 bit-stability. Precondition for #68 (Monte-Carlo) and #52
(random sweep).
Auto-signed under /boss spec auto-sign: objective gates green (precondition,
self-review, grounding-check PASS) and a unanimous five-lens spec-skeptic panel
(criterion, grounding, scope-fork, ambiguity, plan-readiness all SOUND) after one
editorial-repair round.
refs #66
The first real-data backtest from the CLI: `aura run --real <SYMBOL>
[--from <ms>] [--to <ms>]` runs the existing built-in sample signal-quality
harness over real M1 *close* bars instead of synthetic prices, printing the
same RunReport JSON as `aura run`. Bars are streamed lazily through
aura_ingest::M1FieldSource (a Box<dyn Source>) — dogfooding the #71 Source
seam, O(one chunk) resident, never eager. aura-ingest + data-server enter
aura-cli's deps (data-server git line mirrored from aura-ingest).
M1 only this cycle, deliberately. The TickSource is deferred: raw ticks are
the wrong input for the bar-semantic SMA sample (an SMA over ticks is
microstructure noise until a resampler node exists, C2), and tick's real
driver is the realistic broker's bid/ask execution (C10) — it should arrive
with that consumer, not speculatively here. Tick data is also local to only
EURUSD/XAUUSD/GER40, whereas M1 covers the test symbol AAPL.US.
Shape: run_sample_real builds sample_harness(), guards has_symbol /
M1FieldSource::open (unknown symbol or empty window -> 'aura: no local data'
+ exit 2, never a panic), and reuses sim_optimal_manifest + the run_sample
fold. parse_real_args is a pure, unit-tested grammar (mandatory symbol, then
--from/--to in any order; bad/missing/duplicate flag -> Err). Data path is
data_server::DEFAULT_DATA_PATH (no --data-path flag this cycle).
Known simplification: the manifest window is derived by draining a *separate*
single-pass probe source for first/last ts, so an unbounded window streams
the archive twice (once to bound, once to run). Acceptable and O(1)-resident
for now; a future Harness::run could surface the first/last cycle ts to drop
the probe pass.
Verified: cargo build/test/clippy --workspace all green; the gated
run_sample_real_streams_real_close_bars_deterministically RAN (not skipped)
over AAPL.US 2006-08; manual CLI smoke of the success + two error paths.
Architect drift review over 8b330e3..HEAD (the Source-seam cycle #71 plus the
interim #67 optimize work and refactors). No semantic or contract drift: the
seam faithfully preserves C3 (ms→epoch-ns at the one ingestion boundary), C4
(tie-by-source-index, byte-for-byte), and C7 (field-per-source SoA); the full
suite is byte-identical green and the gated streaming tests pass on real data.
Regression gate: `cargo test --workspace` green (the project declares no
dedicated regression script; the suite is the gate).
Tidied stale prose the cycle left behind (no behaviour change):
- docs/design/INDEX.md (C12): replaced the cycle-0011 "deliberate eager gap"
status note with a cycle-0041 realization note — the producer `Source` seam
+ streaming `M1FieldSource` now deliver per-source O(one-chunk) streaming;
precisely scopes what remains open (cross-*sim* Arc<[T]> window sharing,
still unbuilt until the orchestration families #66/#68/#69 consume it).
- aura-core/src/lib.rs: dropped the now-shipped "Source trait + data-server
ingestion" from the module doc's "still to come" list (aura-engine's twin
line was fixed in the cycle; aura-core's was missed).
- aura-ingest/src/lib.rs: module header now documents both coexisting paths
(eager load_m1_window/M1Columns vs lazy streaming M1FieldSource).
- aura-registry/src/lib.rs: rank_by/optimize doc comments linked to the
private `metric_cmp` (a public→private intra-doc-link warning from the #67
commit); demoted to plain code spans, so `cargo doc --workspace` is now
warning-clean.
Brummel