Drop the gitignore entries for both dirs; align the CLAUDE.md note from "git-ignored, local-only" to "git-tracked — committed while the cycle is live, removed (git rm) at cycle close". The durable record stays the ledger + git history; specs/plans are transient in the repo, present only for their own cycle.
docs/specs and docs/plans were retired (prior commit); the source/test
comments that cited them ("spec 0050 §4.1", "spec §Testing N", "per spec",
"the spec's ...") now point at nothing. Strip every such pointer while
preserving the technical substance, the design-ledger contract refs
(C1/C11/C20/C34/C12.1/...), and the Gitea issue refs (#41).
Comment/doc edits only across 20 files — no logic change; full workspace
suite green, clippy clean.
Per-cycle specs/plans are step-scoped: every file maps to already-shipped
code, their snippets drift hard against current APIs (InputSpec->PortSpec,
Sim->Harness, schema() removed, C16 zero-dep reversed, ascii-dag retired),
and a stale spec read as an API reference misleads agents into non-compiling
code. No durable knowledge lost — rationale lives in the ledger + code
rustdoc, history in git + the tracker.
- gitignore docs/specs/ and docs/plans/
- delete all 117 existing specs/plans (recoverable via git history)
- CLAUDE.md: specs/plans are local-only ephemeral artifacts
The residency probe was an inherent method on the concrete M1FieldSource, so a Box<dyn Source> consumer could not read it without downcasting (the adjacent friction noted in #95). Hoist it onto the Source trait as resident_records(&self) -> Option<usize> with a default of None, mirroring bounds()'s Option-for-unknown idiom: None = "does not report" (the eager VecSource, which has no streaming ring), distinct from Some(0) = "reports, holds none" (e.g. exhausted).
M1FieldSource overrides it to return Some(chunk len). Callers updated (streaming_seam, the two deep-dive fieldtest bins). Tests: a hermetic VecSource default-is-None check, and a &dyn Source probe in the gated residency test proving M1FieldSource residency is readable without a downcast.
refs #95
The C12 realization and the 0041 spec claimed streaming residency is O(one chunk) at the process level ("a 20-year window streams in the same memory as a one-day one"). That holds only for the aura source ring (M1FieldSource::resident_records(), bounded by one chunk and correctly tested); whole-process RSS grows O(records-touched) because data-server's FileCache retains each window's parsed chunks (~56 B/record) read-only for the pass. Reproduced: 6 MiB @ 1mo -> 173 MiB @ ~10y, linear.
That retention is the replay-many optimization (load-once; close+volume share one parse via the field-agnostic FileKey), not a leak, and no always-on eviction can bound a single forward pass without regressing it. This narrows the docs/comments to the true per-source-ring property and names the FileCache per-window retention as the real, replay-amortized process cost. The streaming_seam assert is unchanged (it was always the ring bound; only its labelling confused ring vs process). The deferred opt-in non-retaining streaming read path is parked as Brummel/data-server#2.
closes#95
data-server does no timezone math (delphi_to_unix_ms is pure arithmetic), so the
archive carries whatever the raw Pepperstone files contain, labelled as UTC. The
Session node's UTC->Berlin DST conversion is unit-tested, but the assumption that
the raw timestamps are themselves UTC was unverified — a broker-local archive
would shift the whole session and the error would jump across the DST boundary.
Verified empirically via the DAX cash-open volume spike (GER40): it lands at
Berlin 09:00 in BOTH seasons, at 07:00 UTC (summer, CEST, UTC+2) and 08:00 UTC
(winter, CET, UTC+1) — a clean +1h DST shift, the genuine-UTC signature. This
gated test pins it so a future archive/convention drift cannot silently corrupt
every session strategy's bar alignment.
Add a C20 realization recording the resolution of the GER40 deep-dive's #94
finding: a hand-wired FlatGraph is not a shippable strategy (no param_space, so
World-opaque); the canonical form is the Composite blueprint, the FlatGraph its
compiled substrate (C23). Pins the bar-period-as-structural-construction-arg
decision (spec 0051) to the structural-axis-vs-tuning-param split.
refs #94
The shipped Composite blueprint (ger40_breakout_blueprint) is now driven by the
World orchestration families on real GER40, with NO re-authoring — the #94
friction is gone:
- examples/ger40_breakout_sweep.rs — sweep the {entry_bar, exit_bar} grid built
directly from param_space() (best entry=3,exit=4 -> 214.0 over 2024).
- examples/ger40_breakout_walkforward.rs — non-degenerate walk_forward: a real
non-empty space optimized per window, chosen_params populated everywhere (#97).
- examples/ger40_breakout_compare.rs — one blueprint across two structural axes:
instrument (GER40 vs FRA40) and bar period (15m vs 30m — different strategies,
C34), no re-authoring.
- tests/ger40_breakout_world.rs — gated: sweep returns the ranked 9-point grid
over the two named params; walk_forward yields non-empty windows each with a
non-empty chosen_params (executable #97 proof).
No blueprint reshape needed — the factory returns a fresh (Composite, Taps) per
call, so the World families re-instantiate per grid point / window (the SMA-cross
pattern); baked recorders do not block multi-bootstrap.
closes#94, closes#96, closes#97
Author the shipped session-breakout as a reusable Composite blueprint via
GraphBuilder — the World-consumable form (param_space + bootstrap) — alongside
the hand-wired FlatGraph it reproduces. Applies the spec-0051 decisions:
- bar_period_minutes is a construction arg binding BOTH Resample.period_minutes
and Session's period, locked equal, so a sweep cannot desync them (#96);
- delay.lag is bound out (a structural constant, not a tuning knob — C34);
- the two EqConst targets (named entry_bar / exit_bar) are the ONLY params, so
param_space() == {entry_bar.target, exit_bar.target} — the genuine tuning
knobs, which makes walk_forward non-degenerate (#97).
The ger40_breakout_real example now bootstraps from the blueprint (still prints
-45.0 for 2024-09). New gated test ger40_breakout_blueprint.rs proves: (a) the
param_space is exactly the two targets (no lag, no period); (b) C23 — the
blueprint reproduces the FlatGraph bit-identically over real GER40 2024-09;
(c) determinism.
First increment of the milestone; the sweep / walk_forward / compare World-family
demos follow.
refs #94, #96, #97
Milestone spec: ship the GER40 session-breakout as a Composite blueprint so the
same strategy runs across backtest / sweep / walk_forward / compare without
re-authoring (the #94 fieldtest friction). Records the design decision that the
bar period is a structural axis (a construction arg binding both Resample and
Session, so a sweep cannot desync them — #96) and that the canonical shippable
strategy form is a Composite (FlatGraph is its compiled substrate). Authoring-
side; no engine-core change (the existing bind() + GraphBuilder + param_space()
layer already suffices).
refs #94, #96, #97
Public-API research deep-dive driving the just-shipped GER40 real-data
session-breakout demo (5b5f034) through the escalating sequence a serious
researcher attempts next: scale to multi-year, walk-forward, structural-param
sweep, multi-instrument compare. Standalone consumer crate (path-deps only;
public interface = ledger + glossary + example corpus + cargo doc; no
crates/*/src read). All runs against the real /mnt/tickdata/Pepperstone archive.
Findings: 3 bugs, 1 friction, 3 spec_gap, 1 working. Triaged to the tracker:
- spec_gap: the flagship breakout ships as a raw FlatGraph the World API cannot
consume (sweep / walk_forward / compare all forced a hand re-author to a
Composite, which reproduced the FlatGraph numbers EXACTLY) — refs #94.
- bug: process residency is O(window) (6->173 MiB across 1mo->10y) despite the
documented O(one-chunk) claim; the aura ring stays <=1024 but data-server
retention scales, and streaming_seam.rs only measures the ring — refs #95.
- bug: the bar-period is one knob split across Resample (tunable param) and
Session (baked) and desyncs silently to 0.0 pips; Delay.lag leaks into the
default param_space (the C34 deform-vs-tune discriminator) — refs #96.
- spec_gap: the walk_forward empty-space idiom for param-free strategies is
undefined — refs #97.
- spec_gap (no per-instrument pip registry) and friction (one-directional
ns->ms conversion) re-confirm the known #22 and #80.
- working: C1 determinism over 7y / 148k bars, the lazy streaming source, and
the sweep/WFO cores held under real multi-year load; the Composite re-author
was behaviour-preserving (C23).
Scratch consumer crate + run transcripts committed under fieldtests/ per the
fieldtest convention (matching cycle-0049). The crate path-deps the engine and
reads the real archive; its bins are the repros cited in the issues above.
Run the Phase-1 session-breakout node vocabulary over REAL GER40 M1 bars
from the data-server archive — the first real-data exercise of the shipped
strategy nodes (Resample/Delay/Gt/Session/EqConst/And/Latch + SimBroker).
It is a pure composition of shipped aura-std nodes (no project-specific
signal code), so it lives in the engine repo's examples/ carveout (C9), not
a separate consumer crate. The breakout FlatGraph is reused VERBATIM from
aura-engine/tests/ger40_breakout.rs; only the sources change — four real
M1FieldSource (open/high/low/close, in the fixed C4 merge order) replace the
synthetic VecSources, with pip_size = 1.0 (GER40 is an index).
- examples/ger40_breakout_real.rs — runnable demo over a Sept-2024 window:
prints the RunReport metrics plus a per-session entry/exit trace.
- examples/shared/breakout_real.rs — the single 13-node wiring (the proven
11 + two trace taps), shared by the example and the test via #[path] so it
is never duplicated.
- tests/ger40_breakout_real.rs — gated determinism test (mirrors
real_bars.rs): skips where the archive is absent; where present, asserts
finite metrics, binary Latch exposure (held in {0.0, 1.0}), and
bit-identical C1 reruns of both the report and the recorded series.
chrono / chrono-tz added as aura-ingest dev-dependencies (the Session
timezone and the UTC window bounds), test-only — never on the normal path.
Hand-wires all seven new aura-std nodes (Resample, Delay, Gt, Session, EqConst
x2, And, Latch) + SimBroker into a raw-index FlatGraph over one synthetic
Frankfurt session, pinning the whole vocabulary composed end-to-end:
- held = [0,0,1,1,0] over the 5 bar emissions: flat, latched bar3-close (the
strict breakout landing on session bar 3) through bar4, exit at bar5-close;
- equity = [0,0,0,3,8] pips (SimBroker lagged-exposure integration, pip=1.0);
- a no-entry control (bar3 close == bar2 high) pins the strict-> semantic;
- two disjoint runs byte-identical (C1).
Both spec traps exercised (bar6 rollover closes bar5; bars 1-2 warm Delay[1]).
Phase-1 C9 deliverable; build-step 8 (capstone) of milestone 'Strategy node
vocabulary I'.
closes#91
Maps ctx.now() (epoch-ns UTC) to a Frankfurt session bar index: emits
bars_since_open:i64 = (local wall-clock minutes past the 09:00 open) /
period_minutes, in tz-aware DST-correct local time. So local 09:45 reads 3 in
both CEST (UTC+2) and CET (UTC+1) — the close-instant convention (spec 0050
§4.1). Trigger is an f64 Any input (value ignored, wired from close15) so the
node fires once per completed bar. Open/tz/period are baked structural config,
not scalar params (a timezone is not a scalar, C11). Admits chrono/chrono-tz to
aura-std — vetted DST math, never hand-rolled (per-case dep policy). Last node;
build-step 7 of milestone 'Strategy node vocabulary I'.
closes#90
Aggregates a fine 4-field OHLC stream into period_minutes buckets (open=first,
high=max, low=min, close=last), emitting a completed bar ONLY on bucket rollover
(ctx.now() crossing into the next bucket) — C2: a bar is actionable only once
complete, partials are never emitted and the last partial bar is dropped (no EOF
flush). Four Barrier(0) f64 inputs, a 4-field f64 record output (the first
multi-field-output node in aura-std, mirroring the Ohlcv engine fixture). The bar
carries no timestamp; the engine stamps the close-instant. Build-step 6 of
milestone 'Strategy node vocabulary I'.
closes#89
Level-sensitive SR latch: held -> 1.0 on set, -> 0.0 on reset (reset-dominant),
sample-and-held across quiet cycles; None while both legs are cold. Emits f64
directly (1.0 long / 0.0 flat) so it feeds SimBroker's exposure slot with no
cast — the resolution of the bool->f64 seam (no separate Exposure node in the
strategy DAG). State persisted in the struct across cycles, like SimBroker.
Build-step 5 of milestone 'Strategy node vocabulary I'.
closes#88
Emits the value its series input carried 'lag' fired cycles ago, held in a
node-owned ring (lookbacks()=[1], the past in node state not the input column).
Warm-up is skip-emit (None for the first 'lag' cycles — never a fabricated
sentinel). Output is a pure function of pre-cycle state, the precondition for a
future engine to close a feedback loop (the RTL register C5 names). prevHigh15 =
Delay[1] on high15. Build-step 4 of milestone 'Strategy node vocabulary I'.
closes#87
Stateless bool x bool -> bool, out = (a && b). Computes entry = breakout &&
isBar3 in the session-breakout strategy. The bool-input twin of Gt; seed of the
logic family (Or/Not are separate node types, the operator is topology, never a
swept param). Build-step 3 of milestone 'Strategy node vocabulary I'.
closes#86
Stateless f64 x f64 -> bool, out = (a > b), STRICT: a == b emits false (a close
exactly equal to the previous bar's high is not a breakout). Computes
breakout = close15 > prevHigh15 in the session-breakout strategy. First
bool-emitting f64 comparator; the operator is topology (relational siblings are
separate types, never a swept param). Build-step 2 of milestone 'Strategy node
vocabulary I'.
closes#85
Stateless gate, out = (value == target): the i64->bool comparator the
session-breakout strategy needs to turn Session's bars_since_open into
isBar3/isBar5Close (two instances bound to 3 and 5). The operator is topology
(a concrete node type), only target is a knob (C11/C12). Build-step 1 of
milestone 'Strategy node vocabulary I'.
closes#84
Milestone 'Strategy node vocabulary I — temporal, logic, resample & session':
the 7 aura-std nodes (EqConst, Gt, And, Delay, Latch, Resample, Session) plus a
synthetic e2e fixture, driven by the GER40 15m session-breakout. Records the
design pass's cross-cutting decisions: close-instant timestamp convention,
Latch emits f64 (Exposure dropped from the DAG), the EqConst i64->bool gate,
the Resample-only Barrier(0) firing regime, and the build order. refs #84-#91.
The cycle-0047 typed-Scalar params migration (86746e3) changed each
RunManifest param value on the C18 flat runs store from a bare JSON
number (2.0) to an externally-tagged Scalar ({"F64":2.0}), but did not
migrate the persisted runs.jsonl. Registry::load could no longer
deserialize any pre-0047 line, so `aura runs list` / `runs rank` errored
out on a researcher's existing run history (bug F5, surfaced by the
"The World, part II" milestone fieldtest).
Fix: a localized read-side back-compat mirror (src/compat.rs). load()
now reads each line via RunReportRead, whose ScalarRead accepts BOTH the
tagged form AND a bare JSON number (coerced to Scalar::f64), then
converts into the canonical RunReport. The forward write path (append /
RunReport::to_json) is untouched — it still emits the tagged form. This
is a one-directional read widening, not a format change.
The secondary clause of #83 (parse error exits 0) was triaged as
not-a-bug: runs list/rank already exit 2 via load_runs_or_exit
(main.rs:742); the "(exit 0)" in the fieldtest transcript was a pipe
measurement artifact ($? read head's exit code, not the binary's).
RED test (committed 20511d5) now green; full workspace suite + clippy
green.
closes#83
RED test pinning the real defect of bug F5 (surfaced by the milestone
"The World, part II" fieldtest). Registry::load cannot deserialize a
runs.jsonl line written before the cycle-0047 typed-Scalar params
migration, where each param value is a bare JSON float (e.g.
["sma_cross.fast",2.0]) rather than the tagged {"F64":2.0}. The test
feeds a minimal autonomous legacy line and asserts it loads — the bare
float read back via the documented f64 coercion — instead of erroring at
the first param.
Fails RED at the diagnosed cause (Parse "expected value" on the bare
float). GREEN side (back-compat read, no change to the typed-Scalar
forward format) follows.
refs #83
Milestone-close gate for "The World, part II — orchestration families".
Four end-to-end scenarios derived top-down from the milestone promise,
each built from HEAD and run as a downstream consumer through the public
API + CLI only (no crates/*/src read):
- mw_3 (headline): real EURUSD-M1 walk-forward composing all four axes
(grid sweep + optimize in-sample, OOS stitch) over the M1FieldSource
seam + lineage — 5 windows, 145 200-pt stitched curve, C2 held.
- mw_2: monte_carlo over 128 seeds, reproducible McFamily (C1) + aggregate.
- mw_1: random sweep -> optimize -> 64-member lineage round-trip.
- mw_4: family-lineage CLI transcript.
Findings: 4 working (promise empirically delivered), 1 bug, 3 friction.
- bug F5: `aura runs list`/`rank` cannot read a pre-0047 runs.jsonl — the
cycle-0047 typed-Scalar params migration (86746e3) left the persisted
flat store unmigrated; load() fails at the first bare float (+ exits 0).
- friction F6: no epoch_ns_to_unix_ms inverse for real-data walk-forward.
- friction F7: DataServer/DEFAULT_DATA_PATH not re-exported by aura-ingest.
- friction F8: family store is directory-keyed, undocumented on Registry::open.
Spec: docs/specs/fieldtest-milestone-the-world-orchestration-families.md
Add the typed/named convenience view on `WalkForwardResult`, closing the
symmetry cycle 0048 left open: it shares the `(space, tag-free Vec<Cell>)`
idiom with `SweepFamily` but had no named-params accessor, so a consumer
hand-wrote `zip_params(&result.space, &result.windows[i].run.chosen_params)`.
`named_params(window)` pairs each param-space name with that window's chosen
value in slot order, delegating to `zip_params` — the direct mirror of
`SweepFamily::named_params`. A derived view, no new stored state.
closes#76
Add `RandomBinder`, the by-name sibling to `SweepBinder`, so a random
sweep (`RandomSpace`, C12.1) can be built by name against `param_space()`
instead of by positional `Vec<ParamRange>` slot order. By-name resolution
makes a same-kind transposition (e.g. swapping the I64 ranges for
`fast.length` and `slow.length`) structurally impossible — the failure
class `RandomSpace::new`'s positional validation passes silently.
Direct structural mirror of `SweepBinder`: `Composite::range` opens the
binder, `.range(name, ParamRange)` accumulates, `.sweep(count, seed, run)`
resolves the named ranges via the shared `resolve_into` (new `resolve_ranges`
caller) and runs the disjoint sweep. New `BindError::EmptyRange` mirrors
`EmptyAxis`; `ParamRange::is_empty` homes the non-empty invariant the named
layer pre-checks so `RandomSpace::new` cannot fail.
closes#79
spec auto-sign is now fixed /boss behaviour in the skills plugin (no longer a per-project toggle), so the explicit 'enabled' entry is redundant. No behaviour change — auto-sign was already on for this project.
The SweepError type-level rustdoc still read "A structural fault constructing a
`GridSpace`" though the enum now also gates `RandomSpace::new` (the three
random-only variants are each documented individually, but the type summary was
stale). Broaden it to name both spaces and group the grid vs. random faults.
Doc-only, behaviour-preserving — the same low-grade doc-debt class the 0049
audit fixed inline for the module doc. Surfaced by the cycle-0049 fieldtest.
refs #52
Public-API field test of the random param-sweep surface, from a standalone
downstream-consumer crate (path-deps only; the public interface = ledger +
glossary + spec 0049 + cargo doc rustdoc; no crates/*/src read). Four bins,
each built from HEAD and run: continuous tuning (200-point random tune ranked
by total_pips), the typed validation gate (all five reachable SweepError
variants pre-run), reproducibility + seed-sensitivity + the full i64::MIN..=MAX
sampler edge, and Space-trait interchangeability (one tune_and_rank<S: Space>
over both GridSpace and RandomSpace).
Findings: 0 bugs, 4 working, 2 spec_gap, 1 friction. The four working findings
confirm the cycle's acceptance criterion empirically — the headline tune reads
as the code a researcher would write, the gate is precise and fires before any
run, the C1 reproducibility promise is checkable in one line, and the Space
trait delivers one-consumer/both-enumerations.
Triage of the actionable findings:
- friction (no named-axis builder for RandomSpace — positional Vec<ParamRange>
must align with param_space() by hand, and a same-kind transposition passes
validation silently): filed as a feature for a future cycle, refs #79
(a RandomBinder sibling to the grid's SweepBinder).
- spec_gap (SweepError rustdoc summary named only GridSpace): fixed inline in a
follow-up doc commit.
- spec_gap (NonNumericRange / Bool-slot ranges unreachable with the shipped
aura-std node roster — no node declares a Bool/Timestamp knob): RATIFIED as
intentional. The variant is a forward-looking structural guard for the C16
"author your own node" path (a Bool/Timestamp param-slot a custom node may
declare); its current untriggerability with the standard roster is expected,
not drift.
refs #79
Architect drift review over 3de00e2..HEAD (the 0049 spec/plan/feat plus the
two intervening refactors that had not been audited: aura-ingest M1 transpose
6390093, aura-std SMA perf 67c1f51). Verdict: feature clean — C1 determinism
preserved (RandomSpace points seed-determined before any run; grid path
behaviour-preserving via trait-forwards-to-inherent), the #52/#71 source-seam
firewall honoured in code (sweep stays &S: Space + Fn(&[Cell]) -> RunReport, no
Source type enters the sweep layer; SplitMix64 a code-path-disjoint instance),
and both intervening refactors behaviour-preserving (pinned by
incremental_matches_full_resum_within_tolerance and
chunked_accumulation_equals_single_transpose).
Two low-grade doc-debt items found and fixed inline (doc-only, behaviour-
preserving):
- sweep.rs module doc named only the grid axis; refreshed to name RandomSpace
+ the Space trait the module now also owns.
- C12 ledger had no realization note for axis-1 (param-sweep); added one
recording the grid (0028) + random (0049) landing and the Space-trait
unification.
Regression gate: the project configures no regression scripts, so the test
suite + lint are the gate. Independently verified: cargo test --workspace green
(incl. the 17 new engine tests + 6 public-API E2E), cargo clippy --workspace
--all-targets -- -D warnings clean.
Drift-clean, not a milestone close (no milestone fieldtest run this cycle).
Ship the random half of the C12.1 param-sweep axis (grid landed in 0028).
A new `Space` trait (`points`/`param_specs`) generalizes `sweep`/`sweep_with_threads`
from `&GridSpace` to `&S: Space`; `GridSpace`'s trait impl forwards to its existing
inherent methods, so the grid path is behaviour-preserving (C1, every pre-existing
grid sweep test stays green). `RandomSpace` is the second `Space` producer: N seeded
uniform points over per-slot typed `ParamRange { lo, hi }` ranges (I64 inclusive
[lo,hi], F64 half-open [lo,hi)), validated in `new` against the param-space — a
non-numeric slot, a range-kind mismatch, and an empty range are the three new typed
`SweepError` variants, all caught before any run. Sampling reuses the existing
bit-stable `SplitMix64` (promoted to `pub(crate)`) as a code-path-disjoint instance
from the data-edge seed RNG — the #52/#71 World-II source-seam firewall: they share
only the `u64` type, never a path. The sweep-layer signature stays param-only
(`Fn(&[Cell]) -> RunReport`); no `Source`/stream type enters it.
Deviation from the spec's verbatim sampler, accepted as a correctness hardening:
the I64 draw guards the full-width span ([i64::MIN, i64::MAX] computes a span of
2^64 that wraps a u64 to 0) and uses `lo.wrapping_add(draw as i64)` instead of a
plain `+`. This avoids both the `% 0` divide-by-zero at the full domain and the
signed-overflow panic the literal form hits on wide ranges in debug builds, while
preserving uniform-over-[lo,hi] draws and determinism (an extra RED test,
random_points_full_range_i64_does_not_panic, pins it). Modulo bias for spans not
dividing 2^64 remains the spec's documented, accepted simplification (param search
needs no cryptographic uniformity).
Includes a public-API E2E suite (tests/random_sweep_e2e.rs) exercising the feature
exactly as a downstream researcher would — reproducibility at the report boundary,
seed-sensitivity, in-range draws, the typed NonNumericRange gate, a well-formed
empty (count==0) family, and Grid/Random interchangeability through `Space`.
Verified: cargo test --workspace green (incl. 17 new engine tests + 6 E2E);
cargo clippy --workspace --all-targets -- -D warnings clean.
closes#52
Five bite-sized tasks for the RandomSpace cut: (1) the Space trait +
behaviour-preserving generalization of sweep/sweep_with_threads, (2) typed
ParamRange, (3) RandomSpace::new validation + three SweepError variants,
(4) SplitMix64 -> pub(crate) + the seeded sampler, (5) public exports +
random-sweep integration. Each task is RED-first with single-substring test
filters; the grid suite is the C1 behaviour-preservation regression guard.
refs #52
RandomSpace + a Space trait + typed ParamRange: the random half of the
C12.1 param-sweep axis (grid shipped in 0028). N seeded uniform draws over
declared continuous ranges, fed to the same enumeration-agnostic sweep core.
Param-only sweep signature and a code-path-disjoint SplitMix64 instance keep
the World-II source-seam firewall (refs #71) intact.
Auto-signed under /boss spec auto-sign: fresh grounding-check PASS and a
unanimous five-lens spec-skeptic panel (criterion, grounding, scope-fork,
ambiguity, plan-readiness all SOUND).
refs #52
Sma::eval re-summed the whole window every tick — O(length)/tick, multiplied
across millions of bars and every sweep point. Replace with the industry-standard
incremental running sum (ta-lib shape): the node owns the window ring and keeps a
running sum, adding the new sample and subtracting the evicted one each cycle, so
the input column drops to depth 1 (lookbacks() = [1], like Ema). The running sum
carries Kahan/Neumaier compensation (the fix pandas' rolling mean adopted) so it
does not drift over long runs; Ema needs none — its recurrence is contractive.
Determinism (C1) holds: same input -> same run, reproducibly. The float output
differs from the full re-sum in the last ULPs (a new, deterministic baseline); the
suite needed only one arity-assumption update (Sma lookback 3 -> 1), no equity
golden changed.
closes#39
load_m1_window collected every bar into an intermediate Vec<M1Parsed> (a full
AoS materialization, un-presized so it grew by reallocation) before transpose_m1
copied it again into the SoA columns — ~2x peak load memory plus realloc churn.
Transpose each chunk straight into the columns instead, via a new shared
M1Columns::extend_from_bars (reserves per chunk for amortized growth). One fewer
full copy, halved peak. Behaviour-preserving: the resulting M1Columns is
byte-identical and transpose_m1's / load_m1_window's signatures are unchanged.
Pinned by a new parity test that needs no live DataServer.
closes#78
cycle 0048 tidy. Architect drift review (86746e3..9febdb9, against the design
ledger C7/C1/C12/C20/C23) found the walk-forward param-plane cut drift-clean:
- C7 made honest: WindowRun.chosen_params is now genuinely tag-free Vec<Cell>;
the param kind lives once on WalkForwardResult.space. No residual per-value
tag. The deleted scalar_as_f64's per-value unreachable! became a per-slot
schema cost (the coerce builder map), correctly invoking C20 only on the
timestamp arm.
- C1 behaviour-preserving: the i64->f64 / f64 coercions are value-identical to
the deleted helper; the migrated fixtures keep mean 2.5 / p50 2.5 / mean 1.5.
The Bool->0/1 arm is the only new behaviour, pinned by
param_stability_reduces_bool_slot_to_fraction_true (0.75).
- SweepFamily symmetry is real: the (space + tag-free Vec<Cell> points +
zip_params) idiom is mirrored while the two stay distinct types. The arity
debug_assert reads space.len() against windows (not the moved-out runs) —
correct. walkforward_family derives space from the same blueprint sweep_over
uses, so the kinds genuinely match.
- No stale references: the only "Vec<Scalar>" / "self-describing record"
matches are in docs/plans/0048 (a historical plan quoting the BEFORE state by
design) and ledger lines describing Scalar itself (still true). No live code
or current-state doc carries the old Fork-A chosen_params rationale.
Regression check: no regression script configured (project facts list only
build/test/lint/doc) — no-op; architect is the gate. The four gates
(build/test/clippy -D warnings/doc) and the acceptance greps were verified
green before the close commit.
Follow-on: the out-of-scope WalkForwardResult::named_params (closing the last
asymmetry vs SweepFamily::named_params) was filed as a droppable idea (#76).
Not actionable drift — no consumer is blocked (zip_params already serves it).
Make the walk-forward param plane C7-clean and symmetric with the sweep param plane. WalkForwardResult gains `space: Vec<ParamSpec>` (the kinds, once, mirroring SweepFamily.space) and WindowRun.chosen_params changes Vec<Scalar> -> Vec<Cell> (the tag-free coordinate the inner sweep already produces). param_stability now resolves its numeric coercion once per slot against the schema (result.space) into a Vec<fn(Cell)->f64>, then runs a type-blind reduction loop — replacing the old per-window-value scalar_as_f64 with its per-value unreachable!. scalar_as_f64 is deleted.
This overturns cycle 0047's "Fork A" (chosen_params stays Vec<Scalar>), which was justified by Scalar's new serializability. That defense was moot: WalkForwardResult and WindowRun carry no serde derives — they are never serialized; only the embedded oos_report (a RunReport, typed via the 0047 lift) is. So chosen_params is a pure in-memory substrate for param_stability, and Vec<Scalar> made it C7-impure (carrying the kind N×M times at the value, where the kind is per-slot constant). The substantive case — kind at the schema, SweepFamily symmetry, per-value unreachable! eliminated — wins, the same argument-form that retired the {kind,cell} Scalar struct in 0047.
The seam ([A]): walk_forward / walk_forward_with_threads take `space: Vec<ParamSpec>` by value, a sibling of the run-window closure; the Fn(WindowBounds)->WindowRun+Sync bound is unchanged and run_indexed is untouched (space is never captured by the Sync closure, moved onto the result at the end). A debug_assert in walk_forward_with_threads guards the arity coupling (every window's chosen_params.len() == space.len(), guaranteed by the same blueprint). walkforward_family computes the space once before the roll and passes the tag-free winner (chosen_params: best.params) directly, dropping the from_cell+zip reconstruction.
[D]: a Bool param slot coerces to 0/1 (mean = fraction "on"), keeping the coercer total over the knob kinds; a Timestamp slot is structurally impossible (C20) so the schema pass carries one unreachable! (once per slot, never per value). param_stability keeps an `if result.windows.is_empty() { return Vec::new(); }` guard to preserve its documented empty-on-no-windows contract. WalkForwardResult and SweepFamily share the (space + tag-free Cell points + zip_params) idiom but stay distinct types — different axis (Sweep enumerates input coordinates; WF rolls time and chosen_params is the optimizer's output), and WF additionally carries bounds + oos_equity + stitched_oos_equity.
Behaviour-preserving (C1): the three migrated walk-forward tests stay green with identical MetricStats (param_stability_reduces_chosen_params_per_slot still pins mean 2.5 / p50 2.5 / mean 1.5); the i64->f64 and f64 coercions are value-identical to the deleted scalar_as_f64. The one genuinely new behaviour (Bool 0/1) is unexercised by built-in grids and pinned by the new param_stability_reduces_bool_slot_to_fraction_true (mean 0.75, 3-of-4 true).
Gates: build --workspace --all-targets, test --workspace, clippy --workspace --all-targets -D warnings, doc --workspace --no-deps — all clean.
closes#75
Two-task execution plan for spec 0048: Task 1 lands the whole aura-engine/walkforward.rs cut atomically (struct shapes, walk_forward seam, param_stability rewrite, scalar_as_f64 deletion, fixture migration, new Bool-slot test) gated on a crate-only `cargo test -p aura-engine`; Task 2 threads the aura-cli caller and runs the workspace gates. Keeps a windows.is_empty() guard in param_stability to preserve the documented empty-on-no-windows contract (C1).
refs #75
Settled-design spec for the walk-forward param-plane cut: WalkForwardResult gains space: Vec<ParamSpec>, WindowRun.chosen_params Vec<Scalar> -> Vec<Cell>, walk_forward takes the space by value (Fn-closure generic untouched), and param_stability schema-checks a per-slot coercer (Bool -> 0/1, Timestamp unreachable per C20) before a type-blind reduction, deleting scalar_as_f64 + its per-value unreachable.
Behaviour-preserving (C1): param_stability yields identical MetricStats over the existing fixtures. Overturns 0047's Fork A (chosen_params stays Vec<Scalar>) — the WF structs carry no serde derives, so the serializability defense is moot; the substantive case is C7 purity + SweepFamily symmetry. grounding-check PASS.
refs #75
Scalar was `struct { kind: ScalarKind, cell: Cell }` — "a Cell wearing a kind hat." The recorded reason for that shape was migration ease, which is not a design rationale (CLAUDE.md: rationale != effort), so the struct had no substantive defense. Redefine it as the native tagged union it conceptually is:
enum Scalar { I64(i64), F64(f64), Bool(bool), Timestamp(Timestamp) }
This is substantively better on four axes: kind/bits skew becomes unrepresentable (illegal states gone); accessors panic loudly on the wrong variant instead of a release-mode silent bit reinterpret; PartialEq derives the documented IEEE-754 / cross-kind value semantics (the hand-roll, needed only because the struct inherited Cell's bitwise compare, is gone); and serde is a plain derive emitting the externally-tagged wire form ({"I64":10}/{"F64":2.5}) — the private ScalarRepr shadow enum that motivated this was never needed. It is also C7-honest: erased-on-the-hot-path (Cell) and self-describing-at-the-edge (Scalar) are two disjoint types bridged by explicit conversion.
The whole public API is preserved (Scalar's fields were private), so call sites do not churn: the enum change is contained to scalar.rs, where cell() now encodes and from_cell() decodes per kind. Cell and ScalarKind are untouched.
With Scalar serializable, lift RunManifest.params from Vec<(String, f64)> to Vec<(String, Scalar)>: the param's kind (an i64 length vs an f64 scale) now survives into the C18 record (runs.jsonl) and the CLI JSON instead of collapsing to f64. scalar_as_param_f64 is deleted; sim_optimal_manifest passes typed params through. This is a deliberate wire-shape change — params now render as tagged scalars; the JSON-asserting tests are updated to the new shape on purpose.
Hand-authored manifest params across the CLI's single-run/mc/macd sites use honest kinds (lengths -> i64, scales -> f64) so they match the sweep path (which already derives correct kinds via zip_params); their in-binary JSON assertions are re-tagged accordingly.
Walk-forward fork resolves with no code change: WindowRun.chosen_params stays Vec<Scalar> (the serializable record carrier), reduced to f64 only at the param_stability statistic boundary (scalar_as_f64 retained). Glossary 'cell' entry updated to describe Scalar as the disjoint tagged union, not 'a cell plus its kind tag'.
Gates: build --all-targets, test --workspace (incl. new scalar_serde_round_trips), clippy -D warnings, doc --no-deps — all clean.
Cell becomes the carrier of the construction path; Scalar narrows to the author/render boundaries. The validated/enumerated param point carries no redundant kind (it lives once, in the declared param-space); at the author edge the kind is a checksum — two independent sources, the typed value vs. the slot — so the self-describing Scalar stays there. The name->slot binding is dynamic (C23), so the check is necessarily a runtime one and the value must self-describe for it.
Base/frontend split (the AnyColumn push/push_cell pattern one level up): compile_with_cells / bootstrap_with_cells are the kind-check-free base; compile_with_params / bootstrap_with_params are the frontend that adds only the per-value kind checksum, strips to cells (new Scalar::cell(), the partner of Scalar::from_cell), and delegates. lower_items loses its per-primitive kind-check; PrimitiveBuilder::build and the std node builders (sma/ema/exposure/lincomb) read cells.
Boundary: construction -> Cell (PrimitiveBuilder::build, lower_items, GridSpace.axes/points, SweepPoint.params, the sweep closure). Author edges stay Scalar (GridSpace::new, bind, compile_with_params/bootstrap_with_params). walkforward chosen_params stays a self-describing Scalar report record (Option A — WalkForwardResult carries no space); the cell winner is reconstructed once at the WindowRun site via from_cell.
injective-check moved ahead of the arity-check in the frontend to preserve the pre-split error order (DuplicateParamPath before ParamArity). The lossy i64->f64 param projection (scalar_as_f64 / scalar_as_param_f64) is deliberately untouched — a separate follow-up.
Behaviour-preserving (C1): build --all-targets / test / clippy -D warnings / doc all clean across the workspace.
cycle 0047 tidy. Architect drift review (b188773..82635fa, against the
design ledger C7/C8/C1) found the carrier swap drift-clean:
- Node::eval -> Option<&[Cell]>, all 8 aura-std out-buffers [Cell; N], the
inter-node forward via the branch-free push_cell (harness.rs:426); Scalar
survives only on the three keep-set boundaries (param plane,
AnyColumn::get, source ingestion at harness.rs:402). The convert/keep
partition and the three-role test rule are honoured site-for-site.
- The dual API is coherent: fallible push (ingestion + kind-guard tests)
vs infallible push_cell (bootstrap-verified inter-node forward) are each
load-bearing and tested. No dead code, no C7 violation (no tag on the hot
path; scratch: Vec<Cell> reused via clear(), no per-cycle alloc).
- Both C7 realization notes + the C8 prose accurately describe the landed
state. 285 tests green; build/clippy/test/doc clean.
Tidy fix (pre-existing, surfaced by the review):
- docs/glossary.md `node` entry said a node implements `schema()`, but
Node::schema() was removed in cycle 0024 (the signature is declared
pre-build on PrimitiveBuilder). Corrected to `lookbacks()` + `eval(ctx)`,
the methods the Node trait actually carries. Record-reality.
Motivation
----------
The C7 realization note (cd3d1ca / 049f22a) deferred this step: with
Scalar split into { kind, cell }, the tag-free Cell could become the
hot-path carrier, but eval and the edges still flowed the fatter
self-describing Scalar. This lands the swap.
Change
------
- Node::eval now returns Option<&[Cell]>; every node out-buffer is
[Cell; N]. The inter-node forward copies the row into a Vec<Cell> and
pushes each field via a new branch-free, infallible AnyColumn::push_cell
— the edge from_field->slot kind match is verified once at bootstrap
(C7), so the runtime push needs no per-value kind check. It only removes
a Result the forward path already .expect()ed.
- Scalar stays on the self-describing dynamic boundaries: the param plane
(build / bind / compile_with_params / sweep points / RunManifest),
AnyColumn::get (the type-erased read for sinks / serde), and source
ingestion (Source::next, the heterogeneous C3 merge). The fallible
AnyColumn::push is retained for ingestion + the kind-guard tests.
Trade
-----
The per-value runtime kind check on node OUTPUT is removed: a node that
builds a wrong-kind cell pushes raw bits with no panic. This is the same
authoring-bug class C8 already leaves to a debug_assert (output width);
node-output-kind correctness is the node's declared FieldSpec contract,
caught by each node's own eval test. Cross-node kind agreement still rests
on the bootstrap kind-check (bootstrap_rejects_*_kind_mismatch, green).
Tests
-----
Behaviour-preserving (C1): 285 green, 0 red. The only test-expectation
edits are carrier re-spellings (Scalar::f64(x) -> Cell::from_f64(x)) of
identical values; test column writes and out-of-graph channel rows stay
Scalar (the spec's three-role rule). Adds an AnyColumn::push_cell
round-trip test plus a cross-kind end-to-end forward test (a mixed
f64/i64 record whose bit patterns diverge across kinds — pins the i64
forward path the f64-only tests cannot catch). Ledger C7/C8 notes updated.
Gates: cargo build / test / clippy --all-targets -D warnings / doc
--workspace all clean.
closes#74
Four tasks for the behaviour-preserving carrier swap: (1) add the
branch-free AnyColumn::push_cell + round-trip test (additive); (2) the
atomic lib-code flip (Node::eval -> Option<&[Cell]>, 8 aura-std nodes,
recorder, harness forward loop) gated by a lib-only build; (3) the
cfg(test) migration (fixtures + eval-output assertions per the spec's
three-role rule) gated by the full four-gate run; (4) the C7/C8 ledger
realization note.
refs #74
Auto-signed under /boss spec auto-sign: all objective gates green
(Step-1.5 precondition, Step-4 self-review, Step-5 grounding-check PASS)
and a unanimous five-lens spec-skeptic panel SOUND, after one editorial
round on the ambiguity lens (the test-migration rule was sharpened to a
three-role classification: eval-output -> Cell, column-write -> Scalar,
channel-send -> Scalar).
Spec for the deferred C7/C8 carrier swap flagged by the C7 realization
note: Cell becomes the hot-path value carrier (Node::eval ->
Option<&[Cell]>, node out-buffers, inter-node forward via a new
AnyColumn::push_cell), and Scalar narrows to the self-describing dynamic
boundaries (the param path, AnyColumn::get, source ingestion).
Behaviour-preserving (C1): only the carrier type narrows.
refs #74
Record the Scalar -> {kind, cell} representation change under C7 (cd3d1ca):
Cell as the tag-free single-value carrier, with the hot-path carrier swap
(eval -> Option<&[Scalar]> and the edges) explicitly flagged as the deferred
next step. Doc-only; no code change.
Motivation
----------
`Scalar` was a tagged enum (I64/F64/Bool/Ts), so every scalar value
physically carried its own kind tag. But the kind is already known from
the schema/port/column the value flows through (C7: the type is a
property of the column, not of the value — the hot path is already
columnar `Column<T>`, and `AnyColumn::get` *reconstructs* the tag from
the column on the way out). The per-value tag was therefore redundant
with the kind the surrounding context already holds.
That redundancy had three costs:
* It baked an implicit `match` (a branch) into every function that read
a Scalar payload — even where the caller statically knew the type.
The tag could never be exploited away.
* Size: a tagged enum is tag + payload = 16 bytes (f64/i64 alignment),
twice the 8 bytes the value needs. A `Column<Scalar>` would be double
the memory and half the cache utilisation.
* It is the shared root of several downstream papercuts we keep hitting
— the lossy f64 manifest field, the `unreachable!` panic on a
non-numeric param, the serde-tag question — all symptoms of "the type
is baked into the value".
Change
------
Introduce `Cell`: a type-erased 64-bit word (`struct Cell(u64)`) that is
not readable without external type context. It is constructed per base
type (`from_i64/from_f64/from_bool/from_ts`) and read only by naming the
type at the call site (`i64()/f64()/bool()/ts()`) — each a branch-free
bit-cast. The hot path resolves the kind once at the boundary (from the
schema) and then reads natively, with no per-value branch. `Cell` knows
nothing of `Scalar` or `ScalarKind`; the dependency is strictly one-way,
and it lives in its own `cell.rs` (more is planned on top of it).
`Scalar` becomes `struct { kind: ScalarKind, cell: Cell }` — the
self-describing form for the dynamic boundaries (builder binding,
serialization, rendering), built on top of `Cell`. Its `as_*` accessors
now `debug_assert` the kind and return the native value (free in
release); calling the wrong accessor is a caller bug, not a checked
`Option`. The variant constructors `Scalar::I64(..)` become associated
fns `Scalar::i64(..)`.
`PartialEq` is hand-written (not derived) to preserve the former enum's
value semantics: kinds must match, then native payloads compare, so f64
keeps IEEE-754 behaviour (`NaN != NaN`, `+0.0 == -0.0`) and a kind
mismatch is never equal even when the raw words coincide. A fixture
(`scalar_eq_is_value_not_bitwise`) pins exactly the cases where bit- and
value-equality diverge, so it can't silently regress. `Cell`'s own
`Eq`/`Hash` stay bitwise — correct for a raw word.
The change is behaviour-preserving: Scalar's observable behaviour is
identical to the pre-Cell enum (the value-equality fixture proves it);
only the internal representation changed. The ~440 call sites across the
workspace are a mechanical constructor rename plus ~12 destructuring
sites (match-arms / `let`-patterns) rewritten to `kind()` + `as_*`.
Verified: cargo build --workspace --all-targets, cargo clippy --workspace
--all-targets -- -D warnings, cargo test --workspace — all green.
cycle 0046 tidy (clean). Architect drift review against the design ledger
(C8/C12/C23 + the #71 firewall) found no drift:
- The named view is a derived, non-load-bearing projection (C23 — slot is
identity); no name reaches the flat graph, no new per-point state, no new
identity mechanism. SweepFamily.space is param_space() carried once, not minted.
- Enumeration and determinism (C1) are untouched (GridSpace::points / run_indexed
unchanged; the new SweepFamily.space is equal across thread counts).
- The run-closure signature Fn(&[Scalar]) -> RunReport is byte-unchanged, so
#52's RandomSpace plugs into the same sweep() layer with zero reconciliation
(#71 firewall held).
- All three CLI hand-zips collapsed to one zip_params call each; no un-collapsed
duplicate remains. zip_params sits in aura-core (a pure projection over
aura-core types) — correct home.
- The f64 manifest field stays Vec<(String,f64)> (the typed-param-space precursor
honestly left deferred, not entrenched).
Regression gate green: cargo test --workspace (all crates, 0 failed; 4 new tests
this cycle), cargo clippy --workspace --all-targets -D warnings clean.
Forward-consistency item (NOT drift, out of #57 scope): McDraw/McFamily and
WindowRun still carry bare Vec<Scalar> with no named view — a future item if a
consumer demands it.
closes#57
Thread a derived named view of a sweep point so consumers stop hand-zipping
param_space() names onto the bare &[Scalar]. The free function
zip_params(space, point) -> Vec<(String, Scalar)> in aura-core is the one
shared projection; GridSpace retains the ParamSpec list it already receives in
new() (it was validated then discarded); SweepFamily carries it (stamped once
in sweep_with_threads) and exposes named_params(i). The three CLI hand-zip
sites collapse to one zip_params call each.
The run-closure signature `Fn(&[Scalar]) -> RunReport` is byte-unchanged — the
named view is a derived projection (C23: slot is identity, name is derived),
not a closure-currency change — so #52's RandomSpace plugs into the same
sweep() execution layer with zero signature reconciliation (#71 firewall held).
sim_optimal_manifest now takes typed Vec<(String, Scalar)> and does the lossy
f64 collapse internally (one place; the manifest's f64-precursor field owns its
own lossiness — the typed-manifest upgrade stays the deferred typed-param-space
item, a non-goal here). All eight call sites migrated: three sweep closures
pass zip_params, five hand-listing callers (run_sample, run_sample_real,
mc_family, run_macd, run_sample_seeded) pass Scalar::F64 literals.
Behaviour-preserving: the collapse output is identical to the old per-site zip,
so aura sweep / walkforward / run output is byte-identical; SweepFamily.space
threaded into the aura-registry optimize test literal.
Verified by the orchestrator: cargo test --workspace green (4 new tests:
zip_params x2, sweep_family_carries_param_space, family_named_params_round_trips),
cargo clippy --workspace --all-targets -D warnings clean. (rust-analyzer
emitted stale mid-edit diagnostics; the real cargo build/test is clean.)
refs #57
Brummel