Consolidate aura-ingest's OHLC source-open surface onto the engine-native
epoch-ns Timestamp currency: a private seam-owned inverse, a Timestamp-window
opener, a canonical OHLC bundle opener (fixed C4 O/H/L/C order), and a
DataServer/DEFAULT_DATA_PATH re-export + default_data_server convenience — so a
real-data source builds from aura-ingest alone, with the one ms<->ns crossing
owned by the seam. Behaviour-preserving (byte-identical RunReports).
Grounded against the tree via the ground-spec-0052 workflow (25/28 assertions
confirmed; compare.rs compiler-invisible site + AC1 scope fix applied).
refs #80, #81, #92
Per-cycle specs/plans are step-scoped: every file maps to already-shipped
code, their snippets drift hard against current APIs (InputSpec->PortSpec,
Sim->Harness, schema() removed, C16 zero-dep reversed, ascii-dag retired),
and a stale spec read as an API reference misleads agents into non-compiling
code. No durable knowledge lost — rationale lives in the ledger + code
rustdoc, history in git + the tracker.
- gitignore docs/specs/ and docs/plans/
- delete all 117 existing specs/plans (recoverable via git history)
- CLAUDE.md: specs/plans are local-only ephemeral artifacts
The C12 realization and the 0041 spec claimed streaming residency is O(one chunk) at the process level ("a 20-year window streams in the same memory as a one-day one"). That holds only for the aura source ring (M1FieldSource::resident_records(), bounded by one chunk and correctly tested); whole-process RSS grows O(records-touched) because data-server's FileCache retains each window's parsed chunks (~56 B/record) read-only for the pass. Reproduced: 6 MiB @ 1mo -> 173 MiB @ ~10y, linear.
That retention is the replay-many optimization (load-once; close+volume share one parse via the field-agnostic FileKey), not a leak, and no always-on eviction can bound a single forward pass without regressing it. This narrows the docs/comments to the true per-source-ring property and names the FileCache per-window retention as the real, replay-amortized process cost. The streaming_seam assert is unchanged (it was always the ring bound; only its labelling confused ring vs process). The deferred opt-in non-retaining streaming read path is parked as Brummel/data-server#2.
closes#95
Add a C20 realization recording the resolution of the GER40 deep-dive's #94
finding: a hand-wired FlatGraph is not a shippable strategy (no param_space, so
World-opaque); the canonical form is the Composite blueprint, the FlatGraph its
compiled substrate (C23). Pins the bar-period-as-structural-construction-arg
decision (spec 0051) to the structural-axis-vs-tuning-param split.
refs #94
Milestone spec: ship the GER40 session-breakout as a Composite blueprint so the
same strategy runs across backtest / sweep / walk_forward / compare without
re-authoring (the #94 fieldtest friction). Records the design decision that the
bar period is a structural axis (a construction arg binding both Resample and
Session, so a sweep cannot desync them — #96) and that the canonical shippable
strategy form is a Composite (FlatGraph is its compiled substrate). Authoring-
side; no engine-core change (the existing bind() + GraphBuilder + param_space()
layer already suffices).
refs #94, #96, #97
Public-API research deep-dive driving the just-shipped GER40 real-data
session-breakout demo (5b5f034) through the escalating sequence a serious
researcher attempts next: scale to multi-year, walk-forward, structural-param
sweep, multi-instrument compare. Standalone consumer crate (path-deps only;
public interface = ledger + glossary + example corpus + cargo doc; no
crates/*/src read). All runs against the real /mnt/tickdata/Pepperstone archive.
Findings: 3 bugs, 1 friction, 3 spec_gap, 1 working. Triaged to the tracker:
- spec_gap: the flagship breakout ships as a raw FlatGraph the World API cannot
consume (sweep / walk_forward / compare all forced a hand re-author to a
Composite, which reproduced the FlatGraph numbers EXACTLY) — refs #94.
- bug: process residency is O(window) (6->173 MiB across 1mo->10y) despite the
documented O(one-chunk) claim; the aura ring stays <=1024 but data-server
retention scales, and streaming_seam.rs only measures the ring — refs #95.
- bug: the bar-period is one knob split across Resample (tunable param) and
Session (baked) and desyncs silently to 0.0 pips; Delay.lag leaks into the
default param_space (the C34 deform-vs-tune discriminator) — refs #96.
- spec_gap: the walk_forward empty-space idiom for param-free strategies is
undefined — refs #97.
- spec_gap (no per-instrument pip registry) and friction (one-directional
ns->ms conversion) re-confirm the known #22 and #80.
- working: C1 determinism over 7y / 148k bars, the lazy streaming source, and
the sweep/WFO cores held under real multi-year load; the Composite re-author
was behaviour-preserving (C23).
Scratch consumer crate + run transcripts committed under fieldtests/ per the
fieldtest convention (matching cycle-0049). The crate path-deps the engine and
reads the real archive; its bins are the repros cited in the issues above.
Milestone 'Strategy node vocabulary I — temporal, logic, resample & session':
the 7 aura-std nodes (EqConst, Gt, And, Delay, Latch, Resample, Session) plus a
synthetic e2e fixture, driven by the GER40 15m session-breakout. Records the
design pass's cross-cutting decisions: close-instant timestamp convention,
Latch emits f64 (Exposure dropped from the DAG), the EqConst i64->bool gate,
the Resample-only Barrier(0) firing regime, and the build order. refs #84-#91.
Milestone-close gate for "The World, part II — orchestration families".
Four end-to-end scenarios derived top-down from the milestone promise,
each built from HEAD and run as a downstream consumer through the public
API + CLI only (no crates/*/src read):
- mw_3 (headline): real EURUSD-M1 walk-forward composing all four axes
(grid sweep + optimize in-sample, OOS stitch) over the M1FieldSource
seam + lineage — 5 windows, 145 200-pt stitched curve, C2 held.
- mw_2: monte_carlo over 128 seeds, reproducible McFamily (C1) + aggregate.
- mw_1: random sweep -> optimize -> 64-member lineage round-trip.
- mw_4: family-lineage CLI transcript.
Findings: 4 working (promise empirically delivered), 1 bug, 3 friction.
- bug F5: `aura runs list`/`rank` cannot read a pre-0047 runs.jsonl — the
cycle-0047 typed-Scalar params migration (86746e3) left the persisted
flat store unmigrated; load() fails at the first bare float (+ exits 0).
- friction F6: no epoch_ns_to_unix_ms inverse for real-data walk-forward.
- friction F7: DataServer/DEFAULT_DATA_PATH not re-exported by aura-ingest.
- friction F8: family store is directory-keyed, undocumented on Registry::open.
Spec: docs/specs/fieldtest-milestone-the-world-orchestration-families.md
Public-API field test of the random param-sweep surface, from a standalone
downstream-consumer crate (path-deps only; the public interface = ledger +
glossary + spec 0049 + cargo doc rustdoc; no crates/*/src read). Four bins,
each built from HEAD and run: continuous tuning (200-point random tune ranked
by total_pips), the typed validation gate (all five reachable SweepError
variants pre-run), reproducibility + seed-sensitivity + the full i64::MIN..=MAX
sampler edge, and Space-trait interchangeability (one tune_and_rank<S: Space>
over both GridSpace and RandomSpace).
Findings: 0 bugs, 4 working, 2 spec_gap, 1 friction. The four working findings
confirm the cycle's acceptance criterion empirically — the headline tune reads
as the code a researcher would write, the gate is precise and fires before any
run, the C1 reproducibility promise is checkable in one line, and the Space
trait delivers one-consumer/both-enumerations.
Triage of the actionable findings:
- friction (no named-axis builder for RandomSpace — positional Vec<ParamRange>
must align with param_space() by hand, and a same-kind transposition passes
validation silently): filed as a feature for a future cycle, refs #79
(a RandomBinder sibling to the grid's SweepBinder).
- spec_gap (SweepError rustdoc summary named only GridSpace): fixed inline in a
follow-up doc commit.
- spec_gap (NonNumericRange / Bool-slot ranges unreachable with the shipped
aura-std node roster — no node declares a Bool/Timestamp knob): RATIFIED as
intentional. The variant is a forward-looking structural guard for the C16
"author your own node" path (a Bool/Timestamp param-slot a custom node may
declare); its current untriggerability with the standard roster is expected,
not drift.
refs #79
Architect drift review over 3de00e2..HEAD (the 0049 spec/plan/feat plus the
two intervening refactors that had not been audited: aura-ingest M1 transpose
6390093, aura-std SMA perf 67c1f51). Verdict: feature clean — C1 determinism
preserved (RandomSpace points seed-determined before any run; grid path
behaviour-preserving via trait-forwards-to-inherent), the #52/#71 source-seam
firewall honoured in code (sweep stays &S: Space + Fn(&[Cell]) -> RunReport, no
Source type enters the sweep layer; SplitMix64 a code-path-disjoint instance),
and both intervening refactors behaviour-preserving (pinned by
incremental_matches_full_resum_within_tolerance and
chunked_accumulation_equals_single_transpose).
Two low-grade doc-debt items found and fixed inline (doc-only, behaviour-
preserving):
- sweep.rs module doc named only the grid axis; refreshed to name RandomSpace
+ the Space trait the module now also owns.
- C12 ledger had no realization note for axis-1 (param-sweep); added one
recording the grid (0028) + random (0049) landing and the Space-trait
unification.
Regression gate: the project configures no regression scripts, so the test
suite + lint are the gate. Independently verified: cargo test --workspace green
(incl. the 17 new engine tests + 6 public-API E2E), cargo clippy --workspace
--all-targets -- -D warnings clean.
Drift-clean, not a milestone close (no milestone fieldtest run this cycle).
Five bite-sized tasks for the RandomSpace cut: (1) the Space trait +
behaviour-preserving generalization of sweep/sweep_with_threads, (2) typed
ParamRange, (3) RandomSpace::new validation + three SweepError variants,
(4) SplitMix64 -> pub(crate) + the seeded sampler, (5) public exports +
random-sweep integration. Each task is RED-first with single-substring test
filters; the grid suite is the C1 behaviour-preservation regression guard.
refs #52
RandomSpace + a Space trait + typed ParamRange: the random half of the
C12.1 param-sweep axis (grid shipped in 0028). N seeded uniform draws over
declared continuous ranges, fed to the same enumeration-agnostic sweep core.
Param-only sweep signature and a code-path-disjoint SplitMix64 instance keep
the World-II source-seam firewall (refs #71) intact.
Auto-signed under /boss spec auto-sign: fresh grounding-check PASS and a
unanimous five-lens spec-skeptic panel (criterion, grounding, scope-fork,
ambiguity, plan-readiness all SOUND).
refs #52
Two-task execution plan for spec 0048: Task 1 lands the whole aura-engine/walkforward.rs cut atomically (struct shapes, walk_forward seam, param_stability rewrite, scalar_as_f64 deletion, fixture migration, new Bool-slot test) gated on a crate-only `cargo test -p aura-engine`; Task 2 threads the aura-cli caller and runs the workspace gates. Keeps a windows.is_empty() guard in param_stability to preserve the documented empty-on-no-windows contract (C1).
refs #75
Settled-design spec for the walk-forward param-plane cut: WalkForwardResult gains space: Vec<ParamSpec>, WindowRun.chosen_params Vec<Scalar> -> Vec<Cell>, walk_forward takes the space by value (Fn-closure generic untouched), and param_stability schema-checks a per-slot coercer (Bool -> 0/1, Timestamp unreachable per C20) before a type-blind reduction, deleting scalar_as_f64 + its per-value unreachable.
Behaviour-preserving (C1): param_stability yields identical MetricStats over the existing fixtures. Overturns 0047's Fork A (chosen_params stays Vec<Scalar>) — the WF structs carry no serde derives, so the serializability defense is moot; the substantive case is C7 purity + SweepFamily symmetry. grounding-check PASS.
refs #75
Scalar was `struct { kind: ScalarKind, cell: Cell }` — "a Cell wearing a kind hat." The recorded reason for that shape was migration ease, which is not a design rationale (CLAUDE.md: rationale != effort), so the struct had no substantive defense. Redefine it as the native tagged union it conceptually is:
enum Scalar { I64(i64), F64(f64), Bool(bool), Timestamp(Timestamp) }
This is substantively better on four axes: kind/bits skew becomes unrepresentable (illegal states gone); accessors panic loudly on the wrong variant instead of a release-mode silent bit reinterpret; PartialEq derives the documented IEEE-754 / cross-kind value semantics (the hand-roll, needed only because the struct inherited Cell's bitwise compare, is gone); and serde is a plain derive emitting the externally-tagged wire form ({"I64":10}/{"F64":2.5}) — the private ScalarRepr shadow enum that motivated this was never needed. It is also C7-honest: erased-on-the-hot-path (Cell) and self-describing-at-the-edge (Scalar) are two disjoint types bridged by explicit conversion.
The whole public API is preserved (Scalar's fields were private), so call sites do not churn: the enum change is contained to scalar.rs, where cell() now encodes and from_cell() decodes per kind. Cell and ScalarKind are untouched.
With Scalar serializable, lift RunManifest.params from Vec<(String, f64)> to Vec<(String, Scalar)>: the param's kind (an i64 length vs an f64 scale) now survives into the C18 record (runs.jsonl) and the CLI JSON instead of collapsing to f64. scalar_as_param_f64 is deleted; sim_optimal_manifest passes typed params through. This is a deliberate wire-shape change — params now render as tagged scalars; the JSON-asserting tests are updated to the new shape on purpose.
Hand-authored manifest params across the CLI's single-run/mc/macd sites use honest kinds (lengths -> i64, scales -> f64) so they match the sweep path (which already derives correct kinds via zip_params); their in-binary JSON assertions are re-tagged accordingly.
Walk-forward fork resolves with no code change: WindowRun.chosen_params stays Vec<Scalar> (the serializable record carrier), reduced to f64 only at the param_stability statistic boundary (scalar_as_f64 retained). Glossary 'cell' entry updated to describe Scalar as the disjoint tagged union, not 'a cell plus its kind tag'.
Gates: build --all-targets, test --workspace (incl. new scalar_serde_round_trips), clippy -D warnings, doc --no-deps — all clean.
cycle 0047 tidy. Architect drift review (b188773..82635fa, against the
design ledger C7/C8/C1) found the carrier swap drift-clean:
- Node::eval -> Option<&[Cell]>, all 8 aura-std out-buffers [Cell; N], the
inter-node forward via the branch-free push_cell (harness.rs:426); Scalar
survives only on the three keep-set boundaries (param plane,
AnyColumn::get, source ingestion at harness.rs:402). The convert/keep
partition and the three-role test rule are honoured site-for-site.
- The dual API is coherent: fallible push (ingestion + kind-guard tests)
vs infallible push_cell (bootstrap-verified inter-node forward) are each
load-bearing and tested. No dead code, no C7 violation (no tag on the hot
path; scratch: Vec<Cell> reused via clear(), no per-cycle alloc).
- Both C7 realization notes + the C8 prose accurately describe the landed
state. 285 tests green; build/clippy/test/doc clean.
Tidy fix (pre-existing, surfaced by the review):
- docs/glossary.md `node` entry said a node implements `schema()`, but
Node::schema() was removed in cycle 0024 (the signature is declared
pre-build on PrimitiveBuilder). Corrected to `lookbacks()` + `eval(ctx)`,
the methods the Node trait actually carries. Record-reality.
Motivation
----------
The C7 realization note (cd3d1ca / 049f22a) deferred this step: with
Scalar split into { kind, cell }, the tag-free Cell could become the
hot-path carrier, but eval and the edges still flowed the fatter
self-describing Scalar. This lands the swap.
Change
------
- Node::eval now returns Option<&[Cell]>; every node out-buffer is
[Cell; N]. The inter-node forward copies the row into a Vec<Cell> and
pushes each field via a new branch-free, infallible AnyColumn::push_cell
— the edge from_field->slot kind match is verified once at bootstrap
(C7), so the runtime push needs no per-value kind check. It only removes
a Result the forward path already .expect()ed.
- Scalar stays on the self-describing dynamic boundaries: the param plane
(build / bind / compile_with_params / sweep points / RunManifest),
AnyColumn::get (the type-erased read for sinks / serde), and source
ingestion (Source::next, the heterogeneous C3 merge). The fallible
AnyColumn::push is retained for ingestion + the kind-guard tests.
Trade
-----
The per-value runtime kind check on node OUTPUT is removed: a node that
builds a wrong-kind cell pushes raw bits with no panic. This is the same
authoring-bug class C8 already leaves to a debug_assert (output width);
node-output-kind correctness is the node's declared FieldSpec contract,
caught by each node's own eval test. Cross-node kind agreement still rests
on the bootstrap kind-check (bootstrap_rejects_*_kind_mismatch, green).
Tests
-----
Behaviour-preserving (C1): 285 green, 0 red. The only test-expectation
edits are carrier re-spellings (Scalar::f64(x) -> Cell::from_f64(x)) of
identical values; test column writes and out-of-graph channel rows stay
Scalar (the spec's three-role rule). Adds an AnyColumn::push_cell
round-trip test plus a cross-kind end-to-end forward test (a mixed
f64/i64 record whose bit patterns diverge across kinds — pins the i64
forward path the f64-only tests cannot catch). Ledger C7/C8 notes updated.
Gates: cargo build / test / clippy --all-targets -D warnings / doc
--workspace all clean.
closes#74
Four tasks for the behaviour-preserving carrier swap: (1) add the
branch-free AnyColumn::push_cell + round-trip test (additive); (2) the
atomic lib-code flip (Node::eval -> Option<&[Cell]>, 8 aura-std nodes,
recorder, harness forward loop) gated by a lib-only build; (3) the
cfg(test) migration (fixtures + eval-output assertions per the spec's
three-role rule) gated by the full four-gate run; (4) the C7/C8 ledger
realization note.
refs #74
Auto-signed under /boss spec auto-sign: all objective gates green
(Step-1.5 precondition, Step-4 self-review, Step-5 grounding-check PASS)
and a unanimous five-lens spec-skeptic panel SOUND, after one editorial
round on the ambiguity lens (the test-migration rule was sharpened to a
three-role classification: eval-output -> Cell, column-write -> Scalar,
channel-send -> Scalar).
Spec for the deferred C7/C8 carrier swap flagged by the C7 realization
note: Cell becomes the hot-path value carrier (Node::eval ->
Option<&[Cell]>, node out-buffers, inter-node forward via a new
AnyColumn::push_cell), and Scalar narrows to the self-describing dynamic
boundaries (the param path, AnyColumn::get, source ingestion).
Behaviour-preserving (C1): only the carrier type narrows.
refs #74
Record the Scalar -> {kind, cell} representation change under C7 (cd3d1ca):
Cell as the tag-free single-value carrier, with the hot-path carrier swap
(eval -> Option<&[Scalar]> and the edges) explicitly flagged as the deferred
next step. Doc-only; no code change.
Three tasks for spec 0046. T1: zip_params (aura-core), RED-first. T2:
GridSpace retains its ParamSpec list + SweepFamily carries it + named_params
(aura-engine), RED-first, with the aura-registry optimize-test SweepFamily
literal threaded in the same task. T3: sim_optimal_manifest typed-Scalar
migration + all eight call sites (aura-cli), behaviour-preserving. Each
signature/field change threads every one of its sites within its own task,
before that task's workspace compile gate.
refs #57
Thread a derived named view of a sweep point so consumers stop re-zipping
param_space() names onto the bare &[Scalar]. A free function zip_params over
(ParamSpec names ⊗ positional point) in aura-core; GridSpace retains the
ParamSpec list it already receives in new(); SweepFamily carries it and
exposes named_params(i). The run-closure signature
`F: Fn(&[Scalar]) -> RunReport + Sync` stays byte-for-byte (honours #52's
#71-firewall constraint, so RandomSpace plugs in with zero reconciliation).
The lossy f64 collapse moves into the manifest constructor; the view stays
typed. Behaviour-preserving; SweepPoint and enumeration untouched.
Design ratified in-context (brainstorm, approach C over A, free function
over a NamedPoint type); a reconciliation comment on #57 records the
resolved forks with provenance. Human sign-off after the auto-sign panel's
grounding lens caught a blast-radius undercount in the first draft —
corrected to all eight sim_optimal_manifest call sites plus the
aura-registry SweepFamily struct-literal.
refs #57
Behaviour-preserving normalisation of the lineage record. The fused
`family_id: String` ("{name}-{counter}") is split into its two factors —
`family: String` + `run: usize` — and the user-facing handle becomes a derived
`family_id()` method ("{family}-{run}"), never stored or parsed.
Why: the fused field forced the counter logic to generate-and-check candidate
strings (to stay robust to '-' inside a name) rather than read it; with `run` a
plain int, `next_run` is a clean numeric max+1 over the field. The split is more
normalised (one fact per field) and the only property the fused token bought — a
single paste-able CLI handle — is recovered by deriving it. Storage and all
user-visible output stay byte-identical: append_family still returns
"{name}-{run}", Family.id is still "{name}-{run}", and the CLI prints the same
"family_id" lines (the json! key + the returned String, both unchanged).
Internal-only: FamilyRunRecord and group_families (now keyed on the (family,
run) tuple, first-seen order preserved) change; lib.rs re-exports, the
extractors, and all of aura-cli are untouched (everything downstream goes
through Family.id + append_family's returned String). The on-disk
families.jsonl key changes ({"family","run"} vs {"family_id"}) — a gitignored
runtime artifact with no committed fixture, and tests round-trip within a temp
dir, so no migration is owed. Doc reconciliation: the lineage/lib module headers
and the C18 ledger realization note now describe the split shape.
Verification: a workflow ran the refactor in an implementer agent, then three
adversarial lenses (behaviour-preservation, new-logic correctness,
completeness/doc-lag) tried to refute it — the first two found nothing, the
third flagged the four doc-lag sites now fixed here. Self-run gates:
cargo test --workspace green (registry 11, incl. a strengthened round-trip test
pinning the split fields + derived handle), cargo clippy --workspace
--all-targets -D warnings clean, cargo doc clean.
Architect drift review (eeba218..HEAD) + regression gate (the project's
cargo test --workspace is the gate; no separate regression script).
What holds (architect): C9 layering intact — aura-engine gains no registry
dependency, lineage lives entirely in aura-registry. #71 firewall held —
RunManifest/aura-core byte-untouched, no blob/path/payload field on
FamilyRunRecord, the member window is producer-supplied via
Source::bounds()/window_of, never a Vec scan. C2 clean — bounds() is
cursor-independent (no look-ahead). Family store is a disjoint sibling of the
flat store (pinned by a disjointness test). Regression gate green: 228 tests,
clippy --workspace --all-targets -D warnings clean.
Drift resolved:
- [fix, this commit] C18 ledger gap: docs/design/INDEX.md C18 carried no
realization note for the run registry. Added the cycle-0029 (flat registry)
and cycle-0045 (lineage as related records / family store) realization
notes, in the ledger's established per-cycle style.
- [forward-queue] Flat-store CLI dead-end: after this cycle no CLI command
writes the flat runs.jsonl (sweep/walkforward moved to the family store,
aura run never persisted), so aura runs list / rank read an unfed store.
This is a spec-accepted state (0045 kept runs list "for standalone runs
only") and a genuine design fork (give aura run a persisting path vs. retire
the flat-store CLI surface) with no clear default — filed as #73 rather than
decided here. Recorded as a deferred follow-up in the C18 ledger note.
- [carry-on] Low/cosmetic: family-wrapped CLI stdout alphabetizes nested
manifest keys (serde_json::json!) vs. declaration order (to_json()); both
valid JSON, the stored families.jsonl uses declaration order, round-trip
unaffected. Noted as a sub-item on #73.
refs #70
Decompose spec 0045 into two iterations. Iteration 1 (engine + registry
core): Source::bounds() + window_of, M1FieldSource::bounds(), and the
aura-registry lineage module (FamilyKind / FamilyRunRecord / Family,
group_families, next_family_id, the three *_member_reports extractors,
Registry::append_family / load_family_members) with round-trip and
per-name-counter tests. Iteration 2 (CLI surface): aura mc, family-aware
aura runs (families / family [rank]), sweep/walkforward family persistence
via append_family with --name, and the window_of migration of the
manifest-window scans.
refs #70
Persist sweep / Monte-Carlo / walk-forward runs as named, linked families
(C18 lineage / C21): a FamilyRunRecord stamps each member with a shared
family_id = "{name}-{counter}" in a sibling families.jsonl store, group_families
re-derives a family as a unit, and the CLI gains aura mc + family-aware aura runs.
Producer-supplied window via Source::bounds()/window_of (the #71 firewall: no Vec
scan, no input artifact). family_id form (name + counter, not a content-hash)
resolved by the user in-session; reconciliation comment on the issue records it.
Auto-signed under the boss spec-auto-sign gate: all objective gates green
(precondition, self-review, grounding-check PASS) and a unanimous five-lens
spec-skeptic panel (criterion, grounding, scope-fork, ambiguity, plan-readiness).
refs #70
Bite-sized, placeholder-free plan for the walk-forward axis, in three tasks:
(1) extract MetricStats::from_values from McAggregate::from_draws + serde derive
(behaviour-preserving, the 7 mc tests guard); (2) the walkforward module —
WindowRoller (bounds-only iterator), walk_forward over the shared run_indexed
core, continuous OOS stitch (empty segment -> +0.0), on-demand param_stability,
+ crate exports + module-header doc-lag fix; (3) the aura walkforward CLI demo
bridging engine+registry via optimize, with serde_json for the summary line.
RED-first tests 1-11 (incl. 7b empty-segment stitch) per the spec.
refs #69
Walk-forward orchestration: a WindowRoller rolls (in-sample, out-of-sample)
splits over a time span, walk_forward runs a disjoint harness per split via the
shared run_indexed core (C1), and stitches the OOS pip-equity segments into one
continuous curve. The varying dimension is the data window (axis 3), realized
eager-agnostically (#71): the public surface carries WindowBounds + a per-window
closure, never a materialized stream Vec; C2 no-look-ahead is a pure bounds
invariant (oos.0 > is.1), checkable with zero ticks.
The in-sample optimize (axis 2) is closure-supplied, not called by the engine:
aura-engine cannot depend on aura-registry (C9), and C12 forbids baking search
policy into the primitive. The CLI bridges both crates in run_walkforward.
Param-stability shape (the load-bearing fork) resolved with the user as R2:
on-demand, not stored. WalkForwardResult stores only raw per-window outcomes +
the stitched curve; param_stability(&result) -> Vec<MetricStats> is a public
on-demand reduction over the retained per-window chosen params (sweep-precedent:
summaries computed on demand, not cached, vs McFamily's stored aggregate). A
stored summary would be recomputable from the raw windows (redundant) and would
force one statistic canonical when "stability" admits several. Recorded with
provenance in the #69 reconciliation comment. MetricStats::from_values is
extracted from McAggregate::from_draws (behaviour-preserving; the 7 mc tests
guard) so the MC aggregate and the helper share one reduction. Empty OOS segment
contributes 0.0 to the stitch offset (mirrors summarize's unwrap_or(0.0)).
Gates: Step-1.5 precondition clean (no fork silently picked), self-review clean,
grounding-check PASS (twice — re-run after each edit). Auto-sign panel escalated
on the scope-fork design lens (param-stability shape) to human sign-off; the user
resolved it (R2), so this is a user-signed spec, not boss-signed.
refs #69
Three tasks: (1) extract the disjoint-parallel core out of sweep_with_threads
into a shared `pub(crate) run_indexed<T>` (behaviour-preserving; the three
pre-existing sweep tests are the green guard), (2) add the `mc.rs` module
(McDraw/McFamily/McAggregate/MetricStats, type-7 `quantile`,
`McAggregate::from_draws`, `monte_carlo`/`monte_carlo_with_threads` driving
run_indexed) with 7 RED tests, (3) wire `mod mc;` + the five public exports into
lib.rs. RED-first; module-declaration ordering keeps mc.rs's tests dark until the
final wire-up task.
refs #68
C12 axis 4: Monte-Carlo as a sweep over seeds. `monte_carlo(base_point, seeds,
closure) -> McFamily`, the analog of SweepFamily, reusing the disjoint-parallel
executor (extracted into a shared `run_indexed` core) rather than forking a new
run loop. Each realization is a disjoint C1 unit; parallelism is across draws.
The aggregate (V1) covers all three run metrics with mean + p5/p25/p50/p75/p95,
a pure post-run reduction over the retained raw draws.
Eager-agnostic firewall (#71): the API takes seeds + a per-draw closure, never a
materialized stream Vec; seed->Source construction stays a closure-body concern.
Signed under the /boss spec auto-sign gate: objective gates green (precondition,
self-review, grounding-check PASS) and a unanimous five-lens spec-skeptic panel
(criterion, grounding, scope-fork, ambiguity, plan-readiness) returned SOUND.
The aggregate fork was settled to V1 by the user (issue #68 reconciliation
comment, verbatim "ja, wir machen v1", 2026-06-15).
refs #68
Two tasks: (1) aura-engine SplitMix64 PRNG + pub SyntheticSpec::source(seed) ->
impl Source beside VecSource, with producer RED tests; (2) aura-cli
sim_optimal_manifest gains a seed param (four callers pass 0), plus a test-only
run_sample_seeded + SeededTrace and three e2e RED tests pinning the #66
acceptance. RED-first throughout.
refs #66
A seeded source whose stream is fully determined by a u64 seed, wired so the
seed reaches RunManifest.seed at the manifest-construction site (Fork B: seed at
the data-generation edge, outside the engine graph). Contract is a producer
Fn(u64) -> impl Source, never seed -> Vec, so the Monte-Carlo family can re-seed
N times without materializing N streams. Deterministic dependency-free PRNG
(SplitMix64) for C1 bit-stability. Precondition for #68 (Monte-Carlo) and #52
(random sweep).
Auto-signed under /boss spec auto-sign: objective gates green (precondition,
self-review, grounding-check PASS) and a unanimous five-lens spec-skeptic panel
(criterion, grounding, scope-fork, ambiguity, plan-readiness all SOUND) after one
editorial-repair round.
refs #66
Architect drift review over 8b330e3..HEAD (the Source-seam cycle #71 plus the
interim #67 optimize work and refactors). No semantic or contract drift: the
seam faithfully preserves C3 (ms→epoch-ns at the one ingestion boundary), C4
(tie-by-source-index, byte-for-byte), and C7 (field-per-source SoA); the full
suite is byte-identical green and the gated streaming tests pass on real data.
Regression gate: `cargo test --workspace` green (the project declares no
dedicated regression script; the suite is the gate).
Tidied stale prose the cycle left behind (no behaviour change):
- docs/design/INDEX.md (C12): replaced the cycle-0011 "deliberate eager gap"
status note with a cycle-0041 realization note — the producer `Source` seam
+ streaming `M1FieldSource` now deliver per-source O(one-chunk) streaming;
precisely scopes what remains open (cross-*sim* Arc<[T]> window sharing,
still unbuilt until the orchestration families #66/#68/#69 consume it).
- aura-core/src/lib.rs: dropped the now-shipped "Source trait + data-server
ingestion" from the module doc's "still to come" list (aura-engine's twin
line was fixed in the cycle; aura-core's was missed).
- aura-ingest/src/lib.rs: module header now documents both coexisting paths
(eager load_m1_window/M1Columns vs lazy streaming M1FieldSource).
- aura-registry/src/lib.rs: rank_by/optimize doc comments linked to the
private `metric_cmp` (a public→private intra-doc-link warning from the #67
commit); demoted to plain code spans, so `cargo doc --workspace` is now
warning-clean.
Decomposes the boss-signed spec into four tasks: (1) the `Source` trait +
`VecSource` adapter in aura-engine; (2) the atomic `Harness::run` re-type to
`Vec<Box<dyn Source>>` with all 53 call sites threaded behaviour-preserving;
(3) the streaming `M1FieldSource` over a data-server window; (4) a gated
streaming e2e + the measured residency predicate.
refs #71
Re-types the engine's ingestion input from a materialized
Vec<Vec<(Timestamp, Scalar)>> into a `Source` producer trait the k-way
merge drives by peek/next, and proves it against a real, lazily-streamed
data-server window in the same cycle — closing the cycle-0011 deliberate
eager-materialization gap (C12 Arc<[T]> cross-sim sharing). A VecSource
adapter keeps every existing run call site byte-identical
(behaviour-preserving); an M1FieldSource streams data-server chunks with
O(one-chunk) residency, independent of window length.
Foundational cycle of the World-II milestone; #66/#68/#69 inherit the
seam. Signed under the boss auto-sign gate: precondition clean, grounding
PASS on the final bytes, unanimous five-lens spec-skeptic panel (two
editorial ambiguity rounds on the residency predicate repaired to a
measured per-pull CHUNK_SIZE ceiling).
refs #71
"Compilat" (German "Kompilat") was a coined noun for the product of the
bootstrap compilation — neither English nor a natural fit. The runtime
artifact already has a code identifier for exactly this thing: the
`FlatGraph` struct (harness.rs). Replace the coinage with that identifier:
- prose mentions -> "flat graph" (mirrors the type, reads plainly)
- definitional anchors -> `FlatGraph` (C11, C23, the running-graph line)
- `render_compilat` -> `render_flat_graph` (historical render symbol;
keeps the `render_blueprint` / `render_flat_graph`
source-vs-product pairing)
- `intra-compilat` -> `intra-graph`
- `from_compilat` (test) -> `from_flat`
"compilation" / "re-compilation" / "bootstrap-as-compilation" (the process,
ordinary English) are deliberately left untouched. Behaviour-preserving:
only comments, design ledger, specs/plans, one test-local variable and its
assert messages change. Full workspace test suite green; clippy clean.
A graph that leaves an interior input slot unconnected, or wires one slot from more
than one producer, is now a compile-time error instead of a silently-wrong run.
Until now an unwired interior slot was accepted and bootstrapped to an empty column
(harness.rs:137-148), so a forgotten connection ran a deterministic but wrong
backtest; two producers into one slot — ill-formed, since a slot holds one column —
was likewise uncompiled-against.
A single name-free check, `check_ports_connected`, is added to `validate_wiring`
after the existing index/kind checks and before the nested-composite recursion. It
counts coverage of each (interior-node, slot) uniformly across interior edges and
role targets, and rejects zero coverage (new `CompileError::UnconnectedPort`) or >1
(new `CompileError::DoubleWiredPort`). Because `validate_wiring` is called once from
`compile_with_params` and recurses, both the raw `Composite::new` path and the
`GraphBuilder::build()` path inherit it at every nesting level with no builder-side
code (mirrors `check_param_namespace_injective`, the param-side sibling). A
composite's own open input roles (source: None) are coverage providers — the
wired-by-enclosing boundary, the root already guarded by UnboundRootRole — not
consumers. Index-based and name-free: nothing reaches the compilat, so C23 holds.
Supporting fix: the check computes `signature()` on every interior node before the
recursion validates that node's interior, so `derive_signature` and
`interior_slot_kind` are made bounds-total (a placeholder kind for an out-of-range
OutField/target, never a panic; the real fault is still reported by validate_wiring's
guarded OutputPortOutOfRange/BadInteriorIndex). This latent panic was exposed by the
new check, not introduced by it.
Test maintenance (blast radius enumerated empirically, not hand-traced): four
existing tests relied incidentally on under-wiring being accepted — three negative
tests get a covering root role so their intended deep fault still surfaces via the
recursion, and one param-order nesting test is fully wired (param order unchanged).
Six new tests: five raw-surface (unwired, double-wire via edge+role and edge+edge,
nested, open-role boundary) and one builder-surface forgotten-leg (proving the
GraphBuilder inherits the check).
Narrowed scope of #65: the original "promote names to load-bearing wiring keys /
close the exposure-price swap structurally" goal was dropped — #64 already moved
authoring to handles + visible port names, so the residual same-kind swap is a
valid-but-wrong name choice no structural check catches, and a structural fix would
push domain semantics into the domain-free engine (C10/C7). This ships the name-free
half that stands on its own: wiring completeness, which catches forgotten and doubled
connections.
Verified: cargo test --workspace 231 passed / 0 failed; cargo clippy --workspace
--all-targets -- -D warnings clean.
closes#65
Implement-time discovery: the new check (and the existing edge/role checks) compute
signature() on every interior node before the recursion validates that node's
interior. For a structurally-invalid composite (an out-of-range OutField or role
target), derive_signature indexed unguarded and PANICKED — exposed by
output_port_out_of_range_rejected the moment its covering root role forces
c.signature(). Added Task 1 Step 2b: make derive_signature and interior_slot_kind
bounds-total (a placeholder kind for an invalid index; the real fault is still
reported by validate_wiring's guarded OutputPortOutOfRange / BadInteriorIndex). Fix
verified empirically before re-dispatch. Not a design change to the check itself.
refs #65
Seven-task plan for the wiring-totality check (parent spec
docs/specs/0040-wiring-totality-check.md, 07b1ae1 + blast-radius correction
69b16f7): add the two CompileError variants + check_ports_connected + its
validate_wiring call site (T1); re-wire the three under-wired negative tests with a
covering root role (T2); fully wire the param-order nesting test (T3); add five
raw-surface tests — unwired, double-wire via edge+role and edge+edge, nested,
open-role boundary (T4); add the builder-surface forgotten-leg test (T5); land the
C8 ledger realization note (T6); full-suite + clippy gate (T7).
The blast radius (T1's expected-4-failures gate, T2/T3's fixes) was enumerated
empirically by a throwaway probe, not a hand-trace, after the hand-trace under-counted
it by one (param_space_mirrors_..._under_nesting).
refs #65
The boss-signed 0040 spec's blast-radius section claimed "exactly three" flipping
tests, named from a hand-trace. A throwaway probe (the check implemented, full
workspace suite run, then discarded) found FOUR: the three named negative tests plus
`param_space_mirrors_compiled_flat_node_param_order_under_nesting`, a compile-success
param-order test that today compiles a deliberately under-wired nested graph (LinComb's
two term inputs never wired) just to read its flat param order. The new check rejects
it where it currently returns Ok.
Correction is editorial, not a design change: the check itself is unchanged. The
fourth test's fix differs from the three negative tests — it needs the contrived graph
fully wired (fan fast_slow's output into both LinComb terms + a covering root role),
not just a covering role — so the §Blast radius and §Acceptance criteria now enumerate
all four with their distinct fixes. Verified empirically (119 passed, 4 failed; every
other crate green), the strongest possible grounding for the count.
refs #65
Reject graphs with an unwired or double-wired interior input port. A new
name-free structural check `check_ports_connected` in `validate_wiring`
(crates/aura-engine/src/blueprint.rs) requires every interior node's every
declared input slot to be covered by exactly one wiring act — one `Edge{to,slot}`
or one `Role` `Target{node,slot}`, counted uniformly. Zero coverage is the new
`CompileError::UnconnectedPort` (a forgotten connection that today bootstraps a
silent empty column, harness.rs:137-148); more than one is `DoubleWiredPort` (a
slot holds one column). Run once at the existing compile boundary and recursive
per nesting level, so both the raw `Composite::new` and the `GraphBuilder::build`
surfaces inherit it. Index-based, name-free — the compilat is untouched (C23).
Narrowed scope of #65: the original "promote names to load-bearing wiring keys /
close the exposure-price swap structurally" goal is dropped (see the reconciliation
comment on the issue) — #64 already moved authoring to handles + visible port
names, so the residual same-kind swap is a valid-but-wrong name choice no
structural check catches, and a structural fix would push domain semantics into
the domain-free engine (C10/C7). This cycle ships only the name-free half that
stands on its own.
Auto-signed under /boss: precondition gate clean (no fork silently picked),
self-review + parse-trace clean (no spec-validation parser declared -> documented
no-op), grounding-check PASS on the final bytes (one BLOCK on an incomplete
blast-radius survey, repaired forward by naming the three flipping negative tests),
and a unanimous five-lens spec-skeptic panel (criterion, grounding, scope-fork,
ambiguity, plan-readiness all SOUND).
refs #65
Task-by-task plan for the typed-handle GraphBuilder: the From<Composite> lift,
the builder module (types + accumulators + the fallible build() resolver), the
lib.rs wiring, and the test suite (structural parity, harness FlatGraph parity,
the five BuildError variants, #21 legibility, coexistence + clippy gate).
refs #64
Add a typed-handle GraphBuilder authoring surface for blueprint topology:
node references become Copy NodeHandle values, ports/fields resolve by name
against the existing PortSpec.name/FieldSpec.name at a single fallible build()
terminal, lowering to the unchanged index-wired Composite (C23 holds by
construction — names never reach the compilat, mirroring param-name resolution).
Spec auto-signed under /boss: all objective gates green (precondition,
self-review, grounding-check PASS) and a unanimous five-lens spec-skeptic panel
(criterion, grounding, scope-fork, ambiguity, plan-readiness all SOUND).
The structural close of the SimBroker exposure/price swap (#21) is explicitly
out of scope, tracked as #65.
refs #64
Collapse the three verbatim-duplicated SMA-cross harness test fixtures
(synthetic_prices, sma_cross, composite_sma_cross_harness) in aura-engine's
blueprint.rs and sweep.rs #[cfg(test)] modules into one shared crate-root
#[cfg(test)] mod test_fixtures. Behaviour-preserving, test-only.
Auto-signed under /boss: all objective gates green (precondition, self-review,
grounding-check PASS) and a unanimous five-lens spec-skeptic panel returned
SOUND.
refs #53
Six bite-sized tasks for spec 0037: aura-core records a BoundParam (original slot
position) in bind + a bound_params() accessor; aura-engine emits a conditional
"bound" field via scalar_str; graph-viewer.js merges free+bound params into slot
order and renders the bound slot dimmed as name=value; re-capture the sample-model
fixture; a new headless render guard (trailing + middle bind); full-workspace gate.
refs #63
Surface a bind-bound param in the graph model and the `aura graph` viewer
signature instead of dropping it. A node built with `.bind(slot, value)` keeps
the slot off the tunable surface (param_space / sweep axes — unchanged, correct)
but now ANNOTATES it in the render: all slots shown, the bound one as `name=value`,
dimmed. The cycle-0036 `blend` renders `LinComb[weights[0], weights[1], weights[2]=0.5]`.
Structurally a twin of cycle 0035's instance-name thread: a conditional field
(here `"bound"`) threaded engine -> model -> viewer, plus goldens and a headless
render guard. Render/debug surface only — dropped at lowering (C23), model stays
deterministic (C14).
Auto-signed under the /boss spec auto-sign gate: precondition clean (the one open
notation decision was settled with the user in-session and recorded as a
provenance-bearing reconciliation comment on the issue), self-review + parse-trace
clean, grounding-check PASS, and a unanimous five-lens spec-skeptic panel.
refs #63
Five tasks for the sample-showcase cycle: (1) the enriched blueprint source —
LinComb import, the `signals` composite (trend = sma_cross, momentum = macd, blend
= LinComb(3) with weights[2] bound), the node-0 swap in sample_blueprint_with_sinks,
the re-pathed sweep_family axes + a showcase_prices warm-up stream; (2) re-path the
two in-crate tests (bootstrap-runs-drains, sweep odometer); (3) re-capture the
sample-model.json render fixture; (4) a new headless depth-2 nesting guard
(viewer_nested_depth.mjs + .rs); (5) the workspace build/test/clippy gate.
refs #62
Enrich the `aura graph` / `aura sweep` sample blueprint into a "trend + momentum,
weighted blend" strategy that showcases four authoring/viewer capabilities the
single-level sample left dark: multiply-nested composites, a multi-param node
inside a composite, a multi-output (MACD-like) node, and bind() (a param fixed as
a structural constant, dropped from the sweep surface). Pure authoring over
already-shipped primitives — no engine semantics or viewer change; reuses the
existing sma_cross/macd builders.
Auto-signed under /boss (spec auto-sign enabled): objective gates green
(precondition clean, self-review clean, grounding-check PASS) and a unanimous
five-lens spec-skeptic panel (criterion, grounding, scope-fork, ambiguity,
plan-readiness) returned SOUND. Design provenance recorded on #62.
refs #62
Brummel