The visual face pivots from the ledger's egui-native, in-process
zero-copy direction to a web frontend served from disk-persisted
recorder traces: the engine writes recorded traces to disk, a browser
charts them. Settled in an in-context design discussion (provenance in
issue #101): raw per-tap trace persistence + serve-time join_on_ts
alignment + a static self-contained HTML chart (uPlot). The engine
stays headless (C14) — arguably more strictly, since a browser reading
a serialized file is further from the hot path than egui reaching into
the live SoA columns. Amends the C14 visual-face note and the open
"Playground & World UI surface" thread; the C22 realization note lands
when the seam ships.
refs #101
Milestone-close fieldtest for "Runway — real-data ergonomics & honesty
hardening". Four curated end-to-end scenarios derived top-down from the
milestone promise, run as a downstream consumer against real GER40/EURUSD M1
bars (public interface only; no crates/*/src read):
1. per-instrument pip honesty (#22) — GER40 index pip 1.0 AND EURUSD forex
pip 0.0001 both honest end-to-end; AAPL.US/FOOBAR un-vetted refused
exit 2, no stdout leak
2. real GER40 M1 open via open_ohlc from aura-ingest ALONE (#80/#81/#92) —
no data-server dep, 53173 real bars, C4 O/H/L/C order, ts-window inverse
3. multi-tap ts-join (#93) — two taps at different cadences, 27744 rows,
0 by-ts mismatches; warm-up None rows where a positional zip would corrupt
4. run-registry coherence (#73/#82) — list/rank retired exit 2, families/
family live, two-store Registry read-back + rustdoc
Findings: 4 working, 1 friction, 1 spec_gap.
- friction: Cell exposes only from_i64/from_f64/... while the side-by-side
Scalar is a tagged enum with public I64(..)/F64(..) variants. The natural
param-authoring entry point is Scalar (Cell is the hot-path buffer carrier),
so the wrong reach is a self-correcting compile snag — ergonomic only.
Filed as a follow-up idea.
- spec_gap: unknown family id exits 0 empty — ratified into the C18 note in
the preceding commit (treat-as-empty contract).
Gate verdict: GREEN. All seven axes mapped to a covering scenario and exercised
on real bars; both finding classifications and the one material coverage gap
were adversarially re-verified. That gap — #22's FX-vetted pip face, untested by
the original scenario set — was probed (EURUSD -> sim-optimal(pip_size=0.0001),
FX-scaled, not index-mis-scaled) and folded into scenario 1, so the pip promise
is demonstrated on both positive faces. Every scenario re-run independently on
HEAD before commit.
The Runway milestone fieldtest surfaced a spec_gap: `aura runs family <id>`
with an unknown-but-well-formed id exits 0 with empty output, and neither the
ledger nor the glossary constrained that case. The behaviour is already the
documented, tested CLI contract (treat-as-empty, mirroring Registry::load's
missing-store = Ok(empty)); this lifts that contract to the durable C18 note
so a consumer reasoning from the spec can derive it, and records that tightening
to a non-zero "no such family" exit (typo-safety) stays an available future UX
choice rather than a current contract.
Ratifies the spec_gap from docs/specs/0055-fieldtest-runway-real-data-seam.md.
The milestone-close architect flagged that the runs-retire note over-claimed
"retained because used": after dropping runs list/rank, Registry::append/load (the
flat-store half) have no in-tree caller — only rank_by/optimize keep live
consumers. Note now distinguishes the live half from the public-but-uncalled half
(a latent dead-code surface for a later sweep). Doc-only.
Since cycle 0045 the flat runs.jsonl store has had no producer (sweep/mc/
walkforward persist to the family store; `aura run` does not persist), so
`aura runs list` / `runs rank` read a store nothing populates — a live dead-end
surface, the honesty gap the Runway milestone closes.
Resolve the C18 deferred fork (INDEX.md) as (b) RETIRE: drop the two CLI commands
+ their dispatch arms, usage fragments, and the now-unused load_runs_or_exit.
Families (sweep/mc/walkforward, C21) subsume standalone over-time comparison. The
aura-registry flat lib API (append/load/rank_by/optimize) is RETAINED — optimize
backs walk-forward's in-sample step, rank_by backs `runs family ... rank`.
`runs families` / `runs family` are untouched.
Decision recorded on #73 (user-directed (b), 2026-06-18). Cosmetic sub-item
(json! family-wrapper -> to_json() stdout key-order unification) deferred to a
follow-up: json! routes the embedded report through serde_json::Value, which
re-alphabetizes keys, so it needs manual JSON construction, not a trivial swap.
RED-first: the retire test asserted `runs list`/`rank` now exit 2 (was exit 0).
Verified: cargo build --workspace, clippy --all-targets -D warnings, and
cargo test --workspace all green.
closes#73
Audit (architect drift review + cargo test --workspace) clean: code matches the
spec, the engine stayed domain-free (no instrument identity in Ctx / the hot
path), InstrumentSpec sits beside the hot path per C7/C15. No regression scripts
configured (no-op gate); the full suite is green.
Ratify: the architect flagged one ledger-realization gap — C10/C15 described
per-instrument pip as standing design intent but carried no realization note for
the now-shipped channel. Added dated realization notes to C10 (the per-instrument
pip channel + the refuse-don't-guess honesty rule) and C15 (instrument specs as
metadata, first realized by InstrumentSpec/instrument_spec).
Retire the ephemeral cycle artefacts (spec 0054, plan 0054) per the active-cycle
lifecycle. Deferred follow-up: #98 (route the runnable GER40 examples through the
lookup).
refs #22
Cycle 0053 (multi-tap trace join on timestamp, #93) closes drift-clean.
Audit (architect drift review over 066638e..HEAD against docs/design + the
CLAUDE.md domain invariants):
- C8/C18 honoured: join_on_ts is a post-run reduction over recorded
(Timestamp, Vec<Scalar>) sink streams, a true sibling of summarize/f64_field
in the report layer — correct architectural home.
- C3 not violated: the join runs post-run over already-recorded output, not
in-graph; no as-of join smuggled into the DAG. C1 ts-uniqueness is a
documented precondition, not a runtime check — consistent with how the
engine leaves wiring-time invariants to contract.
- Option-per-side keeps the engine honest (reports presence; the consumer owns
the 0.0/-1/false defaults). The hand-rolled *_by_ts HashMaps are fully gone;
the two RED unit tests pin the #93 cardinality-misalignment + spine-drop.
- No drift, no debt, no ledger entry needs updating. Recommendation: carry on.
Regression gate: no dedicated regression scripts configured (optional per the
conventions); build/test/clippy --workspace --all-targets is the gate and ran
green — the gated GER40 byte-identity tests executed against the local archive
(not skipped), proving drain_trace's trace is byte-preserved across the
refactor.
Per the artifact lifecycle (committed while the cycle is live, git rm at cycle
close), the ephemeral spec and plan are retired here; their durable rationale
lives in the implementation commits 74324d1 / 35c5adc and the git history.
Bite-sized execution plan for spec 0053: Task 1 the join_on_ts helper +
JoinedRow + two RED unit tests in aura-engine report.rs (re-exported from
lib.rs); Task 2 the behaviour-preserving drain_trace refactor onto the helper;
Task 3 the Recorder prose cadence/join note. Final gate is workspace
build/test/clippy.
refs #93
Ratified design for a reusable, spine-anchored join_on_ts helper in
aura-engine that fuses multi-tap Recorder traces on the recorded timestamp
(Option-per-side: the engine reports presence, the consumer interprets
absence). Refactors drain_trace onto it and documents the cadence rule on
the Recorder API. Resolves the #93 zip-by-index panic.
refs #93
Cycle 0052 (real-data source-open seam, Runway-1) closes drift-clean.
Audit (architect drift review over aedaa5d..HEAD against docs/design + the
CLAUDE.md domain invariants):
- C3 (one ms<->ns crossing) strengthened — the inverse is seam-private and the
only consumer-side divide left; every consumer ns_to_ms / unix_ms_to_epoch_ns
wrap is gone.
- C4 (O/H/L/C merge tie-break order) now in exactly one vetted home (open_ohlc);
the hand-spelled four-field loops are gone.
- C12 (one shared Arc<DataServer>/cache) preserved; C7 surface additive.
- No drift, no debt, no ledger entry needs updating. Recommendation: carry on.
Regression gate: no dedicated regression scripts configured (optional per the
conventions); build/test/clippy --workspace is the gate and ran green — the gated
open_ohlc_seam behaviour-preservation test executed against the local archive
(not skipped), proving byte-identity vs the ms-path.
Per the artifact lifecycle (committed while the cycle is live, git rm at cycle
close), the ephemeral spec and plan are retired here; their durable rationale
lives in the implementation commit 8c9a1b4 and the git history.
Consolidate aura-ingest's OHLC source-open surface onto the engine-native
epoch-ns Timestamp currency: a private seam-owned inverse, a Timestamp-window
opener, a canonical OHLC bundle opener (fixed C4 O/H/L/C order), and a
DataServer/DEFAULT_DATA_PATH re-export + default_data_server convenience — so a
real-data source builds from aura-ingest alone, with the one ms<->ns crossing
owned by the seam. Behaviour-preserving (byte-identical RunReports).
Grounded against the tree via the ground-spec-0052 workflow (25/28 assertions
confirmed; compare.rs compiler-invisible site + AC1 scope fix applied).
refs #80, #81, #92
Per-cycle specs/plans are step-scoped: every file maps to already-shipped
code, their snippets drift hard against current APIs (InputSpec->PortSpec,
Sim->Harness, schema() removed, C16 zero-dep reversed, ascii-dag retired),
and a stale spec read as an API reference misleads agents into non-compiling
code. No durable knowledge lost — rationale lives in the ledger + code
rustdoc, history in git + the tracker.
- gitignore docs/specs/ and docs/plans/
- delete all 117 existing specs/plans (recoverable via git history)
- CLAUDE.md: specs/plans are local-only ephemeral artifacts
The C12 realization and the 0041 spec claimed streaming residency is O(one chunk) at the process level ("a 20-year window streams in the same memory as a one-day one"). That holds only for the aura source ring (M1FieldSource::resident_records(), bounded by one chunk and correctly tested); whole-process RSS grows O(records-touched) because data-server's FileCache retains each window's parsed chunks (~56 B/record) read-only for the pass. Reproduced: 6 MiB @ 1mo -> 173 MiB @ ~10y, linear.
That retention is the replay-many optimization (load-once; close+volume share one parse via the field-agnostic FileKey), not a leak, and no always-on eviction can bound a single forward pass without regressing it. This narrows the docs/comments to the true per-source-ring property and names the FileCache per-window retention as the real, replay-amortized process cost. The streaming_seam assert is unchanged (it was always the ring bound; only its labelling confused ring vs process). The deferred opt-in non-retaining streaming read path is parked as Brummel/data-server#2.
closes#95
Add a C20 realization recording the resolution of the GER40 deep-dive's #94
finding: a hand-wired FlatGraph is not a shippable strategy (no param_space, so
World-opaque); the canonical form is the Composite blueprint, the FlatGraph its
compiled substrate (C23). Pins the bar-period-as-structural-construction-arg
decision (spec 0051) to the structural-axis-vs-tuning-param split.
refs #94
Milestone spec: ship the GER40 session-breakout as a Composite blueprint so the
same strategy runs across backtest / sweep / walk_forward / compare without
re-authoring (the #94 fieldtest friction). Records the design decision that the
bar period is a structural axis (a construction arg binding both Resample and
Session, so a sweep cannot desync them — #96) and that the canonical shippable
strategy form is a Composite (FlatGraph is its compiled substrate). Authoring-
side; no engine-core change (the existing bind() + GraphBuilder + param_space()
layer already suffices).
refs #94, #96, #97
Public-API research deep-dive driving the just-shipped GER40 real-data
session-breakout demo (5b5f034) through the escalating sequence a serious
researcher attempts next: scale to multi-year, walk-forward, structural-param
sweep, multi-instrument compare. Standalone consumer crate (path-deps only;
public interface = ledger + glossary + example corpus + cargo doc; no
crates/*/src read). All runs against the real /mnt/tickdata/Pepperstone archive.
Findings: 3 bugs, 1 friction, 3 spec_gap, 1 working. Triaged to the tracker:
- spec_gap: the flagship breakout ships as a raw FlatGraph the World API cannot
consume (sweep / walk_forward / compare all forced a hand re-author to a
Composite, which reproduced the FlatGraph numbers EXACTLY) — refs #94.
- bug: process residency is O(window) (6->173 MiB across 1mo->10y) despite the
documented O(one-chunk) claim; the aura ring stays <=1024 but data-server
retention scales, and streaming_seam.rs only measures the ring — refs #95.
- bug: the bar-period is one knob split across Resample (tunable param) and
Session (baked) and desyncs silently to 0.0 pips; Delay.lag leaks into the
default param_space (the C34 deform-vs-tune discriminator) — refs #96.
- spec_gap: the walk_forward empty-space idiom for param-free strategies is
undefined — refs #97.
- spec_gap (no per-instrument pip registry) and friction (one-directional
ns->ms conversion) re-confirm the known #22 and #80.
- working: C1 determinism over 7y / 148k bars, the lazy streaming source, and
the sweep/WFO cores held under real multi-year load; the Composite re-author
was behaviour-preserving (C23).
Scratch consumer crate + run transcripts committed under fieldtests/ per the
fieldtest convention (matching cycle-0049). The crate path-deps the engine and
reads the real archive; its bins are the repros cited in the issues above.
Milestone 'Strategy node vocabulary I — temporal, logic, resample & session':
the 7 aura-std nodes (EqConst, Gt, And, Delay, Latch, Resample, Session) plus a
synthetic e2e fixture, driven by the GER40 15m session-breakout. Records the
design pass's cross-cutting decisions: close-instant timestamp convention,
Latch emits f64 (Exposure dropped from the DAG), the EqConst i64->bool gate,
the Resample-only Barrier(0) firing regime, and the build order. refs #84-#91.
Milestone-close gate for "The World, part II — orchestration families".
Four end-to-end scenarios derived top-down from the milestone promise,
each built from HEAD and run as a downstream consumer through the public
API + CLI only (no crates/*/src read):
- mw_3 (headline): real EURUSD-M1 walk-forward composing all four axes
(grid sweep + optimize in-sample, OOS stitch) over the M1FieldSource
seam + lineage — 5 windows, 145 200-pt stitched curve, C2 held.
- mw_2: monte_carlo over 128 seeds, reproducible McFamily (C1) + aggregate.
- mw_1: random sweep -> optimize -> 64-member lineage round-trip.
- mw_4: family-lineage CLI transcript.
Findings: 4 working (promise empirically delivered), 1 bug, 3 friction.
- bug F5: `aura runs list`/`rank` cannot read a pre-0047 runs.jsonl — the
cycle-0047 typed-Scalar params migration (86746e3) left the persisted
flat store unmigrated; load() fails at the first bare float (+ exits 0).
- friction F6: no epoch_ns_to_unix_ms inverse for real-data walk-forward.
- friction F7: DataServer/DEFAULT_DATA_PATH not re-exported by aura-ingest.
- friction F8: family store is directory-keyed, undocumented on Registry::open.
Spec: docs/specs/fieldtest-milestone-the-world-orchestration-families.md
Public-API field test of the random param-sweep surface, from a standalone
downstream-consumer crate (path-deps only; the public interface = ledger +
glossary + spec 0049 + cargo doc rustdoc; no crates/*/src read). Four bins,
each built from HEAD and run: continuous tuning (200-point random tune ranked
by total_pips), the typed validation gate (all five reachable SweepError
variants pre-run), reproducibility + seed-sensitivity + the full i64::MIN..=MAX
sampler edge, and Space-trait interchangeability (one tune_and_rank<S: Space>
over both GridSpace and RandomSpace).
Findings: 0 bugs, 4 working, 2 spec_gap, 1 friction. The four working findings
confirm the cycle's acceptance criterion empirically — the headline tune reads
as the code a researcher would write, the gate is precise and fires before any
run, the C1 reproducibility promise is checkable in one line, and the Space
trait delivers one-consumer/both-enumerations.
Triage of the actionable findings:
- friction (no named-axis builder for RandomSpace — positional Vec<ParamRange>
must align with param_space() by hand, and a same-kind transposition passes
validation silently): filed as a feature for a future cycle, refs #79
(a RandomBinder sibling to the grid's SweepBinder).
- spec_gap (SweepError rustdoc summary named only GridSpace): fixed inline in a
follow-up doc commit.
- spec_gap (NonNumericRange / Bool-slot ranges unreachable with the shipped
aura-std node roster — no node declares a Bool/Timestamp knob): RATIFIED as
intentional. The variant is a forward-looking structural guard for the C16
"author your own node" path (a Bool/Timestamp param-slot a custom node may
declare); its current untriggerability with the standard roster is expected,
not drift.
refs #79
Architect drift review over 3de00e2..HEAD (the 0049 spec/plan/feat plus the
two intervening refactors that had not been audited: aura-ingest M1 transpose
6390093, aura-std SMA perf 67c1f51). Verdict: feature clean — C1 determinism
preserved (RandomSpace points seed-determined before any run; grid path
behaviour-preserving via trait-forwards-to-inherent), the #52/#71 source-seam
firewall honoured in code (sweep stays &S: Space + Fn(&[Cell]) -> RunReport, no
Source type enters the sweep layer; SplitMix64 a code-path-disjoint instance),
and both intervening refactors behaviour-preserving (pinned by
incremental_matches_full_resum_within_tolerance and
chunked_accumulation_equals_single_transpose).
Two low-grade doc-debt items found and fixed inline (doc-only, behaviour-
preserving):
- sweep.rs module doc named only the grid axis; refreshed to name RandomSpace
+ the Space trait the module now also owns.
- C12 ledger had no realization note for axis-1 (param-sweep); added one
recording the grid (0028) + random (0049) landing and the Space-trait
unification.
Regression gate: the project configures no regression scripts, so the test
suite + lint are the gate. Independently verified: cargo test --workspace green
(incl. the 17 new engine tests + 6 public-API E2E), cargo clippy --workspace
--all-targets -- -D warnings clean.
Drift-clean, not a milestone close (no milestone fieldtest run this cycle).
Five bite-sized tasks for the RandomSpace cut: (1) the Space trait +
behaviour-preserving generalization of sweep/sweep_with_threads, (2) typed
ParamRange, (3) RandomSpace::new validation + three SweepError variants,
(4) SplitMix64 -> pub(crate) + the seeded sampler, (5) public exports +
random-sweep integration. Each task is RED-first with single-substring test
filters; the grid suite is the C1 behaviour-preservation regression guard.
refs #52
RandomSpace + a Space trait + typed ParamRange: the random half of the
C12.1 param-sweep axis (grid shipped in 0028). N seeded uniform draws over
declared continuous ranges, fed to the same enumeration-agnostic sweep core.
Param-only sweep signature and a code-path-disjoint SplitMix64 instance keep
the World-II source-seam firewall (refs #71) intact.
Auto-signed under /boss spec auto-sign: fresh grounding-check PASS and a
unanimous five-lens spec-skeptic panel (criterion, grounding, scope-fork,
ambiguity, plan-readiness all SOUND).
refs #52
Two-task execution plan for spec 0048: Task 1 lands the whole aura-engine/walkforward.rs cut atomically (struct shapes, walk_forward seam, param_stability rewrite, scalar_as_f64 deletion, fixture migration, new Bool-slot test) gated on a crate-only `cargo test -p aura-engine`; Task 2 threads the aura-cli caller and runs the workspace gates. Keeps a windows.is_empty() guard in param_stability to preserve the documented empty-on-no-windows contract (C1).
refs #75
Settled-design spec for the walk-forward param-plane cut: WalkForwardResult gains space: Vec<ParamSpec>, WindowRun.chosen_params Vec<Scalar> -> Vec<Cell>, walk_forward takes the space by value (Fn-closure generic untouched), and param_stability schema-checks a per-slot coercer (Bool -> 0/1, Timestamp unreachable per C20) before a type-blind reduction, deleting scalar_as_f64 + its per-value unreachable.
Behaviour-preserving (C1): param_stability yields identical MetricStats over the existing fixtures. Overturns 0047's Fork A (chosen_params stays Vec<Scalar>) — the WF structs carry no serde derives, so the serializability defense is moot; the substantive case is C7 purity + SweepFamily symmetry. grounding-check PASS.
refs #75
Scalar was `struct { kind: ScalarKind, cell: Cell }` — "a Cell wearing a kind hat." The recorded reason for that shape was migration ease, which is not a design rationale (CLAUDE.md: rationale != effort), so the struct had no substantive defense. Redefine it as the native tagged union it conceptually is:
enum Scalar { I64(i64), F64(f64), Bool(bool), Timestamp(Timestamp) }
This is substantively better on four axes: kind/bits skew becomes unrepresentable (illegal states gone); accessors panic loudly on the wrong variant instead of a release-mode silent bit reinterpret; PartialEq derives the documented IEEE-754 / cross-kind value semantics (the hand-roll, needed only because the struct inherited Cell's bitwise compare, is gone); and serde is a plain derive emitting the externally-tagged wire form ({"I64":10}/{"F64":2.5}) — the private ScalarRepr shadow enum that motivated this was never needed. It is also C7-honest: erased-on-the-hot-path (Cell) and self-describing-at-the-edge (Scalar) are two disjoint types bridged by explicit conversion.
The whole public API is preserved (Scalar's fields were private), so call sites do not churn: the enum change is contained to scalar.rs, where cell() now encodes and from_cell() decodes per kind. Cell and ScalarKind are untouched.
With Scalar serializable, lift RunManifest.params from Vec<(String, f64)> to Vec<(String, Scalar)>: the param's kind (an i64 length vs an f64 scale) now survives into the C18 record (runs.jsonl) and the CLI JSON instead of collapsing to f64. scalar_as_param_f64 is deleted; sim_optimal_manifest passes typed params through. This is a deliberate wire-shape change — params now render as tagged scalars; the JSON-asserting tests are updated to the new shape on purpose.
Hand-authored manifest params across the CLI's single-run/mc/macd sites use honest kinds (lengths -> i64, scales -> f64) so they match the sweep path (which already derives correct kinds via zip_params); their in-binary JSON assertions are re-tagged accordingly.
Walk-forward fork resolves with no code change: WindowRun.chosen_params stays Vec<Scalar> (the serializable record carrier), reduced to f64 only at the param_stability statistic boundary (scalar_as_f64 retained). Glossary 'cell' entry updated to describe Scalar as the disjoint tagged union, not 'a cell plus its kind tag'.
Gates: build --all-targets, test --workspace (incl. new scalar_serde_round_trips), clippy -D warnings, doc --no-deps — all clean.
cycle 0047 tidy. Architect drift review (b188773..82635fa, against the
design ledger C7/C8/C1) found the carrier swap drift-clean:
- Node::eval -> Option<&[Cell]>, all 8 aura-std out-buffers [Cell; N], the
inter-node forward via the branch-free push_cell (harness.rs:426); Scalar
survives only on the three keep-set boundaries (param plane,
AnyColumn::get, source ingestion at harness.rs:402). The convert/keep
partition and the three-role test rule are honoured site-for-site.
- The dual API is coherent: fallible push (ingestion + kind-guard tests)
vs infallible push_cell (bootstrap-verified inter-node forward) are each
load-bearing and tested. No dead code, no C7 violation (no tag on the hot
path; scratch: Vec<Cell> reused via clear(), no per-cycle alloc).
- Both C7 realization notes + the C8 prose accurately describe the landed
state. 285 tests green; build/clippy/test/doc clean.
Tidy fix (pre-existing, surfaced by the review):
- docs/glossary.md `node` entry said a node implements `schema()`, but
Node::schema() was removed in cycle 0024 (the signature is declared
pre-build on PrimitiveBuilder). Corrected to `lookbacks()` + `eval(ctx)`,
the methods the Node trait actually carries. Record-reality.
Motivation
----------
The C7 realization note (cd3d1ca / 049f22a) deferred this step: with
Scalar split into { kind, cell }, the tag-free Cell could become the
hot-path carrier, but eval and the edges still flowed the fatter
self-describing Scalar. This lands the swap.
Change
------
- Node::eval now returns Option<&[Cell]>; every node out-buffer is
[Cell; N]. The inter-node forward copies the row into a Vec<Cell> and
pushes each field via a new branch-free, infallible AnyColumn::push_cell
— the edge from_field->slot kind match is verified once at bootstrap
(C7), so the runtime push needs no per-value kind check. It only removes
a Result the forward path already .expect()ed.
- Scalar stays on the self-describing dynamic boundaries: the param plane
(build / bind / compile_with_params / sweep points / RunManifest),
AnyColumn::get (the type-erased read for sinks / serde), and source
ingestion (Source::next, the heterogeneous C3 merge). The fallible
AnyColumn::push is retained for ingestion + the kind-guard tests.
Trade
-----
The per-value runtime kind check on node OUTPUT is removed: a node that
builds a wrong-kind cell pushes raw bits with no panic. This is the same
authoring-bug class C8 already leaves to a debug_assert (output width);
node-output-kind correctness is the node's declared FieldSpec contract,
caught by each node's own eval test. Cross-node kind agreement still rests
on the bootstrap kind-check (bootstrap_rejects_*_kind_mismatch, green).
Tests
-----
Behaviour-preserving (C1): 285 green, 0 red. The only test-expectation
edits are carrier re-spellings (Scalar::f64(x) -> Cell::from_f64(x)) of
identical values; test column writes and out-of-graph channel rows stay
Scalar (the spec's three-role rule). Adds an AnyColumn::push_cell
round-trip test plus a cross-kind end-to-end forward test (a mixed
f64/i64 record whose bit patterns diverge across kinds — pins the i64
forward path the f64-only tests cannot catch). Ledger C7/C8 notes updated.
Gates: cargo build / test / clippy --all-targets -D warnings / doc
--workspace all clean.
closes#74
Four tasks for the behaviour-preserving carrier swap: (1) add the
branch-free AnyColumn::push_cell + round-trip test (additive); (2) the
atomic lib-code flip (Node::eval -> Option<&[Cell]>, 8 aura-std nodes,
recorder, harness forward loop) gated by a lib-only build; (3) the
cfg(test) migration (fixtures + eval-output assertions per the spec's
three-role rule) gated by the full four-gate run; (4) the C7/C8 ledger
realization note.
refs #74
Auto-signed under /boss spec auto-sign: all objective gates green
(Step-1.5 precondition, Step-4 self-review, Step-5 grounding-check PASS)
and a unanimous five-lens spec-skeptic panel SOUND, after one editorial
round on the ambiguity lens (the test-migration rule was sharpened to a
three-role classification: eval-output -> Cell, column-write -> Scalar,
channel-send -> Scalar).
Spec for the deferred C7/C8 carrier swap flagged by the C7 realization
note: Cell becomes the hot-path value carrier (Node::eval ->
Option<&[Cell]>, node out-buffers, inter-node forward via a new
AnyColumn::push_cell), and Scalar narrows to the self-describing dynamic
boundaries (the param path, AnyColumn::get, source ingestion).
Behaviour-preserving (C1): only the carrier type narrows.
refs #74
Record the Scalar -> {kind, cell} representation change under C7 (cd3d1ca):
Cell as the tag-free single-value carrier, with the hot-path carrier swap
(eval -> Option<&[Scalar]> and the edges) explicitly flagged as the deferred
next step. Doc-only; no code change.
Three tasks for spec 0046. T1: zip_params (aura-core), RED-first. T2:
GridSpace retains its ParamSpec list + SweepFamily carries it + named_params
(aura-engine), RED-first, with the aura-registry optimize-test SweepFamily
literal threaded in the same task. T3: sim_optimal_manifest typed-Scalar
migration + all eight call sites (aura-cli), behaviour-preserving. Each
signature/field change threads every one of its sites within its own task,
before that task's workspace compile gate.
refs #57
Thread a derived named view of a sweep point so consumers stop re-zipping
param_space() names onto the bare &[Scalar]. A free function zip_params over
(ParamSpec names ⊗ positional point) in aura-core; GridSpace retains the
ParamSpec list it already receives in new(); SweepFamily carries it and
exposes named_params(i). The run-closure signature
`F: Fn(&[Scalar]) -> RunReport + Sync` stays byte-for-byte (honours #52's
#71-firewall constraint, so RandomSpace plugs in with zero reconciliation).
The lossy f64 collapse moves into the manifest constructor; the view stays
typed. Behaviour-preserving; SweepPoint and enumeration untouched.
Design ratified in-context (brainstorm, approach C over A, free function
over a NamedPoint type); a reconciliation comment on #57 records the
resolved forks with provenance. Human sign-off after the auto-sign panel's
grounding lens caught a blast-radius undercount in the first draft —
corrected to all eight sim_optimal_manifest call sites plus the
aura-registry SweepFamily struct-literal.
refs #57
Behaviour-preserving normalisation of the lineage record. The fused
`family_id: String` ("{name}-{counter}") is split into its two factors —
`family: String` + `run: usize` — and the user-facing handle becomes a derived
`family_id()` method ("{family}-{run}"), never stored or parsed.
Why: the fused field forced the counter logic to generate-and-check candidate
strings (to stay robust to '-' inside a name) rather than read it; with `run` a
plain int, `next_run` is a clean numeric max+1 over the field. The split is more
normalised (one fact per field) and the only property the fused token bought — a
single paste-able CLI handle — is recovered by deriving it. Storage and all
user-visible output stay byte-identical: append_family still returns
"{name}-{run}", Family.id is still "{name}-{run}", and the CLI prints the same
"family_id" lines (the json! key + the returned String, both unchanged).
Internal-only: FamilyRunRecord and group_families (now keyed on the (family,
run) tuple, first-seen order preserved) change; lib.rs re-exports, the
extractors, and all of aura-cli are untouched (everything downstream goes
through Family.id + append_family's returned String). The on-disk
families.jsonl key changes ({"family","run"} vs {"family_id"}) — a gitignored
runtime artifact with no committed fixture, and tests round-trip within a temp
dir, so no migration is owed. Doc reconciliation: the lineage/lib module headers
and the C18 ledger realization note now describe the split shape.
Verification: a workflow ran the refactor in an implementer agent, then three
adversarial lenses (behaviour-preservation, new-logic correctness,
completeness/doc-lag) tried to refute it — the first two found nothing, the
third flagged the four doc-lag sites now fixed here. Self-run gates:
cargo test --workspace green (registry 11, incl. a strengthened round-trip test
pinning the split fields + derived handle), cargo clippy --workspace
--all-targets -D warnings clean, cargo doc clean.
Architect drift review (eeba218..HEAD) + regression gate (the project's
cargo test --workspace is the gate; no separate regression script).
What holds (architect): C9 layering intact — aura-engine gains no registry
dependency, lineage lives entirely in aura-registry. #71 firewall held —
RunManifest/aura-core byte-untouched, no blob/path/payload field on
FamilyRunRecord, the member window is producer-supplied via
Source::bounds()/window_of, never a Vec scan. C2 clean — bounds() is
cursor-independent (no look-ahead). Family store is a disjoint sibling of the
flat store (pinned by a disjointness test). Regression gate green: 228 tests,
clippy --workspace --all-targets -D warnings clean.
Drift resolved:
- [fix, this commit] C18 ledger gap: docs/design/INDEX.md C18 carried no
realization note for the run registry. Added the cycle-0029 (flat registry)
and cycle-0045 (lineage as related records / family store) realization
notes, in the ledger's established per-cycle style.
- [forward-queue] Flat-store CLI dead-end: after this cycle no CLI command
writes the flat runs.jsonl (sweep/walkforward moved to the family store,
aura run never persisted), so aura runs list / rank read an unfed store.
This is a spec-accepted state (0045 kept runs list "for standalone runs
only") and a genuine design fork (give aura run a persisting path vs. retire
the flat-store CLI surface) with no clear default — filed as #73 rather than
decided here. Recorded as a deferred follow-up in the C18 ledger note.
- [carry-on] Low/cosmetic: family-wrapped CLI stdout alphabetizes nested
manifest keys (serde_json::json!) vs. declaration order (to_json()); both
valid JSON, the stored families.jsonl uses declaration order, round-trip
unaffected. Noted as a sub-item on #73.
refs #70
Decompose spec 0045 into two iterations. Iteration 1 (engine + registry
core): Source::bounds() + window_of, M1FieldSource::bounds(), and the
aura-registry lineage module (FamilyKind / FamilyRunRecord / Family,
group_families, next_family_id, the three *_member_reports extractors,
Registry::append_family / load_family_members) with round-trip and
per-name-counter tests. Iteration 2 (CLI surface): aura mc, family-aware
aura runs (families / family [rank]), sweep/walkforward family persistence
via append_family with --name, and the window_of migration of the
manifest-window scans.
refs #70
Persist sweep / Monte-Carlo / walk-forward runs as named, linked families
(C18 lineage / C21): a FamilyRunRecord stamps each member with a shared
family_id = "{name}-{counter}" in a sibling families.jsonl store, group_families
re-derives a family as a unit, and the CLI gains aura mc + family-aware aura runs.
Producer-supplied window via Source::bounds()/window_of (the #71 firewall: no Vec
scan, no input artifact). family_id form (name + counter, not a content-hash)
resolved by the user in-session; reconciliation comment on the issue records it.
Auto-signed under the boss spec-auto-sign gate: all objective gates green
(precondition, self-review, grounding-check PASS) and a unanimous five-lens
spec-skeptic panel (criterion, grounding, scope-fork, ambiguity, plan-readiness).
refs #70
Bite-sized, placeholder-free plan for the walk-forward axis, in three tasks:
(1) extract MetricStats::from_values from McAggregate::from_draws + serde derive
(behaviour-preserving, the 7 mc tests guard); (2) the walkforward module —
WindowRoller (bounds-only iterator), walk_forward over the shared run_indexed
core, continuous OOS stitch (empty segment -> +0.0), on-demand param_stability,
+ crate exports + module-header doc-lag fix; (3) the aura walkforward CLI demo
bridging engine+registry via optimize, with serde_json for the summary line.
RED-first tests 1-11 (incl. 7b empty-segment stitch) per the spec.
refs #69
Walk-forward orchestration: a WindowRoller rolls (in-sample, out-of-sample)
splits over a time span, walk_forward runs a disjoint harness per split via the
shared run_indexed core (C1), and stitches the OOS pip-equity segments into one
continuous curve. The varying dimension is the data window (axis 3), realized
eager-agnostically (#71): the public surface carries WindowBounds + a per-window
closure, never a materialized stream Vec; C2 no-look-ahead is a pure bounds
invariant (oos.0 > is.1), checkable with zero ticks.
The in-sample optimize (axis 2) is closure-supplied, not called by the engine:
aura-engine cannot depend on aura-registry (C9), and C12 forbids baking search
policy into the primitive. The CLI bridges both crates in run_walkforward.
Param-stability shape (the load-bearing fork) resolved with the user as R2:
on-demand, not stored. WalkForwardResult stores only raw per-window outcomes +
the stitched curve; param_stability(&result) -> Vec<MetricStats> is a public
on-demand reduction over the retained per-window chosen params (sweep-precedent:
summaries computed on demand, not cached, vs McFamily's stored aggregate). A
stored summary would be recomputable from the raw windows (redundant) and would
force one statistic canonical when "stability" admits several. Recorded with
provenance in the #69 reconciliation comment. MetricStats::from_values is
extracted from McAggregate::from_draws (behaviour-preserving; the 7 mc tests
guard) so the MC aggregate and the helper share one reduction. Empty OOS segment
contributes 0.0 to the stitch offset (mirrors summarize's unwrap_or(0.0)).
Gates: Step-1.5 precondition clean (no fork silently picked), self-review clean,
grounding-check PASS (twice — re-run after each edit). Auto-sign panel escalated
on the scope-fork design lens (param-stability shape) to human sign-off; the user
resolved it (R2), so this is a user-signed spec, not boss-signed.
refs #69
Three tasks: (1) extract the disjoint-parallel core out of sweep_with_threads
into a shared `pub(crate) run_indexed<T>` (behaviour-preserving; the three
pre-existing sweep tests are the green guard), (2) add the `mc.rs` module
(McDraw/McFamily/McAggregate/MetricStats, type-7 `quantile`,
`McAggregate::from_draws`, `monte_carlo`/`monte_carlo_with_threads` driving
run_indexed) with 7 RED tests, (3) wire `mod mc;` + the five public exports into
lib.rs. RED-first; module-declaration ordering keeps mc.rs's tests dark until the
final wire-up task.
refs #68
C12 axis 4: Monte-Carlo as a sweep over seeds. `monte_carlo(base_point, seeds,
closure) -> McFamily`, the analog of SweepFamily, reusing the disjoint-parallel
executor (extracted into a shared `run_indexed` core) rather than forking a new
run loop. Each realization is a disjoint C1 unit; parallelism is across draws.
The aggregate (V1) covers all three run metrics with mean + p5/p25/p50/p75/p95,
a pure post-run reduction over the retained raw draws.
Eager-agnostic firewall (#71): the API takes seeds + a per-draw closure, never a
materialized stream Vec; seed->Source construction stays a closure-body concern.
Signed under the /boss spec auto-sign gate: objective gates green (precondition,
self-review, grounding-check PASS) and a unanimous five-lens spec-skeptic panel
(criterion, grounding, scope-fork, ambiguity, plan-readiness) returned SOUND.
The aggregate fork was settled to V1 by the user (issue #68 reconciliation
comment, verbatim "ja, wir machen v1", 2026-06-15).
refs #68
Two tasks: (1) aura-engine SplitMix64 PRNG + pub SyntheticSpec::source(seed) ->
impl Source beside VecSource, with producer RED tests; (2) aura-cli
sim_optimal_manifest gains a seed param (four callers pass 0), plus a test-only
run_sample_seeded + SeededTrace and three e2e RED tests pinning the #66
acceptance. RED-first throughout.
refs #66
A seeded source whose stream is fully determined by a u64 seed, wired so the
seed reaches RunManifest.seed at the manifest-construction site (Fork B: seed at
the data-generation edge, outside the engine graph). Contract is a producer
Fn(u64) -> impl Source, never seed -> Vec, so the Monte-Carlo family can re-seed
N times without materializing N streams. Deterministic dependency-free PRNG
(SplitMix64) for C1 bit-stability. Precondition for #68 (Monte-Carlo) and #52
(random sweep).
Auto-signed under /boss spec auto-sign: objective gates green (precondition,
self-review, grounding-check PASS) and a unanimous five-lens spec-skeptic panel
(criterion, grounding, scope-fork, ambiguity, plan-readiness all SOUND) after one
editorial-repair round.
refs #66
Architect drift review over 8b330e3..HEAD (the Source-seam cycle #71 plus the
interim #67 optimize work and refactors). No semantic or contract drift: the
seam faithfully preserves C3 (ms→epoch-ns at the one ingestion boundary), C4
(tie-by-source-index, byte-for-byte), and C7 (field-per-source SoA); the full
suite is byte-identical green and the gated streaming tests pass on real data.
Regression gate: `cargo test --workspace` green (the project declares no
dedicated regression script; the suite is the gate).
Tidied stale prose the cycle left behind (no behaviour change):
- docs/design/INDEX.md (C12): replaced the cycle-0011 "deliberate eager gap"
status note with a cycle-0041 realization note — the producer `Source` seam
+ streaming `M1FieldSource` now deliver per-source O(one-chunk) streaming;
precisely scopes what remains open (cross-*sim* Arc<[T]> window sharing,
still unbuilt until the orchestration families #66/#68/#69 consume it).
- aura-core/src/lib.rs: dropped the now-shipped "Source trait + data-server
ingestion" from the module doc's "still to come" list (aura-engine's twin
line was fixed in the cycle; aura-core's was missed).
- aura-ingest/src/lib.rs: module header now documents both coexisting paths
(eager load_m1_window/M1Columns vs lazy streaming M1FieldSource).
- aura-registry/src/lib.rs: rank_by/optimize doc comments linked to the
private `metric_cmp` (a public→private intra-doc-link warning from the #67
commit); demoted to plain code spans, so `cargo doc --workspace` is now
warning-clean.
Decomposes the boss-signed spec into four tasks: (1) the `Source` trait +
`VecSource` adapter in aura-engine; (2) the atomic `Harness::run` re-type to
`Vec<Box<dyn Source>>` with all 53 call sites threaded behaviour-preserving;
(3) the streaming `M1FieldSource` over a data-server window; (4) a gated
streaming e2e + the measured residency predicate.
refs #71
Re-types the engine's ingestion input from a materialized
Vec<Vec<(Timestamp, Scalar)>> into a `Source` producer trait the k-way
merge drives by peek/next, and proves it against a real, lazily-streamed
data-server window in the same cycle — closing the cycle-0011 deliberate
eager-materialization gap (C12 Arc<[T]> cross-sim sharing). A VecSource
adapter keeps every existing run call site byte-identical
(behaviour-preserving); an M1FieldSource streams data-server chunks with
O(one-chunk) residency, independent of window length.
Foundational cycle of the World-II milestone; #66/#68/#69 inherit the
seam. Signed under the boss auto-sign gate: precondition clean, grounding
PASS on the final bytes, unanimous five-lens spec-skeptic panel (two
editorial ambiguity rounds on the residency predicate repaired to a
measured per-pull CHUNK_SIZE ceiling).
refs #71
"Compilat" (German "Kompilat") was a coined noun for the product of the
bootstrap compilation — neither English nor a natural fit. The runtime
artifact already has a code identifier for exactly this thing: the
`FlatGraph` struct (harness.rs). Replace the coinage with that identifier:
- prose mentions -> "flat graph" (mirrors the type, reads plainly)
- definitional anchors -> `FlatGraph` (C11, C23, the running-graph line)
- `render_compilat` -> `render_flat_graph` (historical render symbol;
keeps the `render_blueprint` / `render_flat_graph`
source-vs-product pairing)
- `intra-compilat` -> `intra-graph`
- `from_compilat` (test) -> `from_flat`
"compilation" / "re-compilation" / "bootstrap-as-compilation" (the process,
ordinary English) are deliberately left untouched. Behaviour-preserving:
only comments, design ledger, specs/plans, one test-local variable and its
assert messages change. Full workspace test suite green; clippy clean.
Brummel