Family runs (sweep, Monte-Carlo, walk-forward) ran a full harness per
member and drained its equity+exposure recorders, but immediately folded
the streams to metrics and discarded them — so the C21 family-comparison
view had no per-member data to chart. A new opt-in `--trace <name>` on
`aura sweep|mc|walkforward` now persists each member's streams to disk,
mirroring the single-run `aura run --trace` path. Any single member is
chartable today via `aura chart <name>/<member_key>`; the comparison view
itself is a later cycle.
Design:
- Persist INSIDE each per-member closure (aura-cli), reusing persist_traces
verbatim — the engine and the family types are untouched. The closures
already held the drained rows; they now bind them once, build the
manifest, persist when --trace is set, then fold as before.
- Content-derived, deterministic member keys (MC seed{N}, sweep f{fast}s{slow},
walk-forward oos{ns}). The engine HOFs are Fn + Sync — members run in
parallel, so a runtime ordinal counter would be non-deterministic (a C1
break); keys derive from what each closure receives deterministically.
sweep_member_key panics (naming the axis) if the grid lacks a key axis,
rather than silently collapsing two points to a shared dir.
- Each member is a standalone run-dir at runs/traces/<name>/<member_key>/
(persist_traces with a slash-name; TraceStore nests via create_dir_all),
so the existing single-run viewer charts a member with no view-side code.
- Opt-in: without --trace, stdout and the run registry are byte-unchanged;
the cardinality cost of N members x full resolution stays user-chosen.
Out of scope (deferred): the family-comparison view (overlay/small-multiples),
decimation/LOD, a binary trace container.
Verified locally: cargo build --workspace, cargo test --workspace (all green,
incl. 3 new family persistence tests + a sweep determinism test + 2
sweep_member_key unit tests), cargo clippy --workspace --all-targets
-D warnings (clean). The pre-existing family determinism and single-run
trace tests pass unchanged (no-trace path is byte-identical).
closes#104
Three tasks (MC, sweep, walk-forward) wiring per-member trace persistence
into the family closures + a full-suite/lint gate. Each task threads its
builder signature and every caller together so the workspace compiles
green per task.
refs #104
Persist each family member's equity/exposure streams to disk (opt-in
--trace on sweep|mc|walkforward), mirroring the single-run trace path, so
the C21 family-comparison view has per-member data and any single member
is chartable today. Write side only; the comparison view, decimation,
and a binary trace container are deferred.
Autonomously signed under /boss on the Step-5 grounding-check PASS.
refs #104
Keyboard control for the served chart, deliberately the discrete counterpart to the
glitchy hand-rolled pointer drag that was just removed: each key is one clean
setScale("x", …), deterministic, with none of uPlot's pointer-pan jank.
- arrows pan (15% of the visible span), +/- zoom, 0/Home reset, x toggles the spine.
- zoom anchors on the CURSOR: the x-value under the mouse keeps its relative position,
so the point under the pointer stays put while zooming (centre when the mouse is off
the chart). The mouse position is tracked per chart via posToVal on mousemove.
- keys drive every chart instance together, so panels stay x-aligned.
- uPlot's default mouse (drag = box-zoom, double-click = reset) is retained — you have
both.
The load-bearing math is the pure keyXRange(key, lo, hi, anchor, ext) helper (the rest
is browser-only DOM wiring in mount); it is node-guarded by chart_viewer_keys.mjs, which
pins the zoom-toward-cursor invariant (anchor keeps its relative position) with an
off-centre anchor so a centre-zoom regression fails loudly. All five chart guards green.
refs #103
The hand-rolled wheel-zoom + drag-pan (wheelZoomPlugin) was hakelig/glitchy —
uPlot is not built to be driven that way. Remove the plugin and the cursor.drag
override so uPlot's own defaults govern the mouse: drag = box-zoom on x,
double-click = reset to full range, with the legend value-readout and crosshair
unchanged.
Everything else from the Tier-1 pass stays: the ordinal (gap-collapsed) x-axis
default + continuous toggle, the ordinal-label honesty (ticks/legend report the
real xs[i]), and panel cursor-sync (cursor.sync key retained; only the drag
override and the plugin are dropped). The x-mode guard is flipped to pin the
revert — overlay carries no custom plugin and no cursor.drag override, panels
carry no plugin — and stays green alongside the other three chart guards.
refs #103
Architect drift review over 5bff7e0..HEAD: clean, no drift, no debt. C14/C7
firewall intact (main.rs byte-unchanged → build_chart_data still returns
ChartData{xs,series}, the injected window.AURA_TRACES payload shape untouched;
the ordinal index spine is derived client-side). C22 holds — the
ordinal/continuous toggle + wheel-zoom/drag-pan/dblclick-reset are view
affordances over a recorded trace (read-only setScale on the x range), not a
scene editor. The #102 manifest-header gap is neither closed nor regressed,
still tracked.
Regression gate: cargo test --workspace green (incl. the four chart guards +
render inline tests), clippy --all-targets -D warnings clean; aura has no
separate regression script, so the suite + architect are the gates. cycle
clean.
Spec/plan 0057 retired per the ephemeral-artifact lifecycle (durable record:
the feature commit 45cc1c4, the #103 decision-log comments, and git history).
refs #103
The served trace chart's x-axis no longer plots against raw timestamps by
default — non-trading gaps (nights, weekends) left horizontal blanks that
dominated the view. The x spine is now ordinal by default: every recorded slot
sits at an even position, so gaps collapse and the signal fills the width. A
header toggle flips to continuous (real-time-proportional) spacing and back;
default ordinal honours the requested "x-axis should not be continuous", with
continuous as the one-click option.
Comfort, all on the read/render side (the engine and the Rust serve contract
are untouched — window.AURA_TRACES still carries the real xs, the ordinal
indices are derived client-side):
- ordinal mode keeps the axis honest: ticks and the legend value-readout map a
collapsed index back to the real xs[i], never the index itself.
- wheel-zoom (x, cursor-anchored), drag-pan, double-click-reset on the plot
(uPlot's default drag-box-zoom is disabled so drag pans).
- panels share a cursor.sync key, so the crosshair lines up across stacked
panels.
Confined to chart-viewer.js (buildCharts gains an xMode param + a wheelZoomPlugin
factory; mount re-renders on the toggle) and render.rs (CHART_HEAD toggle button
+ a chart-only style). The two twin node guards (build/gaps) re-point to the
ordinal default in lockstep; a new x-mode guard pins ordinal-default / continuous
/ panel-sync / plugin wiring, and a label-honesty guard (gappy-traces fixture,
xs far from the index spine) pins that ordinal ticks report the real timestamp.
Verified: cargo test --workspace green (incl. the four chart guards + render
inline tests), clippy --all-targets -D warnings clean, e2e (aura run --trace +
aura chart) emits a self-contained page defaulting to ordinal with the toggle.
Spec docs/specs/0057, plan docs/plans/0057 (boss-signed; grounding-check PASS).
Derived-fork decision log in the issue comments.
closes#103
Tier-1 comfort pass for the served trace chart (#103): an ordinal x-axis
default (collapses non-trading gaps) with a one-click continuous toggle, plus
wheel-zoom / drag-pan / dblclick-reset, a value-readout legend, and panel
cursor-sync. Confined to the read/render side (chart-viewer.js + render.rs
CHART_HEAD); the Rust serve contract is unchanged.
Boss-signed: grounding-check PASS (five current-behaviour assumptions ratified
by green node guards + render.rs inline tests). Derived-fork decision log in
the #103 comments.
refs #103
Cycle-close audit for the trace-charts visual face (#101, shipped across
8bb5256 persist + 15ee6d5 serve). Architect drift review (13941a2..HEAD)
+ the regression gate (cargo build/test/clippy --workspace, all exit 0,
orchestrator-verified).
What holds (carry-on): C14 clean — aura-engine/aura-core carry zero
UI/HTML/uPlot knowledge; ColumnarTrace does struct->encoding only, file
I/O in aura-registry, rendering in aura-cli. C1/C3 clean — persistence
and serve-time alignment are post-run pure reductions (join_on_ts at
serve time only, never in-graph). C7 clean — on-disk f64 columns +
kinds-tagged base types, the documented coercion honest, no 5th scalar.
Resolutions:
- [fix] Added the C22 realization note now that the web-from-disk seam
shipped (the amendment 3b56efb had deferred it to "when the seam
ships").
- [ratify] The served page injects the chart SERIES but drops the run
manifest (no run-context header) — a deliberate, defensible naive-viewer
scope call (the manifest is on disk in index.json; a header matters most
for the deferred families/comparison work, C21). The deviation from spec
0056 §1/§4 landed unflagged in 15ee6d5; it is now flagged here and
tracked as a follow-up. refs #102
- [tidy] Retired this cycle's ephemera (spec 0056, plans 0055/0056) and
the stale debris from the prior Runway close (spec 0055-fieldtest) per
the ephemeral-artifact convention; the durable record is the ledger +
git history.
Cycle drift-clean (modulo the ratified, tracked manifest-header gap).
refs #101
Iteration 2 (serve half) of the visual face — spec 0056 / plan 0056:
read a persisted run's traces and serve them as a self-contained uPlot
HTML chart. Completes the goal "tap data from a graph, serve it as a
chart in the browser".
- chart-viewer.js (first-party): a pure buildCharts(traces, mode) that
maps the injected window.AURA_TRACES to uPlot configs (no DOM) — overlay
= one plot, every series on its own y-scale over the shared xs; panels =
one plot per series, all on the same timestamp x. A browser-only mount()
instantiates the vendored global uPlot. Guarded by a node .mjs DOM test
(+ a sparse/gaps variant) shelling out exactly like viewer_dot.rs.
- render_chart_html + ChartMode/ChartData/Series (aura-cli/render.rs):
mirrors render_html — self-contained page, uPlot (1.6.32, vendored
d534f10) + chart-viewer.js inlined via include_str!, data injected as
window.AURA_TRACES.
- emit_chart + build_chart_data (aura-cli/main.rs): the spec-§6 3-step
union-spine alignment — xs = sorted-dedup union of every tap's ts; a
synthetic empty-payload spine + ALL taps as symmetric join_on_ts sides
(no tap privileged, no point dropped); flatten each (tap,column) to a
Series of Option<f64>. New arms `aura chart <name> [--panels]`; a
missing run is stderr + exit 2 (TraceStoreError::NotFound), never a
panic. USAGE advertises the verb in lockstep.
Verified by the orchestrator: `cargo test --workspace` green (incl. the
node guards, render_chart_html, and the chart e2e); `cargo clippy
--workspace --all-targets -D warnings` exit 0; and a real end-to-end —
`aura run --trace demo && aura chart demo` emits a 57KB self-contained
page (doctype + inlined uPlot + AURA_TRACES carrying the equity/exposure
series), `--panels` reflects the panels mode, `aura chart nope` exits 2.
Implementer deviation folded in (compiler-prescribed): serde added to
aura-cli/Cargo.toml for the Series derive (Cargo.lock updated).
refs #101
Vendors uPlot.iife.min.js (the IIFE bundle exposing the global `uPlot`) + uPlot.min.css under assets/, pinned at uplot@1.6.32 via unpkg, mirroring the existing Graphviz-WASM/pan-zoom vendoring (checked in verbatim so include_str! + the build stay hermetic/offline, C1/C8). refresh-assets.sh gains the matching pinned-fetch lines. Consumed by iteration 2's render_chart_html (serve half of spec 0056).
refs #101
Iteration 1 (persist half) of the visual face — spec 0056 / plan 0055:
tap data from a graph and save it to disk so it can later be charted
(the serve/chart half is iteration 2).
- ColumnarTrace (aura-engine): a drained tap <-> columnar SoA form
({tap, kinds, ts, columns}); values coerced to f64 for plotting with
the base type preserved in `kinds` (C7); to_rows rebuilds uniformly-f64
rows for the serve-side join. Pure (C1), beside join_on_ts/summarize.
- TraceStore (aura-registry): a per-run directory under
runs/traces/<name>/ — one <tap>.json per tap + an index.json (manifest
+ tap order), beside the families.jsonl sibling store. Missing run =
NotFound (no panic); corrupt file = Parse{file}.
- CLI: a shared persist_traces helper threaded into run_sample /
run_macd / run_sample_real via an orthogonal `trace: Option<&str>`;
new arms `aura run [--macd] --trace <name>` and `--trace <name>` on
`run --real`. The default (no --trace) run is byte-for-byte unchanged.
Verified by the orchestrator: `cargo test --workspace` green (incl. the
4 columnar_trace, 3 trace_store, 4 cli_run e2e, and the 4-tuple
parse_real_args tests); `cargo clippy --workspace --all-targets -D
warnings` exit 0. Implementer deviations folded in, all
compiler-prescribed: #[derive(Debug)] on RunTraces (the RED tests
panic-format a Result over it), #[allow(clippy::type_complexity)] on
the parse_real_args 4-tuple (mirrors the crate's sample_harness idiom).
Default-run byte-identity stays pinned by run_prints_json_and_exits_zero.
refs #101
Iteration-1 (persist half) plan for spec 0056: ColumnarTrace (aura-engine), TraceStore (aura-registry), and the shared persist_traces helper threaded into all three run forms. RED-first; Task 3 folds all nine signature-change callers in before the build gate (per plan-recon's compile-gate warning). Serve/chart half deferred to iteration 2.
refs #101
First cut of aura's visual face (issue #101; the web-from-disk pivot
ratified into C14/C22 this session, commit 3b56efb): tap arbitrary
recorder streams from a run, persist them to disk as columnar (SoA)
JSON, and serve them as a self-contained uPlot HTML chart — feeds
overlaid on a shared timestamp axis, or timestamp-aligned in stacked
panels. `aura run [--macd] [--trace <name>]` and `aura run --real <SYM>
... [--trace <name>]` persist; `aura chart <name> [--panels]` serves.
Both modes share one synthetic-union-spine join_on_ts alignment.
Signed via the specify auto-sign gate: the user is away under an
explicit "work fully autonomously, document decisions" mandate, so no
manual signature was available. All objective gates green (precondition
clean, self-review clean, grounding-check PASS), and a unanimous 5-lens
adversarial spec-skeptic panel (criterion / grounding / scope-fork /
ambiguity / plan-readiness all SOUND).
The panel earned its keep — two self-correction rounds to unanimity:
- Round 0 caught (a) panels shipped NOT timestamp-aligned, contradicting
the user's verbatim "aligned nach Timestamps in Panels" — corrected to
conform to issue #101 Fork A; (b) the USAGE advertised `--macd --trace`
with no dispatch arm — corrected to an orthogonal `--trace` composing
with every run form via a shared persist helper.
- Round 1 caught the union-spine -> join_on_ts mapping under-specified
(which tap is the spine?) — corrected with an explicit 3-step
synthetic-empty-spine mechanism (no privileged tap, no points dropped),
plus a to_rows uniformly-f64 clarification.
The two delegated naive-viewer decisions (per-series y-scale, union
spine) are documented with provenance on issue #101.
refs #101
The visual face pivots from the ledger's egui-native, in-process
zero-copy direction to a web frontend served from disk-persisted
recorder traces: the engine writes recorded traces to disk, a browser
charts them. Settled in an in-context design discussion (provenance in
issue #101): raw per-tap trace persistence + serve-time join_on_ts
alignment + a static self-contained HTML chart (uPlot). The engine
stays headless (C14) — arguably more strictly, since a browser reading
a serialized file is further from the hot path than egui reaching into
the live SoA columns. Amends the C14 visual-face note and the open
"Playground & World UI surface" thread; the C22 realization note lands
when the seam ships.
refs #101
Milestone-close fieldtest for "Runway — real-data ergonomics & honesty
hardening". Four curated end-to-end scenarios derived top-down from the
milestone promise, run as a downstream consumer against real GER40/EURUSD M1
bars (public interface only; no crates/*/src read):
1. per-instrument pip honesty (#22) — GER40 index pip 1.0 AND EURUSD forex
pip 0.0001 both honest end-to-end; AAPL.US/FOOBAR un-vetted refused
exit 2, no stdout leak
2. real GER40 M1 open via open_ohlc from aura-ingest ALONE (#80/#81/#92) —
no data-server dep, 53173 real bars, C4 O/H/L/C order, ts-window inverse
3. multi-tap ts-join (#93) — two taps at different cadences, 27744 rows,
0 by-ts mismatches; warm-up None rows where a positional zip would corrupt
4. run-registry coherence (#73/#82) — list/rank retired exit 2, families/
family live, two-store Registry read-back + rustdoc
Findings: 4 working, 1 friction, 1 spec_gap.
- friction: Cell exposes only from_i64/from_f64/... while the side-by-side
Scalar is a tagged enum with public I64(..)/F64(..) variants. The natural
param-authoring entry point is Scalar (Cell is the hot-path buffer carrier),
so the wrong reach is a self-correcting compile snag — ergonomic only.
Filed as a follow-up idea.
- spec_gap: unknown family id exits 0 empty — ratified into the C18 note in
the preceding commit (treat-as-empty contract).
Gate verdict: GREEN. All seven axes mapped to a covering scenario and exercised
on real bars; both finding classifications and the one material coverage gap
were adversarially re-verified. That gap — #22's FX-vetted pip face, untested by
the original scenario set — was probed (EURUSD -> sim-optimal(pip_size=0.0001),
FX-scaled, not index-mis-scaled) and folded into scenario 1, so the pip promise
is demonstrated on both positive faces. Every scenario re-run independently on
HEAD before commit.
The Runway milestone fieldtest surfaced a spec_gap: `aura runs family <id>`
with an unknown-but-well-formed id exits 0 with empty output, and neither the
ledger nor the glossary constrained that case. The behaviour is already the
documented, tested CLI contract (treat-as-empty, mirroring Registry::load's
missing-store = Ok(empty)); this lifts that contract to the durable C18 note
so a consumer reasoning from the spec can derive it, and records that tightening
to a non-zero "no such family" exit (typo-safety) stays an available future UX
choice rather than a current contract.
Ratifies the spec_gap from docs/specs/0055-fieldtest-runway-real-data-seam.md.
The milestone-close architect flagged that the runs-retire note over-claimed
"retained because used": after dropping runs list/rank, Registry::append/load (the
flat-store half) have no in-tree caller — only rank_by/optimize keep live
consumers. Note now distinguishes the live half from the public-but-uncalled half
(a latent dead-code surface for a later sweep). Doc-only.
Since cycle 0045 the flat runs.jsonl store has had no producer (sweep/mc/
walkforward persist to the family store; `aura run` does not persist), so
`aura runs list` / `runs rank` read a store nothing populates — a live dead-end
surface, the honesty gap the Runway milestone closes.
Resolve the C18 deferred fork (INDEX.md) as (b) RETIRE: drop the two CLI commands
+ their dispatch arms, usage fragments, and the now-unused load_runs_or_exit.
Families (sweep/mc/walkforward, C21) subsume standalone over-time comparison. The
aura-registry flat lib API (append/load/rank_by/optimize) is RETAINED — optimize
backs walk-forward's in-sample step, rank_by backs `runs family ... rank`.
`runs families` / `runs family` are untouched.
Decision recorded on #73 (user-directed (b), 2026-06-18). Cosmetic sub-item
(json! family-wrapper -> to_json() stdout key-order unification) deferred to a
follow-up: json! routes the embedded report through serde_json::Value, which
re-alphabetizes keys, so it needs manual JSON construction, not a trivial swap.
RED-first: the retire test asserted `runs list`/`rank` now exit 2 (was exit 0).
Verified: cargo build --workspace, clippy --all-targets -D warnings, and
cargo test --workspace all green.
closes#73
Registry::append_family / load_family_members write+read a fixed-name
families.jsonl sibling via self.path.with_file_name(...), so isolation between
registries is per-directory, not per-filename — two Registry::open calls with
different runs filenames in one directory share a single family store. The
sibling was documented on append_family but not on Registry::open, where a caller
binds the path; this records the per-directory-isolation consequence at the bind
point. Pure rustdoc ratify of existing (cycle 0045) behaviour.
closes#82
Audit (architect drift review + cargo test --workspace) clean: code matches the
spec, the engine stayed domain-free (no instrument identity in Ctx / the hot
path), InstrumentSpec sits beside the hot path per C7/C15. No regression scripts
configured (no-op gate); the full suite is green.
Ratify: the architect flagged one ledger-realization gap — C10/C15 described
per-instrument pip as standing design intent but carried no realization note for
the now-shipped channel. Added dated realization notes to C10 (the per-instrument
pip channel + the refuse-don't-guess honesty rule) and C15 (instrument specs as
metadata, first realized by InstrumentSpec/instrument_spec).
Retire the ephemeral cycle artefacts (spec 0054, plan 0054) per the active-cycle
lifecycle. Deferred follow-up: #98 (route the runnable GER40 examples through the
lookup).
refs #22
The data-server symbol stream carries only price bars, no instrument metadata.
So SimBroker was constructed with one global pip_size literal (0.0001, correct
only for 5-decimal FX), and `aura run --real <symbol>` emitted wrong-unit pip
equity for any non-FX instrument (AAPL.US, BTCUSD off by orders of magnitude).
Add the missing channel to *specify* per-instrument pip:
- aura-ingest: `InstrumentSpec { pip_size }` + `instrument_spec(symbol)` — a
typed, Rust-authored vetted lookup (NOT Aura.toml; the later Aura.toml schema
can subsume it). Seeded GER40/FRA40 = 1.0 (index points), EURUSD/GBPUSD/USDCAD
= 0.0001 (5-dp FX majors); None for an un-specced symbol.
- aura-cli: `sample_harness` and `sim_optimal_manifest` gain a `pip_size`
parameter; `run_sample_real` looks the pip up by symbol BEFORE any data access
and refuses (exit 2) an un-specced instrument rather than guessing — honest by
construction. The looked-up pip reaches both the broker divisor and the
manifest broker label (`format!`). Every synthetic caller passes the unchanged
0.0001, now named `SYNTHETIC_PIP_SIZE`. SimBroker (aura-std) is untouched.
Scope: narrowed to the CLI real path — the actual bug. The runnable GER40
examples already bake the correct 1.0 and are NOT buggy; centralizing them
through the lookup is deferred to #98.
Decisions (user-directed, recorded on #22): metadata channel = typed
InstrumentSpec at the source edge; un-specced real symbol refuses; seed =
GER40/FRA40 + FX majors; example refactor deferred. The spec went through the
grounding-check gate; the auto-sign panel surfaced the refusal-behaviour and
scope as genuine design decisions, escalated and resolved by the user.
Tests: instrument_spec unit; manifest pip rendering (1.0 -> "1", 0.0001
unchanged); non-gated CLI refusal (exit 2, no stdout leak, vetted symbol clears
the lookup); gated GER40 binary-boundary honesty (pip_size=1 in the emitted
manifest, deterministic). The pre-existing run_sample_real determinism test was
retargeted AAPL.US -> GER40 (AAPL.US is un-specced and would now refuse at the
lookup); the AAPL.US bounded-window streaming property still lives in
aura-ingest/tests/streaming_seam.rs (literal source, no spec lookup).
Verified: cargo build --workspace, clippy --all-targets -D warnings, and
cargo test --workspace all green; the gated GER40 path ran with local data.
closes#22
Cycle 0053 (multi-tap trace join on timestamp, #93) closes drift-clean.
Audit (architect drift review over 066638e..HEAD against docs/design + the
CLAUDE.md domain invariants):
- C8/C18 honoured: join_on_ts is a post-run reduction over recorded
(Timestamp, Vec<Scalar>) sink streams, a true sibling of summarize/f64_field
in the report layer — correct architectural home.
- C3 not violated: the join runs post-run over already-recorded output, not
in-graph; no as-of join smuggled into the DAG. C1 ts-uniqueness is a
documented precondition, not a runtime check — consistent with how the
engine leaves wiring-time invariants to contract.
- Option-per-side keeps the engine honest (reports presence; the consumer owns
the 0.0/-1/false defaults). The hand-rolled *_by_ts HashMaps are fully gone;
the two RED unit tests pin the #93 cardinality-misalignment + spine-drop.
- No drift, no debt, no ledger entry needs updating. Recommendation: carry on.
Regression gate: no dedicated regression scripts configured (optional per the
conventions); build/test/clippy --workspace --all-targets is the gate and ran
green — the gated GER40 byte-identity tests executed against the local archive
(not skipped), proving drain_trace's trace is byte-preserved across the
refactor.
Per the artifact lifecycle (committed while the cycle is live, git rm at cycle
close), the ephemeral spec and plan are retired here; their durable rationale
lives in the implementation commits 74324d1 / 35c5adc and the git history.
drain_trace now joins its four recording-sink taps through aura-engine's
join_on_ts instead of a hand-rolled per-tap HashMap, resolving the #93
zip-by-index panic at its source: the four taps fire at different
cardinalities (breakout one bar short via cold Delay(1); bars_since_open
filtered by Session), so a positional zip misaligns or panics. The BarTrace
mapping stays a thin layer with its caller-side defaults unchanged
(held->0.0, bars_since_open->-1, breakout->false) — the engine reports
presence, the consumer interprets absence. Behaviour is byte-preserved: the
gated GER40 tests (ger40_breakout_real, ger40_breakout_blueprint,
open_ohlc_seam) stay green against the real archive.
The Recorder doc gains a prose note on the multi-tap cadence rule (join on
ts, never by index), pointing at join_on_ts by name rather than rustdoc link
since aura-std does not depend on aura-engine.
Verified: cargo build/test/clippy --workspace --all-targets clean; the
_by_ts hand-rolled join is gone (acceptance grep empty).
closes#93
Ship a generic, spine-anchored join_on_ts helper + JoinedRow in the report
layer (sibling of summarize/f64_field), re-exported from lib.rs. It fuses N
recorded (Timestamp, Vec<Scalar>) tap streams on the recorded timestamp:
exactly one JoinedRow per spine entry, each side looked up by ts as
Some(row)/None where it did not fire. Option-per-side semantics keep the
engine honest — it reports presence; the consumer interprets absence (the
0.0/-1/false defaults are consumer truths, not engine ones). C8/C18 (post-run
reduction over recorded sink output), C3 (no in-graph join), C1 (one row per
ts, documented precondition).
Two unit tests pin the contract: cardinality-misalignment alignment (the #93
shape — a spine bar a side tap skipped resolves to None, not a zip-misalign)
and the spine-anchored drop of a side row at a non-spine ts. RED accepted on
the grounding-check record (both symbols were absent pre-cycle, so the tests
are RED-by-construction) and verified green here.
drain_trace's migration onto this helper + the Recorder doc note follow.
refs #93
Bite-sized execution plan for spec 0053: Task 1 the join_on_ts helper +
JoinedRow + two RED unit tests in aura-engine report.rs (re-exported from
lib.rs); Task 2 the behaviour-preserving drain_trace refactor onto the helper;
Task 3 the Recorder prose cadence/join note. Final gate is workspace
build/test/clippy.
refs #93
Ratified design for a reusable, spine-anchored join_on_ts helper in
aura-engine that fuses multi-tap Recorder traces on the recorded timestamp
(Option-per-side: the engine reports presence, the consumer interprets
absence). Refactors drain_trace onto it and documents the cadence rule on
the Recorder API. Resolves the #93 zip-by-index panic.
refs #93
Cycle 0052 (real-data source-open seam, Runway-1) closes drift-clean.
Audit (architect drift review over aedaa5d..HEAD against docs/design + the
CLAUDE.md domain invariants):
- C3 (one ms<->ns crossing) strengthened — the inverse is seam-private and the
only consumer-side divide left; every consumer ns_to_ms / unix_ms_to_epoch_ns
wrap is gone.
- C4 (O/H/L/C merge tie-break order) now in exactly one vetted home (open_ohlc);
the hand-spelled four-field loops are gone.
- C12 (one shared Arc<DataServer>/cache) preserved; C7 surface additive.
- No drift, no debt, no ledger entry needs updating. Recommendation: carry on.
Regression gate: no dedicated regression scripts configured (optional per the
conventions); build/test/clippy --workspace is the gate and ran green — the gated
open_ohlc_seam behaviour-preservation test executed against the local archive
(not skipped), proving byte-identity vs the ms-path.
Per the artifact lifecycle (committed while the cycle is live, git rm at cycle
close), the ephemeral spec and plan are retired here; their durable rationale
lives in the implementation commit 8c9a1b4 and the git history.
A real OHLC source now builds from aura-ingest alone, in the engine-native
epoch-ns Timestamp currency, with the one ms<->ns crossing owned by the seam.
Library surface (src/lib.rs), all additive — the ms-based `open` and
`unix_ms_to_epoch_ns` are untouched, so every existing call site is preserved
byte-for-byte:
- epoch_ns_to_unix_ms (private): the seam-owned inverse; consumers never
convert (#80).
- M1FieldSource::open_window: the Timestamp-window mirror of `open`, mapping
each bound through the private inverse and delegating to `open`.
- open_ohlc: the canonical OHLC bundle opener — four M1FieldSources in the fixed
open/high/low/close C4 merge order, sharing one Arc<DataServer> (one cache,
C12); the order lives in one vetted place (#92).
- default_data_server + `pub use data_server::{DataServer, DEFAULT_DATA_PATH}`:
a real-data source builds without naming the external data_server crate (#81).
Consumer migration: the shared breakout_real.rs helpers and every GER40
example/test move to the Timestamp-native surface — open_ohlc_sources removed,
utc_month_window_ms -> utc_month_window (Timestamp), report_from_trace retyped,
the per-consumer ns_to_ms hand-divides deleted, the direct data_server imports
replaced by the re-exports. ger40_breakout_compare.rs (which opened OHLC by its
own hand-loop, reached by neither retyped helper) is migrated by inspection.
Return type is Vec, not [_; 4]: Harness::run consumes Vec<Box<dyn Source>>, so
it feeds straight in; order-safety comes from the single helper either way.
Tests: a hermetic round-trip pinning the inverse; a gated behaviour-preservation
test (open_ohlc Timestamp path == ms-path open, bit-identical recorded series);
a hermetic absent-archive fixture pinning the file-level None contract and the
#81 re-export firewall.
Verified: cargo build --workspace --all-targets clean; cargo test --workspace
green (the gated GER40 tests ran against the local archive — open_ohlc_seam
proves byte-identity on real data, not a skip); clippy --all-targets -D warnings
clean. Acceptance greps pass: open_ohlc_sources gone, no consumer-side ns_to_ms,
no data_server import in any migrated GER40 consumer, epoch_ns_to_unix_ms private.
closes#80, #81, #92
Consolidate aura-ingest's OHLC source-open surface onto the engine-native
epoch-ns Timestamp currency: a private seam-owned inverse, a Timestamp-window
opener, a canonical OHLC bundle opener (fixed C4 O/H/L/C order), and a
DataServer/DEFAULT_DATA_PATH re-export + default_data_server convenience — so a
real-data source builds from aura-ingest alone, with the one ms<->ns crossing
owned by the seam. Behaviour-preserving (byte-identical RunReports).
Grounded against the tree via the ground-spec-0052 workflow (25/28 assertions
confirmed; compare.rs compiler-invisible site + AC1 scope fix applied).
refs #80, #81, #92
Drop the gitignore entries for both dirs; align the CLAUDE.md note from "git-ignored, local-only" to "git-tracked — committed while the cycle is live, removed (git rm) at cycle close". The durable record stays the ledger + git history; specs/plans are transient in the repo, present only for their own cycle.
docs/specs and docs/plans were retired (prior commit); the source/test
comments that cited them ("spec 0050 §4.1", "spec §Testing N", "per spec",
"the spec's ...") now point at nothing. Strip every such pointer while
preserving the technical substance, the design-ledger contract refs
(C1/C11/C20/C34/C12.1/...), and the Gitea issue refs (#41).
Comment/doc edits only across 20 files — no logic change; full workspace
suite green, clippy clean.
Per-cycle specs/plans are step-scoped: every file maps to already-shipped
code, their snippets drift hard against current APIs (InputSpec->PortSpec,
Sim->Harness, schema() removed, C16 zero-dep reversed, ascii-dag retired),
and a stale spec read as an API reference misleads agents into non-compiling
code. No durable knowledge lost — rationale lives in the ledger + code
rustdoc, history in git + the tracker.
- gitignore docs/specs/ and docs/plans/
- delete all 117 existing specs/plans (recoverable via git history)
- CLAUDE.md: specs/plans are local-only ephemeral artifacts
The residency probe was an inherent method on the concrete M1FieldSource, so a Box<dyn Source> consumer could not read it without downcasting (the adjacent friction noted in #95). Hoist it onto the Source trait as resident_records(&self) -> Option<usize> with a default of None, mirroring bounds()'s Option-for-unknown idiom: None = "does not report" (the eager VecSource, which has no streaming ring), distinct from Some(0) = "reports, holds none" (e.g. exhausted).
M1FieldSource overrides it to return Some(chunk len). Callers updated (streaming_seam, the two deep-dive fieldtest bins). Tests: a hermetic VecSource default-is-None check, and a &dyn Source probe in the gated residency test proving M1FieldSource residency is readable without a downcast.
refs #95
The C12 realization and the 0041 spec claimed streaming residency is O(one chunk) at the process level ("a 20-year window streams in the same memory as a one-day one"). That holds only for the aura source ring (M1FieldSource::resident_records(), bounded by one chunk and correctly tested); whole-process RSS grows O(records-touched) because data-server's FileCache retains each window's parsed chunks (~56 B/record) read-only for the pass. Reproduced: 6 MiB @ 1mo -> 173 MiB @ ~10y, linear.
That retention is the replay-many optimization (load-once; close+volume share one parse via the field-agnostic FileKey), not a leak, and no always-on eviction can bound a single forward pass without regressing it. This narrows the docs/comments to the true per-source-ring property and names the FileCache per-window retention as the real, replay-amortized process cost. The streaming_seam assert is unchanged (it was always the ring bound; only its labelling confused ring vs process). The deferred opt-in non-retaining streaming read path is parked as Brummel/data-server#2.
closes#95
data-server does no timezone math (delphi_to_unix_ms is pure arithmetic), so the
archive carries whatever the raw Pepperstone files contain, labelled as UTC. The
Session node's UTC->Berlin DST conversion is unit-tested, but the assumption that
the raw timestamps are themselves UTC was unverified — a broker-local archive
would shift the whole session and the error would jump across the DST boundary.
Verified empirically via the DAX cash-open volume spike (GER40): it lands at
Berlin 09:00 in BOTH seasons, at 07:00 UTC (summer, CEST, UTC+2) and 08:00 UTC
(winter, CET, UTC+1) — a clean +1h DST shift, the genuine-UTC signature. This
gated test pins it so a future archive/convention drift cannot silently corrupt
every session strategy's bar alignment.
Add a C20 realization recording the resolution of the GER40 deep-dive's #94
finding: a hand-wired FlatGraph is not a shippable strategy (no param_space, so
World-opaque); the canonical form is the Composite blueprint, the FlatGraph its
compiled substrate (C23). Pins the bar-period-as-structural-construction-arg
decision (spec 0051) to the structural-axis-vs-tuning-param split.
refs #94
The shipped Composite blueprint (ger40_breakout_blueprint) is now driven by the
World orchestration families on real GER40, with NO re-authoring — the #94
friction is gone:
- examples/ger40_breakout_sweep.rs — sweep the {entry_bar, exit_bar} grid built
directly from param_space() (best entry=3,exit=4 -> 214.0 over 2024).
- examples/ger40_breakout_walkforward.rs — non-degenerate walk_forward: a real
non-empty space optimized per window, chosen_params populated everywhere (#97).
- examples/ger40_breakout_compare.rs — one blueprint across two structural axes:
instrument (GER40 vs FRA40) and bar period (15m vs 30m — different strategies,
C34), no re-authoring.
- tests/ger40_breakout_world.rs — gated: sweep returns the ranked 9-point grid
over the two named params; walk_forward yields non-empty windows each with a
non-empty chosen_params (executable #97 proof).
No blueprint reshape needed — the factory returns a fresh (Composite, Taps) per
call, so the World families re-instantiate per grid point / window (the SMA-cross
pattern); baked recorders do not block multi-bootstrap.
closes#94, closes#96, closes#97
Author the shipped session-breakout as a reusable Composite blueprint via
GraphBuilder — the World-consumable form (param_space + bootstrap) — alongside
the hand-wired FlatGraph it reproduces. Applies the spec-0051 decisions:
- bar_period_minutes is a construction arg binding BOTH Resample.period_minutes
and Session's period, locked equal, so a sweep cannot desync them (#96);
- delay.lag is bound out (a structural constant, not a tuning knob — C34);
- the two EqConst targets (named entry_bar / exit_bar) are the ONLY params, so
param_space() == {entry_bar.target, exit_bar.target} — the genuine tuning
knobs, which makes walk_forward non-degenerate (#97).
The ger40_breakout_real example now bootstraps from the blueprint (still prints
-45.0 for 2024-09). New gated test ger40_breakout_blueprint.rs proves: (a) the
param_space is exactly the two targets (no lag, no period); (b) C23 — the
blueprint reproduces the FlatGraph bit-identically over real GER40 2024-09;
(c) determinism.
First increment of the milestone; the sweep / walk_forward / compare World-family
demos follow.
refs #94, #96, #97
Milestone spec: ship the GER40 session-breakout as a Composite blueprint so the
same strategy runs across backtest / sweep / walk_forward / compare without
re-authoring (the #94 fieldtest friction). Records the design decision that the
bar period is a structural axis (a construction arg binding both Resample and
Session, so a sweep cannot desync them — #96) and that the canonical shippable
strategy form is a Composite (FlatGraph is its compiled substrate). Authoring-
side; no engine-core change (the existing bind() + GraphBuilder + param_space()
layer already suffices).
refs #94, #96, #97
Public-API research deep-dive driving the just-shipped GER40 real-data
session-breakout demo (5b5f034) through the escalating sequence a serious
researcher attempts next: scale to multi-year, walk-forward, structural-param
sweep, multi-instrument compare. Standalone consumer crate (path-deps only;
public interface = ledger + glossary + example corpus + cargo doc; no
crates/*/src read). All runs against the real /mnt/tickdata/Pepperstone archive.
Findings: 3 bugs, 1 friction, 3 spec_gap, 1 working. Triaged to the tracker:
- spec_gap: the flagship breakout ships as a raw FlatGraph the World API cannot
consume (sweep / walk_forward / compare all forced a hand re-author to a
Composite, which reproduced the FlatGraph numbers EXACTLY) — refs #94.
- bug: process residency is O(window) (6->173 MiB across 1mo->10y) despite the
documented O(one-chunk) claim; the aura ring stays <=1024 but data-server
retention scales, and streaming_seam.rs only measures the ring — refs #95.
- bug: the bar-period is one knob split across Resample (tunable param) and
Session (baked) and desyncs silently to 0.0 pips; Delay.lag leaks into the
default param_space (the C34 deform-vs-tune discriminator) — refs #96.
- spec_gap: the walk_forward empty-space idiom for param-free strategies is
undefined — refs #97.
- spec_gap (no per-instrument pip registry) and friction (one-directional
ns->ms conversion) re-confirm the known #22 and #80.
- working: C1 determinism over 7y / 148k bars, the lazy streaming source, and
the sweep/WFO cores held under real multi-year load; the Composite re-author
was behaviour-preserving (C23).
Scratch consumer crate + run transcripts committed under fieldtests/ per the
fieldtest convention (matching cycle-0049). The crate path-deps the engine and
reads the real archive; its bins are the repros cited in the issues above.
Run the Phase-1 session-breakout node vocabulary over REAL GER40 M1 bars
from the data-server archive — the first real-data exercise of the shipped
strategy nodes (Resample/Delay/Gt/Session/EqConst/And/Latch + SimBroker).
It is a pure composition of shipped aura-std nodes (no project-specific
signal code), so it lives in the engine repo's examples/ carveout (C9), not
a separate consumer crate. The breakout FlatGraph is reused VERBATIM from
aura-engine/tests/ger40_breakout.rs; only the sources change — four real
M1FieldSource (open/high/low/close, in the fixed C4 merge order) replace the
synthetic VecSources, with pip_size = 1.0 (GER40 is an index).
- examples/ger40_breakout_real.rs — runnable demo over a Sept-2024 window:
prints the RunReport metrics plus a per-session entry/exit trace.
- examples/shared/breakout_real.rs — the single 13-node wiring (the proven
11 + two trace taps), shared by the example and the test via #[path] so it
is never duplicated.
- tests/ger40_breakout_real.rs — gated determinism test (mirrors
real_bars.rs): skips where the archive is absent; where present, asserts
finite metrics, binary Latch exposure (held in {0.0, 1.0}), and
bit-identical C1 reruns of both the report and the recorded series.
chrono / chrono-tz added as aura-ingest dev-dependencies (the Session
timezone and the UTC window bounds), test-only — never on the normal path.
Hand-wires all seven new aura-std nodes (Resample, Delay, Gt, Session, EqConst
x2, And, Latch) + SimBroker into a raw-index FlatGraph over one synthetic
Frankfurt session, pinning the whole vocabulary composed end-to-end:
- held = [0,0,1,1,0] over the 5 bar emissions: flat, latched bar3-close (the
strict breakout landing on session bar 3) through bar4, exit at bar5-close;
- equity = [0,0,0,3,8] pips (SimBroker lagged-exposure integration, pip=1.0);
- a no-entry control (bar3 close == bar2 high) pins the strict-> semantic;
- two disjoint runs byte-identical (C1).
Both spec traps exercised (bar6 rollover closes bar5; bars 1-2 warm Delay[1]).
Phase-1 C9 deliverable; build-step 8 (capstone) of milestone 'Strategy node
vocabulary I'.
closes#91
Maps ctx.now() (epoch-ns UTC) to a Frankfurt session bar index: emits
bars_since_open:i64 = (local wall-clock minutes past the 09:00 open) /
period_minutes, in tz-aware DST-correct local time. So local 09:45 reads 3 in
both CEST (UTC+2) and CET (UTC+1) — the close-instant convention (spec 0050
§4.1). Trigger is an f64 Any input (value ignored, wired from close15) so the
node fires once per completed bar. Open/tz/period are baked structural config,
not scalar params (a timezone is not a scalar, C11). Admits chrono/chrono-tz to
aura-std — vetted DST math, never hand-rolled (per-case dep policy). Last node;
build-step 7 of milestone 'Strategy node vocabulary I'.
closes#90
Aggregates a fine 4-field OHLC stream into period_minutes buckets (open=first,
high=max, low=min, close=last), emitting a completed bar ONLY on bucket rollover
(ctx.now() crossing into the next bucket) — C2: a bar is actionable only once
complete, partials are never emitted and the last partial bar is dropped (no EOF
flush). Four Barrier(0) f64 inputs, a 4-field f64 record output (the first
multi-field-output node in aura-std, mirroring the Ohlcv engine fixture). The bar
carries no timestamp; the engine stamps the close-instant. Build-step 6 of
milestone 'Strategy node vocabulary I'.
closes#89
Brummel