Tier-1 comfort pass for the served trace chart (#103): an ordinal x-axis
default (collapses non-trading gaps) with a one-click continuous toggle, plus
wheel-zoom / drag-pan / dblclick-reset, a value-readout legend, and panel
cursor-sync. Confined to the read/render side (chart-viewer.js + render.rs
CHART_HEAD); the Rust serve contract is unchanged.
Boss-signed: grounding-check PASS (five current-behaviour assumptions ratified
by green node guards + render.rs inline tests). Derived-fork decision log in
the #103 comments.
refs #103
Cycle-close audit for the trace-charts visual face (#101, shipped across
8bb5256 persist + 15ee6d5 serve). Architect drift review (13941a2..HEAD)
+ the regression gate (cargo build/test/clippy --workspace, all exit 0,
orchestrator-verified).
What holds (carry-on): C14 clean — aura-engine/aura-core carry zero
UI/HTML/uPlot knowledge; ColumnarTrace does struct->encoding only, file
I/O in aura-registry, rendering in aura-cli. C1/C3 clean — persistence
and serve-time alignment are post-run pure reductions (join_on_ts at
serve time only, never in-graph). C7 clean — on-disk f64 columns +
kinds-tagged base types, the documented coercion honest, no 5th scalar.
Resolutions:
- [fix] Added the C22 realization note now that the web-from-disk seam
shipped (the amendment 3b56efb had deferred it to "when the seam
ships").
- [ratify] The served page injects the chart SERIES but drops the run
manifest (no run-context header) — a deliberate, defensible naive-viewer
scope call (the manifest is on disk in index.json; a header matters most
for the deferred families/comparison work, C21). The deviation from spec
0056 §1/§4 landed unflagged in 15ee6d5; it is now flagged here and
tracked as a follow-up. refs #102
- [tidy] Retired this cycle's ephemera (spec 0056, plans 0055/0056) and
the stale debris from the prior Runway close (spec 0055-fieldtest) per
the ephemeral-artifact convention; the durable record is the ledger +
git history.
Cycle drift-clean (modulo the ratified, tracked manifest-header gap).
refs #101
Iteration 2 (serve half) of the visual face — spec 0056 / plan 0056:
read a persisted run's traces and serve them as a self-contained uPlot
HTML chart. Completes the goal "tap data from a graph, serve it as a
chart in the browser".
- chart-viewer.js (first-party): a pure buildCharts(traces, mode) that
maps the injected window.AURA_TRACES to uPlot configs (no DOM) — overlay
= one plot, every series on its own y-scale over the shared xs; panels =
one plot per series, all on the same timestamp x. A browser-only mount()
instantiates the vendored global uPlot. Guarded by a node .mjs DOM test
(+ a sparse/gaps variant) shelling out exactly like viewer_dot.rs.
- render_chart_html + ChartMode/ChartData/Series (aura-cli/render.rs):
mirrors render_html — self-contained page, uPlot (1.6.32, vendored
d534f10) + chart-viewer.js inlined via include_str!, data injected as
window.AURA_TRACES.
- emit_chart + build_chart_data (aura-cli/main.rs): the spec-§6 3-step
union-spine alignment — xs = sorted-dedup union of every tap's ts; a
synthetic empty-payload spine + ALL taps as symmetric join_on_ts sides
(no tap privileged, no point dropped); flatten each (tap,column) to a
Series of Option<f64>. New arms `aura chart <name> [--panels]`; a
missing run is stderr + exit 2 (TraceStoreError::NotFound), never a
panic. USAGE advertises the verb in lockstep.
Verified by the orchestrator: `cargo test --workspace` green (incl. the
node guards, render_chart_html, and the chart e2e); `cargo clippy
--workspace --all-targets -D warnings` exit 0; and a real end-to-end —
`aura run --trace demo && aura chart demo` emits a 57KB self-contained
page (doctype + inlined uPlot + AURA_TRACES carrying the equity/exposure
series), `--panels` reflects the panels mode, `aura chart nope` exits 2.
Implementer deviation folded in (compiler-prescribed): serde added to
aura-cli/Cargo.toml for the Series derive (Cargo.lock updated).
refs #101
Vendors uPlot.iife.min.js (the IIFE bundle exposing the global `uPlot`) + uPlot.min.css under assets/, pinned at uplot@1.6.32 via unpkg, mirroring the existing Graphviz-WASM/pan-zoom vendoring (checked in verbatim so include_str! + the build stay hermetic/offline, C1/C8). refresh-assets.sh gains the matching pinned-fetch lines. Consumed by iteration 2's render_chart_html (serve half of spec 0056).
refs #101
Iteration 1 (persist half) of the visual face — spec 0056 / plan 0055:
tap data from a graph and save it to disk so it can later be charted
(the serve/chart half is iteration 2).
- ColumnarTrace (aura-engine): a drained tap <-> columnar SoA form
({tap, kinds, ts, columns}); values coerced to f64 for plotting with
the base type preserved in `kinds` (C7); to_rows rebuilds uniformly-f64
rows for the serve-side join. Pure (C1), beside join_on_ts/summarize.
- TraceStore (aura-registry): a per-run directory under
runs/traces/<name>/ — one <tap>.json per tap + an index.json (manifest
+ tap order), beside the families.jsonl sibling store. Missing run =
NotFound (no panic); corrupt file = Parse{file}.
- CLI: a shared persist_traces helper threaded into run_sample /
run_macd / run_sample_real via an orthogonal `trace: Option<&str>`;
new arms `aura run [--macd] --trace <name>` and `--trace <name>` on
`run --real`. The default (no --trace) run is byte-for-byte unchanged.
Verified by the orchestrator: `cargo test --workspace` green (incl. the
4 columnar_trace, 3 trace_store, 4 cli_run e2e, and the 4-tuple
parse_real_args tests); `cargo clippy --workspace --all-targets -D
warnings` exit 0. Implementer deviations folded in, all
compiler-prescribed: #[derive(Debug)] on RunTraces (the RED tests
panic-format a Result over it), #[allow(clippy::type_complexity)] on
the parse_real_args 4-tuple (mirrors the crate's sample_harness idiom).
Default-run byte-identity stays pinned by run_prints_json_and_exits_zero.
refs #101
Iteration-1 (persist half) plan for spec 0056: ColumnarTrace (aura-engine), TraceStore (aura-registry), and the shared persist_traces helper threaded into all three run forms. RED-first; Task 3 folds all nine signature-change callers in before the build gate (per plan-recon's compile-gate warning). Serve/chart half deferred to iteration 2.
refs #101
First cut of aura's visual face (issue #101; the web-from-disk pivot
ratified into C14/C22 this session, commit 3b56efb): tap arbitrary
recorder streams from a run, persist them to disk as columnar (SoA)
JSON, and serve them as a self-contained uPlot HTML chart — feeds
overlaid on a shared timestamp axis, or timestamp-aligned in stacked
panels. `aura run [--macd] [--trace <name>]` and `aura run --real <SYM>
... [--trace <name>]` persist; `aura chart <name> [--panels]` serves.
Both modes share one synthetic-union-spine join_on_ts alignment.
Signed via the specify auto-sign gate: the user is away under an
explicit "work fully autonomously, document decisions" mandate, so no
manual signature was available. All objective gates green (precondition
clean, self-review clean, grounding-check PASS), and a unanimous 5-lens
adversarial spec-skeptic panel (criterion / grounding / scope-fork /
ambiguity / plan-readiness all SOUND).
The panel earned its keep — two self-correction rounds to unanimity:
- Round 0 caught (a) panels shipped NOT timestamp-aligned, contradicting
the user's verbatim "aligned nach Timestamps in Panels" — corrected to
conform to issue #101 Fork A; (b) the USAGE advertised `--macd --trace`
with no dispatch arm — corrected to an orthogonal `--trace` composing
with every run form via a shared persist helper.
- Round 1 caught the union-spine -> join_on_ts mapping under-specified
(which tap is the spine?) — corrected with an explicit 3-step
synthetic-empty-spine mechanism (no privileged tap, no points dropped),
plus a to_rows uniformly-f64 clarification.
The two delegated naive-viewer decisions (per-series y-scale, union
spine) are documented with provenance on issue #101.
refs #101
The visual face pivots from the ledger's egui-native, in-process
zero-copy direction to a web frontend served from disk-persisted
recorder traces: the engine writes recorded traces to disk, a browser
charts them. Settled in an in-context design discussion (provenance in
issue #101): raw per-tap trace persistence + serve-time join_on_ts
alignment + a static self-contained HTML chart (uPlot). The engine
stays headless (C14) — arguably more strictly, since a browser reading
a serialized file is further from the hot path than egui reaching into
the live SoA columns. Amends the C14 visual-face note and the open
"Playground & World UI surface" thread; the C22 realization note lands
when the seam ships.
refs #101
Milestone-close fieldtest for "Runway — real-data ergonomics & honesty
hardening". Four curated end-to-end scenarios derived top-down from the
milestone promise, run as a downstream consumer against real GER40/EURUSD M1
bars (public interface only; no crates/*/src read):
1. per-instrument pip honesty (#22) — GER40 index pip 1.0 AND EURUSD forex
pip 0.0001 both honest end-to-end; AAPL.US/FOOBAR un-vetted refused
exit 2, no stdout leak
2. real GER40 M1 open via open_ohlc from aura-ingest ALONE (#80/#81/#92) —
no data-server dep, 53173 real bars, C4 O/H/L/C order, ts-window inverse
3. multi-tap ts-join (#93) — two taps at different cadences, 27744 rows,
0 by-ts mismatches; warm-up None rows where a positional zip would corrupt
4. run-registry coherence (#73/#82) — list/rank retired exit 2, families/
family live, two-store Registry read-back + rustdoc
Findings: 4 working, 1 friction, 1 spec_gap.
- friction: Cell exposes only from_i64/from_f64/... while the side-by-side
Scalar is a tagged enum with public I64(..)/F64(..) variants. The natural
param-authoring entry point is Scalar (Cell is the hot-path buffer carrier),
so the wrong reach is a self-correcting compile snag — ergonomic only.
Filed as a follow-up idea.
- spec_gap: unknown family id exits 0 empty — ratified into the C18 note in
the preceding commit (treat-as-empty contract).
Gate verdict: GREEN. All seven axes mapped to a covering scenario and exercised
on real bars; both finding classifications and the one material coverage gap
were adversarially re-verified. That gap — #22's FX-vetted pip face, untested by
the original scenario set — was probed (EURUSD -> sim-optimal(pip_size=0.0001),
FX-scaled, not index-mis-scaled) and folded into scenario 1, so the pip promise
is demonstrated on both positive faces. Every scenario re-run independently on
HEAD before commit.
The Runway milestone fieldtest surfaced a spec_gap: `aura runs family <id>`
with an unknown-but-well-formed id exits 0 with empty output, and neither the
ledger nor the glossary constrained that case. The behaviour is already the
documented, tested CLI contract (treat-as-empty, mirroring Registry::load's
missing-store = Ok(empty)); this lifts that contract to the durable C18 note
so a consumer reasoning from the spec can derive it, and records that tightening
to a non-zero "no such family" exit (typo-safety) stays an available future UX
choice rather than a current contract.
Ratifies the spec_gap from docs/specs/0055-fieldtest-runway-real-data-seam.md.
The milestone-close architect flagged that the runs-retire note over-claimed
"retained because used": after dropping runs list/rank, Registry::append/load (the
flat-store half) have no in-tree caller — only rank_by/optimize keep live
consumers. Note now distinguishes the live half from the public-but-uncalled half
(a latent dead-code surface for a later sweep). Doc-only.
Since cycle 0045 the flat runs.jsonl store has had no producer (sweep/mc/
walkforward persist to the family store; `aura run` does not persist), so
`aura runs list` / `runs rank` read a store nothing populates — a live dead-end
surface, the honesty gap the Runway milestone closes.
Resolve the C18 deferred fork (INDEX.md) as (b) RETIRE: drop the two CLI commands
+ their dispatch arms, usage fragments, and the now-unused load_runs_or_exit.
Families (sweep/mc/walkforward, C21) subsume standalone over-time comparison. The
aura-registry flat lib API (append/load/rank_by/optimize) is RETAINED — optimize
backs walk-forward's in-sample step, rank_by backs `runs family ... rank`.
`runs families` / `runs family` are untouched.
Decision recorded on #73 (user-directed (b), 2026-06-18). Cosmetic sub-item
(json! family-wrapper -> to_json() stdout key-order unification) deferred to a
follow-up: json! routes the embedded report through serde_json::Value, which
re-alphabetizes keys, so it needs manual JSON construction, not a trivial swap.
RED-first: the retire test asserted `runs list`/`rank` now exit 2 (was exit 0).
Verified: cargo build --workspace, clippy --all-targets -D warnings, and
cargo test --workspace all green.
closes#73
Registry::append_family / load_family_members write+read a fixed-name
families.jsonl sibling via self.path.with_file_name(...), so isolation between
registries is per-directory, not per-filename — two Registry::open calls with
different runs filenames in one directory share a single family store. The
sibling was documented on append_family but not on Registry::open, where a caller
binds the path; this records the per-directory-isolation consequence at the bind
point. Pure rustdoc ratify of existing (cycle 0045) behaviour.
closes#82
Audit (architect drift review + cargo test --workspace) clean: code matches the
spec, the engine stayed domain-free (no instrument identity in Ctx / the hot
path), InstrumentSpec sits beside the hot path per C7/C15. No regression scripts
configured (no-op gate); the full suite is green.
Ratify: the architect flagged one ledger-realization gap — C10/C15 described
per-instrument pip as standing design intent but carried no realization note for
the now-shipped channel. Added dated realization notes to C10 (the per-instrument
pip channel + the refuse-don't-guess honesty rule) and C15 (instrument specs as
metadata, first realized by InstrumentSpec/instrument_spec).
Retire the ephemeral cycle artefacts (spec 0054, plan 0054) per the active-cycle
lifecycle. Deferred follow-up: #98 (route the runnable GER40 examples through the
lookup).
refs #22
The data-server symbol stream carries only price bars, no instrument metadata.
So SimBroker was constructed with one global pip_size literal (0.0001, correct
only for 5-decimal FX), and `aura run --real <symbol>` emitted wrong-unit pip
equity for any non-FX instrument (AAPL.US, BTCUSD off by orders of magnitude).
Add the missing channel to *specify* per-instrument pip:
- aura-ingest: `InstrumentSpec { pip_size }` + `instrument_spec(symbol)` — a
typed, Rust-authored vetted lookup (NOT Aura.toml; the later Aura.toml schema
can subsume it). Seeded GER40/FRA40 = 1.0 (index points), EURUSD/GBPUSD/USDCAD
= 0.0001 (5-dp FX majors); None for an un-specced symbol.
- aura-cli: `sample_harness` and `sim_optimal_manifest` gain a `pip_size`
parameter; `run_sample_real` looks the pip up by symbol BEFORE any data access
and refuses (exit 2) an un-specced instrument rather than guessing — honest by
construction. The looked-up pip reaches both the broker divisor and the
manifest broker label (`format!`). Every synthetic caller passes the unchanged
0.0001, now named `SYNTHETIC_PIP_SIZE`. SimBroker (aura-std) is untouched.
Scope: narrowed to the CLI real path — the actual bug. The runnable GER40
examples already bake the correct 1.0 and are NOT buggy; centralizing them
through the lookup is deferred to #98.
Decisions (user-directed, recorded on #22): metadata channel = typed
InstrumentSpec at the source edge; un-specced real symbol refuses; seed =
GER40/FRA40 + FX majors; example refactor deferred. The spec went through the
grounding-check gate; the auto-sign panel surfaced the refusal-behaviour and
scope as genuine design decisions, escalated and resolved by the user.
Tests: instrument_spec unit; manifest pip rendering (1.0 -> "1", 0.0001
unchanged); non-gated CLI refusal (exit 2, no stdout leak, vetted symbol clears
the lookup); gated GER40 binary-boundary honesty (pip_size=1 in the emitted
manifest, deterministic). The pre-existing run_sample_real determinism test was
retargeted AAPL.US -> GER40 (AAPL.US is un-specced and would now refuse at the
lookup); the AAPL.US bounded-window streaming property still lives in
aura-ingest/tests/streaming_seam.rs (literal source, no spec lookup).
Verified: cargo build --workspace, clippy --all-targets -D warnings, and
cargo test --workspace all green; the gated GER40 path ran with local data.
closes#22
Cycle 0053 (multi-tap trace join on timestamp, #93) closes drift-clean.
Audit (architect drift review over 066638e..HEAD against docs/design + the
CLAUDE.md domain invariants):
- C8/C18 honoured: join_on_ts is a post-run reduction over recorded
(Timestamp, Vec<Scalar>) sink streams, a true sibling of summarize/f64_field
in the report layer — correct architectural home.
- C3 not violated: the join runs post-run over already-recorded output, not
in-graph; no as-of join smuggled into the DAG. C1 ts-uniqueness is a
documented precondition, not a runtime check — consistent with how the
engine leaves wiring-time invariants to contract.
- Option-per-side keeps the engine honest (reports presence; the consumer owns
the 0.0/-1/false defaults). The hand-rolled *_by_ts HashMaps are fully gone;
the two RED unit tests pin the #93 cardinality-misalignment + spine-drop.
- No drift, no debt, no ledger entry needs updating. Recommendation: carry on.
Regression gate: no dedicated regression scripts configured (optional per the
conventions); build/test/clippy --workspace --all-targets is the gate and ran
green — the gated GER40 byte-identity tests executed against the local archive
(not skipped), proving drain_trace's trace is byte-preserved across the
refactor.
Per the artifact lifecycle (committed while the cycle is live, git rm at cycle
close), the ephemeral spec and plan are retired here; their durable rationale
lives in the implementation commits 74324d1 / 35c5adc and the git history.
drain_trace now joins its four recording-sink taps through aura-engine's
join_on_ts instead of a hand-rolled per-tap HashMap, resolving the #93
zip-by-index panic at its source: the four taps fire at different
cardinalities (breakout one bar short via cold Delay(1); bars_since_open
filtered by Session), so a positional zip misaligns or panics. The BarTrace
mapping stays a thin layer with its caller-side defaults unchanged
(held->0.0, bars_since_open->-1, breakout->false) — the engine reports
presence, the consumer interprets absence. Behaviour is byte-preserved: the
gated GER40 tests (ger40_breakout_real, ger40_breakout_blueprint,
open_ohlc_seam) stay green against the real archive.
The Recorder doc gains a prose note on the multi-tap cadence rule (join on
ts, never by index), pointing at join_on_ts by name rather than rustdoc link
since aura-std does not depend on aura-engine.
Verified: cargo build/test/clippy --workspace --all-targets clean; the
_by_ts hand-rolled join is gone (acceptance grep empty).
closes#93
Ship a generic, spine-anchored join_on_ts helper + JoinedRow in the report
layer (sibling of summarize/f64_field), re-exported from lib.rs. It fuses N
recorded (Timestamp, Vec<Scalar>) tap streams on the recorded timestamp:
exactly one JoinedRow per spine entry, each side looked up by ts as
Some(row)/None where it did not fire. Option-per-side semantics keep the
engine honest — it reports presence; the consumer interprets absence (the
0.0/-1/false defaults are consumer truths, not engine ones). C8/C18 (post-run
reduction over recorded sink output), C3 (no in-graph join), C1 (one row per
ts, documented precondition).
Two unit tests pin the contract: cardinality-misalignment alignment (the #93
shape — a spine bar a side tap skipped resolves to None, not a zip-misalign)
and the spine-anchored drop of a side row at a non-spine ts. RED accepted on
the grounding-check record (both symbols were absent pre-cycle, so the tests
are RED-by-construction) and verified green here.
drain_trace's migration onto this helper + the Recorder doc note follow.
refs #93
Bite-sized execution plan for spec 0053: Task 1 the join_on_ts helper +
JoinedRow + two RED unit tests in aura-engine report.rs (re-exported from
lib.rs); Task 2 the behaviour-preserving drain_trace refactor onto the helper;
Task 3 the Recorder prose cadence/join note. Final gate is workspace
build/test/clippy.
refs #93
Ratified design for a reusable, spine-anchored join_on_ts helper in
aura-engine that fuses multi-tap Recorder traces on the recorded timestamp
(Option-per-side: the engine reports presence, the consumer interprets
absence). Refactors drain_trace onto it and documents the cadence rule on
the Recorder API. Resolves the #93 zip-by-index panic.
refs #93
Cycle 0052 (real-data source-open seam, Runway-1) closes drift-clean.
Audit (architect drift review over aedaa5d..HEAD against docs/design + the
CLAUDE.md domain invariants):
- C3 (one ms<->ns crossing) strengthened — the inverse is seam-private and the
only consumer-side divide left; every consumer ns_to_ms / unix_ms_to_epoch_ns
wrap is gone.
- C4 (O/H/L/C merge tie-break order) now in exactly one vetted home (open_ohlc);
the hand-spelled four-field loops are gone.
- C12 (one shared Arc<DataServer>/cache) preserved; C7 surface additive.
- No drift, no debt, no ledger entry needs updating. Recommendation: carry on.
Regression gate: no dedicated regression scripts configured (optional per the
conventions); build/test/clippy --workspace is the gate and ran green — the gated
open_ohlc_seam behaviour-preservation test executed against the local archive
(not skipped), proving byte-identity vs the ms-path.
Per the artifact lifecycle (committed while the cycle is live, git rm at cycle
close), the ephemeral spec and plan are retired here; their durable rationale
lives in the implementation commit 8c9a1b4 and the git history.
A real OHLC source now builds from aura-ingest alone, in the engine-native
epoch-ns Timestamp currency, with the one ms<->ns crossing owned by the seam.
Library surface (src/lib.rs), all additive — the ms-based `open` and
`unix_ms_to_epoch_ns` are untouched, so every existing call site is preserved
byte-for-byte:
- epoch_ns_to_unix_ms (private): the seam-owned inverse; consumers never
convert (#80).
- M1FieldSource::open_window: the Timestamp-window mirror of `open`, mapping
each bound through the private inverse and delegating to `open`.
- open_ohlc: the canonical OHLC bundle opener — four M1FieldSources in the fixed
open/high/low/close C4 merge order, sharing one Arc<DataServer> (one cache,
C12); the order lives in one vetted place (#92).
- default_data_server + `pub use data_server::{DataServer, DEFAULT_DATA_PATH}`:
a real-data source builds without naming the external data_server crate (#81).
Consumer migration: the shared breakout_real.rs helpers and every GER40
example/test move to the Timestamp-native surface — open_ohlc_sources removed,
utc_month_window_ms -> utc_month_window (Timestamp), report_from_trace retyped,
the per-consumer ns_to_ms hand-divides deleted, the direct data_server imports
replaced by the re-exports. ger40_breakout_compare.rs (which opened OHLC by its
own hand-loop, reached by neither retyped helper) is migrated by inspection.
Return type is Vec, not [_; 4]: Harness::run consumes Vec<Box<dyn Source>>, so
it feeds straight in; order-safety comes from the single helper either way.
Tests: a hermetic round-trip pinning the inverse; a gated behaviour-preservation
test (open_ohlc Timestamp path == ms-path open, bit-identical recorded series);
a hermetic absent-archive fixture pinning the file-level None contract and the
#81 re-export firewall.
Verified: cargo build --workspace --all-targets clean; cargo test --workspace
green (the gated GER40 tests ran against the local archive — open_ohlc_seam
proves byte-identity on real data, not a skip); clippy --all-targets -D warnings
clean. Acceptance greps pass: open_ohlc_sources gone, no consumer-side ns_to_ms,
no data_server import in any migrated GER40 consumer, epoch_ns_to_unix_ms private.
closes#80, #81, #92
Consolidate aura-ingest's OHLC source-open surface onto the engine-native
epoch-ns Timestamp currency: a private seam-owned inverse, a Timestamp-window
opener, a canonical OHLC bundle opener (fixed C4 O/H/L/C order), and a
DataServer/DEFAULT_DATA_PATH re-export + default_data_server convenience — so a
real-data source builds from aura-ingest alone, with the one ms<->ns crossing
owned by the seam. Behaviour-preserving (byte-identical RunReports).
Grounded against the tree via the ground-spec-0052 workflow (25/28 assertions
confirmed; compare.rs compiler-invisible site + AC1 scope fix applied).
refs #80, #81, #92
Drop the gitignore entries for both dirs; align the CLAUDE.md note from "git-ignored, local-only" to "git-tracked — committed while the cycle is live, removed (git rm) at cycle close". The durable record stays the ledger + git history; specs/plans are transient in the repo, present only for their own cycle.
docs/specs and docs/plans were retired (prior commit); the source/test
comments that cited them ("spec 0050 §4.1", "spec §Testing N", "per spec",
"the spec's ...") now point at nothing. Strip every such pointer while
preserving the technical substance, the design-ledger contract refs
(C1/C11/C20/C34/C12.1/...), and the Gitea issue refs (#41).
Comment/doc edits only across 20 files — no logic change; full workspace
suite green, clippy clean.
Per-cycle specs/plans are step-scoped: every file maps to already-shipped
code, their snippets drift hard against current APIs (InputSpec->PortSpec,
Sim->Harness, schema() removed, C16 zero-dep reversed, ascii-dag retired),
and a stale spec read as an API reference misleads agents into non-compiling
code. No durable knowledge lost — rationale lives in the ledger + code
rustdoc, history in git + the tracker.
- gitignore docs/specs/ and docs/plans/
- delete all 117 existing specs/plans (recoverable via git history)
- CLAUDE.md: specs/plans are local-only ephemeral artifacts
The residency probe was an inherent method on the concrete M1FieldSource, so a Box<dyn Source> consumer could not read it without downcasting (the adjacent friction noted in #95). Hoist it onto the Source trait as resident_records(&self) -> Option<usize> with a default of None, mirroring bounds()'s Option-for-unknown idiom: None = "does not report" (the eager VecSource, which has no streaming ring), distinct from Some(0) = "reports, holds none" (e.g. exhausted).
M1FieldSource overrides it to return Some(chunk len). Callers updated (streaming_seam, the two deep-dive fieldtest bins). Tests: a hermetic VecSource default-is-None check, and a &dyn Source probe in the gated residency test proving M1FieldSource residency is readable without a downcast.
refs #95
The C12 realization and the 0041 spec claimed streaming residency is O(one chunk) at the process level ("a 20-year window streams in the same memory as a one-day one"). That holds only for the aura source ring (M1FieldSource::resident_records(), bounded by one chunk and correctly tested); whole-process RSS grows O(records-touched) because data-server's FileCache retains each window's parsed chunks (~56 B/record) read-only for the pass. Reproduced: 6 MiB @ 1mo -> 173 MiB @ ~10y, linear.
That retention is the replay-many optimization (load-once; close+volume share one parse via the field-agnostic FileKey), not a leak, and no always-on eviction can bound a single forward pass without regressing it. This narrows the docs/comments to the true per-source-ring property and names the FileCache per-window retention as the real, replay-amortized process cost. The streaming_seam assert is unchanged (it was always the ring bound; only its labelling confused ring vs process). The deferred opt-in non-retaining streaming read path is parked as Brummel/data-server#2.
closes#95
data-server does no timezone math (delphi_to_unix_ms is pure arithmetic), so the
archive carries whatever the raw Pepperstone files contain, labelled as UTC. The
Session node's UTC->Berlin DST conversion is unit-tested, but the assumption that
the raw timestamps are themselves UTC was unverified — a broker-local archive
would shift the whole session and the error would jump across the DST boundary.
Verified empirically via the DAX cash-open volume spike (GER40): it lands at
Berlin 09:00 in BOTH seasons, at 07:00 UTC (summer, CEST, UTC+2) and 08:00 UTC
(winter, CET, UTC+1) — a clean +1h DST shift, the genuine-UTC signature. This
gated test pins it so a future archive/convention drift cannot silently corrupt
every session strategy's bar alignment.
Add a C20 realization recording the resolution of the GER40 deep-dive's #94
finding: a hand-wired FlatGraph is not a shippable strategy (no param_space, so
World-opaque); the canonical form is the Composite blueprint, the FlatGraph its
compiled substrate (C23). Pins the bar-period-as-structural-construction-arg
decision (spec 0051) to the structural-axis-vs-tuning-param split.
refs #94
The shipped Composite blueprint (ger40_breakout_blueprint) is now driven by the
World orchestration families on real GER40, with NO re-authoring — the #94
friction is gone:
- examples/ger40_breakout_sweep.rs — sweep the {entry_bar, exit_bar} grid built
directly from param_space() (best entry=3,exit=4 -> 214.0 over 2024).
- examples/ger40_breakout_walkforward.rs — non-degenerate walk_forward: a real
non-empty space optimized per window, chosen_params populated everywhere (#97).
- examples/ger40_breakout_compare.rs — one blueprint across two structural axes:
instrument (GER40 vs FRA40) and bar period (15m vs 30m — different strategies,
C34), no re-authoring.
- tests/ger40_breakout_world.rs — gated: sweep returns the ranked 9-point grid
over the two named params; walk_forward yields non-empty windows each with a
non-empty chosen_params (executable #97 proof).
No blueprint reshape needed — the factory returns a fresh (Composite, Taps) per
call, so the World families re-instantiate per grid point / window (the SMA-cross
pattern); baked recorders do not block multi-bootstrap.
closes#94, closes#96, closes#97
Author the shipped session-breakout as a reusable Composite blueprint via
GraphBuilder — the World-consumable form (param_space + bootstrap) — alongside
the hand-wired FlatGraph it reproduces. Applies the spec-0051 decisions:
- bar_period_minutes is a construction arg binding BOTH Resample.period_minutes
and Session's period, locked equal, so a sweep cannot desync them (#96);
- delay.lag is bound out (a structural constant, not a tuning knob — C34);
- the two EqConst targets (named entry_bar / exit_bar) are the ONLY params, so
param_space() == {entry_bar.target, exit_bar.target} — the genuine tuning
knobs, which makes walk_forward non-degenerate (#97).
The ger40_breakout_real example now bootstraps from the blueprint (still prints
-45.0 for 2024-09). New gated test ger40_breakout_blueprint.rs proves: (a) the
param_space is exactly the two targets (no lag, no period); (b) C23 — the
blueprint reproduces the FlatGraph bit-identically over real GER40 2024-09;
(c) determinism.
First increment of the milestone; the sweep / walk_forward / compare World-family
demos follow.
refs #94, #96, #97
Milestone spec: ship the GER40 session-breakout as a Composite blueprint so the
same strategy runs across backtest / sweep / walk_forward / compare without
re-authoring (the #94 fieldtest friction). Records the design decision that the
bar period is a structural axis (a construction arg binding both Resample and
Session, so a sweep cannot desync them — #96) and that the canonical shippable
strategy form is a Composite (FlatGraph is its compiled substrate). Authoring-
side; no engine-core change (the existing bind() + GraphBuilder + param_space()
layer already suffices).
refs #94, #96, #97
Public-API research deep-dive driving the just-shipped GER40 real-data
session-breakout demo (5b5f034) through the escalating sequence a serious
researcher attempts next: scale to multi-year, walk-forward, structural-param
sweep, multi-instrument compare. Standalone consumer crate (path-deps only;
public interface = ledger + glossary + example corpus + cargo doc; no
crates/*/src read). All runs against the real /mnt/tickdata/Pepperstone archive.
Findings: 3 bugs, 1 friction, 3 spec_gap, 1 working. Triaged to the tracker:
- spec_gap: the flagship breakout ships as a raw FlatGraph the World API cannot
consume (sweep / walk_forward / compare all forced a hand re-author to a
Composite, which reproduced the FlatGraph numbers EXACTLY) — refs #94.
- bug: process residency is O(window) (6->173 MiB across 1mo->10y) despite the
documented O(one-chunk) claim; the aura ring stays <=1024 but data-server
retention scales, and streaming_seam.rs only measures the ring — refs #95.
- bug: the bar-period is one knob split across Resample (tunable param) and
Session (baked) and desyncs silently to 0.0 pips; Delay.lag leaks into the
default param_space (the C34 deform-vs-tune discriminator) — refs #96.
- spec_gap: the walk_forward empty-space idiom for param-free strategies is
undefined — refs #97.
- spec_gap (no per-instrument pip registry) and friction (one-directional
ns->ms conversion) re-confirm the known #22 and #80.
- working: C1 determinism over 7y / 148k bars, the lazy streaming source, and
the sweep/WFO cores held under real multi-year load; the Composite re-author
was behaviour-preserving (C23).
Scratch consumer crate + run transcripts committed under fieldtests/ per the
fieldtest convention (matching cycle-0049). The crate path-deps the engine and
reads the real archive; its bins are the repros cited in the issues above.
Run the Phase-1 session-breakout node vocabulary over REAL GER40 M1 bars
from the data-server archive — the first real-data exercise of the shipped
strategy nodes (Resample/Delay/Gt/Session/EqConst/And/Latch + SimBroker).
It is a pure composition of shipped aura-std nodes (no project-specific
signal code), so it lives in the engine repo's examples/ carveout (C9), not
a separate consumer crate. The breakout FlatGraph is reused VERBATIM from
aura-engine/tests/ger40_breakout.rs; only the sources change — four real
M1FieldSource (open/high/low/close, in the fixed C4 merge order) replace the
synthetic VecSources, with pip_size = 1.0 (GER40 is an index).
- examples/ger40_breakout_real.rs — runnable demo over a Sept-2024 window:
prints the RunReport metrics plus a per-session entry/exit trace.
- examples/shared/breakout_real.rs — the single 13-node wiring (the proven
11 + two trace taps), shared by the example and the test via #[path] so it
is never duplicated.
- tests/ger40_breakout_real.rs — gated determinism test (mirrors
real_bars.rs): skips where the archive is absent; where present, asserts
finite metrics, binary Latch exposure (held in {0.0, 1.0}), and
bit-identical C1 reruns of both the report and the recorded series.
chrono / chrono-tz added as aura-ingest dev-dependencies (the Session
timezone and the UTC window bounds), test-only — never on the normal path.
Hand-wires all seven new aura-std nodes (Resample, Delay, Gt, Session, EqConst
x2, And, Latch) + SimBroker into a raw-index FlatGraph over one synthetic
Frankfurt session, pinning the whole vocabulary composed end-to-end:
- held = [0,0,1,1,0] over the 5 bar emissions: flat, latched bar3-close (the
strict breakout landing on session bar 3) through bar4, exit at bar5-close;
- equity = [0,0,0,3,8] pips (SimBroker lagged-exposure integration, pip=1.0);
- a no-entry control (bar3 close == bar2 high) pins the strict-> semantic;
- two disjoint runs byte-identical (C1).
Both spec traps exercised (bar6 rollover closes bar5; bars 1-2 warm Delay[1]).
Phase-1 C9 deliverable; build-step 8 (capstone) of milestone 'Strategy node
vocabulary I'.
closes#91
Maps ctx.now() (epoch-ns UTC) to a Frankfurt session bar index: emits
bars_since_open:i64 = (local wall-clock minutes past the 09:00 open) /
period_minutes, in tz-aware DST-correct local time. So local 09:45 reads 3 in
both CEST (UTC+2) and CET (UTC+1) — the close-instant convention (spec 0050
§4.1). Trigger is an f64 Any input (value ignored, wired from close15) so the
node fires once per completed bar. Open/tz/period are baked structural config,
not scalar params (a timezone is not a scalar, C11). Admits chrono/chrono-tz to
aura-std — vetted DST math, never hand-rolled (per-case dep policy). Last node;
build-step 7 of milestone 'Strategy node vocabulary I'.
closes#90
Aggregates a fine 4-field OHLC stream into period_minutes buckets (open=first,
high=max, low=min, close=last), emitting a completed bar ONLY on bucket rollover
(ctx.now() crossing into the next bucket) — C2: a bar is actionable only once
complete, partials are never emitted and the last partial bar is dropped (no EOF
flush). Four Barrier(0) f64 inputs, a 4-field f64 record output (the first
multi-field-output node in aura-std, mirroring the Ohlcv engine fixture). The bar
carries no timestamp; the engine stamps the close-instant. Build-step 6 of
milestone 'Strategy node vocabulary I'.
closes#89
Level-sensitive SR latch: held -> 1.0 on set, -> 0.0 on reset (reset-dominant),
sample-and-held across quiet cycles; None while both legs are cold. Emits f64
directly (1.0 long / 0.0 flat) so it feeds SimBroker's exposure slot with no
cast — the resolution of the bool->f64 seam (no separate Exposure node in the
strategy DAG). State persisted in the struct across cycles, like SimBroker.
Build-step 5 of milestone 'Strategy node vocabulary I'.
closes#88
Emits the value its series input carried 'lag' fired cycles ago, held in a
node-owned ring (lookbacks()=[1], the past in node state not the input column).
Warm-up is skip-emit (None for the first 'lag' cycles — never a fabricated
sentinel). Output is a pure function of pre-cycle state, the precondition for a
future engine to close a feedback loop (the RTL register C5 names). prevHigh15 =
Delay[1] on high15. Build-step 4 of milestone 'Strategy node vocabulary I'.
closes#87
Stateless bool x bool -> bool, out = (a && b). Computes entry = breakout &&
isBar3 in the session-breakout strategy. The bool-input twin of Gt; seed of the
logic family (Or/Not are separate node types, the operator is topology, never a
swept param). Build-step 3 of milestone 'Strategy node vocabulary I'.
closes#86
Stateless f64 x f64 -> bool, out = (a > b), STRICT: a == b emits false (a close
exactly equal to the previous bar's high is not a breakout). Computes
breakout = close15 > prevHigh15 in the session-breakout strategy. First
bool-emitting f64 comparator; the operator is topology (relational siblings are
separate types, never a swept param). Build-step 2 of milestone 'Strategy node
vocabulary I'.
closes#85
Stateless gate, out = (value == target): the i64->bool comparator the
session-breakout strategy needs to turn Session's bars_since_open into
isBar3/isBar5Close (two instances bound to 3 and 5). The operator is topology
(a concrete node type), only target is a knob (C11/C12). Build-step 1 of
milestone 'Strategy node vocabulary I'.
closes#84
Milestone 'Strategy node vocabulary I — temporal, logic, resample & session':
the 7 aura-std nodes (EqConst, Gt, And, Delay, Latch, Resample, Session) plus a
synthetic e2e fixture, driven by the GER40 15m session-breakout. Records the
design pass's cross-cutting decisions: close-instant timestamp convention,
Latch emits f64 (Exposure dropped from the DAG), the EqConst i64->bool gate,
the Resample-only Barrier(0) firing regime, and the build order. refs #84-#91.
The cycle-0047 typed-Scalar params migration (86746e3) changed each
RunManifest param value on the C18 flat runs store from a bare JSON
number (2.0) to an externally-tagged Scalar ({"F64":2.0}), but did not
migrate the persisted runs.jsonl. Registry::load could no longer
deserialize any pre-0047 line, so `aura runs list` / `runs rank` errored
out on a researcher's existing run history (bug F5, surfaced by the
"The World, part II" milestone fieldtest).
Fix: a localized read-side back-compat mirror (src/compat.rs). load()
now reads each line via RunReportRead, whose ScalarRead accepts BOTH the
tagged form AND a bare JSON number (coerced to Scalar::f64), then
converts into the canonical RunReport. The forward write path (append /
RunReport::to_json) is untouched — it still emits the tagged form. This
is a one-directional read widening, not a format change.
The secondary clause of #83 (parse error exits 0) was triaged as
not-a-bug: runs list/rank already exit 2 via load_runs_or_exit
(main.rs:742); the "(exit 0)" in the fieldtest transcript was a pipe
measurement artifact ($? read head's exit code, not the binary's).
RED test (committed 20511d5) now green; full workspace suite + clippy
green.
closes#83
RED test pinning the real defect of bug F5 (surfaced by the milestone
"The World, part II" fieldtest). Registry::load cannot deserialize a
runs.jsonl line written before the cycle-0047 typed-Scalar params
migration, where each param value is a bare JSON float (e.g.
["sma_cross.fast",2.0]) rather than the tagged {"F64":2.0}. The test
feeds a minimal autonomous legacy line and asserts it loads — the bare
float read back via the documented f64 coercion — instead of erroring at
the first param.
Fails RED at the diagnosed cause (Parse "expected value" on the bare
float). GREEN side (back-compat read, no change to the typed-Scalar
forward format) follows.
refs #83
Brummel