From 5bff7e0fb9c71cf1a5a00f68fe0263cc11b2a27c Mon Sep 17 00:00:00 2001 From: Brummel Date: Fri, 19 Jun 2026 00:59:54 +0200 Subject: [PATCH] =?UTF-8?q?audit(trace-charts):=20cycle=20close=20?= =?UTF-8?q?=E2=80=94=20C22=20realization=20note,=20retire=20ephemera?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cycle-close audit for the trace-charts visual face (#101, shipped across 8bb5256 persist + 15ee6d5 serve). Architect drift review (13941a2..HEAD) + the regression gate (cargo build/test/clippy --workspace, all exit 0, orchestrator-verified). What holds (carry-on): C14 clean — aura-engine/aura-core carry zero UI/HTML/uPlot knowledge; ColumnarTrace does struct->encoding only, file I/O in aura-registry, rendering in aura-cli. C1/C3 clean — persistence and serve-time alignment are post-run pure reductions (join_on_ts at serve time only, never in-graph). C7 clean — on-disk f64 columns + kinds-tagged base types, the documented coercion honest, no 5th scalar. Resolutions: - [fix] Added the C22 realization note now that the web-from-disk seam shipped (the amendment 3b56efb had deferred it to "when the seam ships"). - [ratify] The served page injects the chart SERIES but drops the run manifest (no run-context header) — a deliberate, defensible naive-viewer scope call (the manifest is on disk in index.json; a header matters most for the deferred families/comparison work, C21). The deviation from spec 0056 §1/§4 landed unflagged in 15ee6d5; it is now flagged here and tracked as a follow-up. refs #102 - [tidy] Retired this cycle's ephemera (spec 0056, plans 0055/0056) and the stale debris from the prior Runway close (spec 0055-fieldtest) per the ephemeral-artifact convention; the durable record is the ledger + git history. Cycle drift-clean (modulo the ratified, tracked manifest-header gap). refs #101 --- docs/design/INDEX.md | 16 + docs/plans/0055-trace-charts-persist.md | 769 ------------------ docs/plans/0056-trace-charts-serve.md | 583 ------------- .../0055-fieldtest-runway-real-data-seam.md | 173 ---- .../0056-trace-charts-persist-and-serve.md | 432 ---------- 5 files changed, 16 insertions(+), 1957 deletions(-) delete mode 100644 docs/plans/0055-trace-charts-persist.md delete mode 100644 docs/plans/0056-trace-charts-serve.md delete mode 100644 docs/specs/0055-fieldtest-runway-real-data-seam.md delete mode 100644 docs/specs/0056-trace-charts-persist-and-serve.md diff --git a/docs/design/INDEX.md b/docs/design/INDEX.md index c26ecde..2b4fb71 100644 --- a/docs/design/INDEX.md +++ b/docs/design/INDEX.md @@ -1009,6 +1009,22 @@ run records *many* streams (one per recording node) instead of exactly one row. Recorded streams are sparse and timestamped (a record per fired cycle, tagged `ctx.now()`), matching a trace of timestamped events (C18). No new contract; the `Harness` API change (observe removed, `run -> ()`) is recorded here. +**Realization (the web-from-disk visual face, #101).** The C14/C22 web seam shipped +(amendment 3b56efb): a recording run persists each drained tap as a columnar (SoA, +C7) `ColumnarTrace` to `runs/traces//` (`aura-registry::TraceStore`, beside the +run registry's `runs.jsonl`), reachable via `aura run [--macd] [--real …] +--trace `; `aura chart [--panels]` reads them back, aligns all taps on a +synthetic-union timestamp spine via the post-run `join_on_ts` (C3 — not in-graph; +C1 — pure, no live external call), and emits a static self-contained uPlot page +(vendored like `render_html`'s Graphviz-WASM). The engine stays headless (C14): +encoding lives in `aura-engine` (`ColumnarTrace`, struct→JSON only), file I/O in +`aura-registry`, rendering in `aura-cli` (`render_chart_html` + `chart-viewer.js`). +First cut only — overlay (per-series y-scale) / timestamp-aligned panels; the served +page injects the chart *series* but **not yet a run-context header** (the run manifest +is persisted on disk in `index.json` but deliberately not surfaced in this naive +viewer — a ratified scope call, follow-up filed). Families-comparison meta-views, a +local server, and replay-clock controls remain open (see "Open architectural +threads"). ### C23 — Graph compilation and behaviour-preserving optimisation **Guarantee.** The bootstrap (C19) is a **compilation**: it lowers a param-generic, diff --git a/docs/plans/0055-trace-charts-persist.md b/docs/plans/0055-trace-charts-persist.md deleted file mode 100644 index a40fcf1..0000000 --- a/docs/plans/0055-trace-charts-persist.md +++ /dev/null @@ -1,769 +0,0 @@ -# Trace charts — iteration 1 (persist) — Implementation Plan - -> **Parent spec:** `docs/specs/0056-trace-charts-persist-and-serve.md` -> -> **For agentic workers:** REQUIRED SUB-SKILL: use the `implement` skill to run -> this plan. Steps use `- [ ]` checkboxes for tracking. - -**Goal:** Persist a run's drained recorder taps to disk as columnar (SoA) JSON -under `runs/traces//`, reachable via `aura run [--macd] [--trace ]` and -`aura run --real ... [--trace ]`, leaving the default (no-`--trace`) run -byte-for-byte unchanged. (The serve/chart half is iteration 2.) - -**Architecture:** Three layers, each landing beside its existing siblings. -`ColumnarTrace` (a pure transpose of `&[(Timestamp, Vec)]` ↔ columnar f64, -kind-tagged) goes in `aura-engine/report.rs` next to `join_on_ts`/`summarize`. A -`TraceStore` (write/read a per-run directory of `.json` + `index.json`) goes in -a new `aura-registry/trace_store.rs`, mirroring the `families.jsonl` sibling-store -discipline. A CLI-local `persist_traces` helper threads into all three run forms via -a new `trace: Option<&str>` parameter. The engine gains no I/O (it only encodes); the -registry crate does the file I/O; the CLI wires it. - -**Tech Stack:** Rust; `serde`/`serde_json` (already deps of both crates); the -existing `Scalar`/`ScalarKind`/`Timestamp` types; `aura-engine`'s `RunManifest`; the -`std::process::Command`-driven `cli_run.rs` e2e harness. - ---- - -**Files this plan creates or modifies:** - -- Modify: `crates/aura-engine/src/report.rs:187` — add `ColumnarTrace` (struct + - `from_rows`/`to_rows` + two private coercion helpers) after `join_on_ts`, before - `#[cfg(test)]` at :189; add three unit tests inside the existing test module. -- Modify: `crates/aura-engine/src/lib.rs:64` — add `ColumnarTrace` to the - `pub use report::{…}` re-export. -- Create: `crates/aura-registry/src/trace_store.rs` — `TraceStore` + `RunTraces` + - `TraceStoreError` + three temp-dir unit tests. -- Modify: `crates/aura-registry/src/lib.rs:24` — add `mod trace_store;` + - `pub use trace_store::{RunTraces, TraceStore, TraceStoreError};`. -- Modify: `crates/aura-cli/src/main.rs` — imports (:17–26), `persist_traces` helper - (new, before `run_sample` at :158), `run_sample` (:158), `run_sample_real` (:193), - `parse_real_args` (:264), `run_macd` (:823), `USAGE` (:858), the argv arms - (:867–870), and the nine in-file test callers (:1099–1100, :1180–1181, :1215, - :1220, :1230–1239, :1248–1249). -- Test: `crates/aura-cli/tests/cli_run.rs` — one new e2e: `aura run --trace demo` - writes `index.json` + per-tap files; plain `aura run` writes none. - ---- - -### Task 1: `ColumnarTrace` (aura-engine) - -**Files:** -- Modify: `crates/aura-engine/src/report.rs` (add type + helpers before :189; tests in the existing `mod tests`) -- Modify: `crates/aura-engine/src/lib.rs:64` - -- [ ] **Step 1: Write the failing tests** - -Append these three tests inside the existing `#[cfg(test)] mod tests { … }` in -`crates/aura-engine/src/report.rs` (e.g. directly after the `join_on_ts_*` tests, -before the closing `}` of the module). They use `Scalar`/`ScalarKind`/`Timestamp`, -already imported by the test module via `use super::*;`: - -```rust - #[test] - fn columnar_trace_round_trips_f64_rows() { - let rows = vec![ - (Timestamp(2), vec![Scalar::f64(10.0)]), - (Timestamp(3), vec![Scalar::f64(20.0)]), - (Timestamp(4), vec![Scalar::f64(30.0)]), - ]; - let ct = ColumnarTrace::from_rows("equity", &[ScalarKind::F64], &rows); - assert_eq!(ct.tap, "equity"); - assert_eq!(ct.kinds, vec!["F64".to_string()]); - assert_eq!(ct.ts, vec![2, 3, 4]); - assert_eq!(ct.columns, vec![vec![10.0, 20.0, 30.0]]); - // to_rows is the inverse for f64 taps - assert_eq!(ct.to_rows(), rows); - } - - #[test] - fn columnar_trace_empty_rows_yields_empty_columns_sized_to_kinds() { - let ct = ColumnarTrace::from_rows("exposure", &[ScalarKind::F64], &[]); - assert_eq!(ct.ts, Vec::::new()); - assert_eq!(ct.columns, vec![Vec::::new()]); - assert!(ct.to_rows().is_empty()); - } - - #[test] - fn columnar_trace_coerces_non_f64_kinds_to_f64() { - let rows = vec![ - (Timestamp(1), vec![Scalar::i64(7), Scalar::bool(true)]), - (Timestamp(2), vec![Scalar::i64(9), Scalar::bool(false)]), - ]; - let ct = ColumnarTrace::from_rows("mix", &[ScalarKind::I64, ScalarKind::Bool], &rows); - assert_eq!(ct.kinds, vec!["I64".to_string(), "Bool".to_string()]); - assert_eq!(ct.columns, vec![vec![7.0, 9.0], vec![1.0, 0.0]]); - } -``` - -- [ ] **Step 2: Run the tests to verify they fail** - -Run: `cargo test -p aura-engine columnar_trace` -Expected: FAIL — compile error `cannot find type/function ColumnarTrace in this scope` (the type does not exist yet). - -- [ ] **Step 3: Write the implementation** - -Insert this immediately after `join_on_ts` ends (`report.rs:187`) and before -`#[cfg(test)]` (`report.rs:189`). `Scalar`/`ScalarKind`/`Timestamp` are already -imported at `report.rs:11`: - -```rust -/// One drained recorder tap in columnar (SoA) form: parallel arrays, one per -/// recorded column, plus the shared recorded-timestamp axis. Chart-ready (a column -/// is a series of numbers) and kind-tagged (C7) — values are coerced to f64 for -/// plotting; the base type survives in `kinds`. Pure: identical rows always encode -/// to the same value (C1). -#[derive(Clone, Debug, PartialEq, serde::Serialize, serde::Deserialize)] -pub struct ColumnarTrace { - pub tap: String, - pub kinds: Vec, - pub ts: Vec, - pub columns: Vec>, -} - -impl ColumnarTrace { - /// Transpose a drained tap's `(ts, row)` pairs into columns. An empty `rows` - /// yields empty `ts` and one empty column per kind. Each cell is coerced to f64: - /// f64 as-is, i64 as f64, bool 1.0/0.0, timestamp epoch as f64. - pub fn from_rows(tap: &str, kinds: &[ScalarKind], rows: &[(Timestamp, Vec)]) -> Self { - let ts: Vec = rows.iter().map(|(t, _)| t.0).collect(); - let mut columns: Vec> = vec![Vec::with_capacity(rows.len()); kinds.len()]; - for (_, row) in rows { - for (c, scalar) in row.iter().enumerate() { - columns[c].push(scalar_to_f64(*scalar)); - } - } - ColumnarTrace { - tap: tap.to_string(), - kinds: kinds.iter().map(kind_tag).collect(), - ts, - columns, - } - } - - /// Inverse for the serve/align path: rebuild `(ts, row)` pairs with each cell as - /// `Scalar::f64(columns[c][r])` — uniformly f64 (the on-disk store is f64 - /// columns; the base type survives only in `kinds`). A true value round-trip for - /// f64 taps; keeps the serve-side `as_f64` projection total. - pub fn to_rows(&self) -> Vec<(Timestamp, Vec)> { - self.ts - .iter() - .enumerate() - .map(|(r, &t)| { - let row = self.columns.iter().map(|col| Scalar::f64(col[r])).collect(); - (Timestamp(t), row) - }) - .collect() - } -} - -/// The four-kind tag string for a column (the on-disk `kinds` form; `ScalarKind` -/// derives no serde, so tags are plain strings). -fn kind_tag(kind: &ScalarKind) -> String { - match kind { - ScalarKind::F64 => "F64", - ScalarKind::I64 => "I64", - ScalarKind::Bool => "Bool", - ScalarKind::Timestamp => "Timestamp", - } - .to_string() -} - -/// Coerce a recorded scalar to the f64 a chart plots: f64 as-is, i64 as f64, -/// bool 1.0/0.0, timestamp epoch as f64. -fn scalar_to_f64(s: Scalar) -> f64 { - match s.kind() { - ScalarKind::F64 => s.as_f64(), - ScalarKind::I64 => s.as_i64() as f64, - ScalarKind::Bool => { - if s.as_bool() { - 1.0 - } else { - 0.0 - } - } - ScalarKind::Timestamp => s.as_ts().0 as f64, - } -} -``` - -- [ ] **Step 4: Re-export `ColumnarTrace`** - -In `crates/aura-engine/src/lib.rs:64`, change: - -```rust -pub use report::{f64_field, join_on_ts, summarize, JoinedRow, RunManifest, RunMetrics, RunReport}; -``` - -to: - -```rust -pub use report::{ - f64_field, join_on_ts, summarize, ColumnarTrace, JoinedRow, RunManifest, RunMetrics, RunReport, -}; -``` - -- [ ] **Step 5: Run the tests to verify they pass** - -Run: `cargo test -p aura-engine columnar_trace` -Expected: PASS — `columnar_trace_round_trips_f64_rows`, `columnar_trace_empty_rows_yields_empty_columns_sized_to_kinds`, `columnar_trace_coerces_non_f64_kinds_to_f64` all green (3 passed). - ---- - -### Task 2: `TraceStore` (aura-registry) - -**Files:** -- Create: `crates/aura-registry/src/trace_store.rs` -- Modify: `crates/aura-registry/src/lib.rs:24` - -- [ ] **Step 1: Register the new module** - -In `crates/aura-registry/src/lib.rs`, directly after `mod lineage;` (`lib.rs:24`) and -its `pub use lineage::{…}` block (`lib.rs:25–28`), add: - -```rust -mod trace_store; -pub use trace_store::{RunTraces, TraceStore, TraceStoreError}; -``` - -- [ ] **Step 2: Write the new file with its failing tests first** - -Create `crates/aura-registry/src/trace_store.rs` with ONLY the test module + the -`use` header below (the impl is added in Step 4, so the tests fail to compile = RED): - -```rust -//! The per-run trace store: a directory of columnar tap files under -//! `/traces//`, beside the run registry's `runs.jsonl`. One -//! subdirectory per run name; one `.json` (a `ColumnarTrace`) per tap; plus -//! `index.json` carrying the run's `RunManifest` and the tap order. Mirrors the -//! registry's directory-co-located sibling-store discipline; a missing run reads as -//! `NotFound` (treat-as-absent), never a panic. - -use std::fmt; -use std::fs; -use std::path::{Path, PathBuf}; - -use aura_engine::{ColumnarTrace, RunManifest}; -use serde::{Deserialize, Serialize}; - -#[cfg(test)] -mod tests { - use super::*; - use aura_core::{Scalar, ScalarKind, Timestamp}; - - fn temp_traces_root(name: &str) -> PathBuf { - let dir = - std::env::temp_dir().join(format!("aura-tracestore-{}-{}", std::process::id(), name)); - let _ = fs::remove_dir_all(&dir); - fs::create_dir_all(&dir).expect("create temp traces root"); - dir - } - - fn sample_manifest() -> RunManifest { - RunManifest { - commit: "c".to_string(), - params: vec![("p".to_string(), Scalar::f64(1.0))], - window: (Timestamp(1), Timestamp(5)), - seed: 0, - broker: "sim-optimal(pip_size=0.0001)".to_string(), - } - } - - fn sample_taps() -> Vec { - vec![ - ColumnarTrace::from_rows( - "equity", - &[ScalarKind::F64], - &[(Timestamp(1), vec![Scalar::f64(0.0)]), (Timestamp(2), vec![Scalar::f64(0.5)])], - ), - ColumnarTrace::from_rows( - "exposure", - &[ScalarKind::F64], - &[(Timestamp(1), vec![Scalar::f64(1.0)]), (Timestamp(2), vec![Scalar::f64(-1.0)])], - ), - ] - } - - #[test] - fn write_then_read_round_trips() { - let root = temp_traces_root("roundtrip"); - let store = TraceStore::open(&root); - let manifest = sample_manifest(); - let taps = sample_taps(); - store.write("demo", &manifest, &taps).expect("write"); - let back = store.read("demo").expect("read"); - assert_eq!(back.manifest, manifest); - assert_eq!(back.taps, taps); - assert!(root.join("traces/demo/index.json").exists()); - assert!(root.join("traces/demo/equity.json").exists()); - assert!(root.join("traces/demo/exposure.json").exists()); - let _ = fs::remove_dir_all(&root); - } - - #[test] - fn read_missing_run_is_not_found() { - let root = temp_traces_root("missing"); - let store = TraceStore::open(&root); - match store.read("nope") { - Err(TraceStoreError::NotFound(name)) => assert_eq!(name, "nope"), - other => panic!("expected NotFound, got {other:?}"), - } - let _ = fs::remove_dir_all(&root); - } - - #[test] - fn corrupt_tap_file_is_a_parse_error_naming_the_file() { - let root = temp_traces_root("corrupt"); - let store = TraceStore::open(&root); - store.write("demo", &sample_manifest(), &sample_taps()).expect("write"); - fs::write(root.join("traces/demo/equity.json"), "not json").expect("clobber"); - match store.read("demo") { - Err(TraceStoreError::Parse { file, .. }) => assert!(file.ends_with("equity.json")), - other => panic!("expected Parse, got {other:?}"), - } - let _ = fs::remove_dir_all(&root); - } -} -``` - -- [ ] **Step 3: Run the tests to verify they fail** - -Run: `cargo test -p aura-registry trace_store` -Expected: FAIL — compile error `cannot find type TraceStore / TraceStoreError / RunTraces` (impl not written yet). - -- [ ] **Step 4: Write the implementation** - -In `crates/aura-registry/src/trace_store.rs`, insert this BETWEEN the `use` header -and the `#[cfg(test)] mod tests` block: - -```rust -/// A per-run trace directory rooted at `/traces`. One subdir per run name. -pub struct TraceStore { - dir: PathBuf, -} - -/// A run's read-back traces: its manifest plus its taps in index (write) order. -pub struct RunTraces { - pub manifest: RunManifest, - pub taps: Vec, -} - -/// `index.json`: the run manifest + the tap names in write order (= read order). -#[derive(Serialize, Deserialize)] -struct Index { - manifest: RunManifest, - taps: Vec, -} - -impl TraceStore { - /// Bind to `/traces`. No I/O — directories are created on write. - pub fn open(runs_dir: impl AsRef) -> TraceStore { - TraceStore { dir: runs_dir.as_ref().join("traces") } - } - - /// Write one run's taps + index under `traces//`, creating directories as - /// needed. Each tap lands in `.json`; the manifest + tap order in - /// `index.json`. - pub fn write( - &self, - name: &str, - manifest: &RunManifest, - taps: &[ColumnarTrace], - ) -> Result<(), TraceStoreError> { - let run_dir = self.dir.join(name); - fs::create_dir_all(&run_dir)?; - for tap in taps { - let path = run_dir.join(format!("{}.json", tap.tap)); - let body = serde_json::to_string(tap).map_err(|source| TraceStoreError::Parse { - file: path.display().to_string(), - source, - })?; - fs::write(&path, body)?; - } - let index = Index { - manifest: manifest.clone(), - taps: taps.iter().map(|t| t.tap.clone()).collect(), - }; - let index_path = run_dir.join("index.json"); - let body = serde_json::to_string(&index).map_err(|source| TraceStoreError::Parse { - file: index_path.display().to_string(), - source, - })?; - fs::write(&index_path, body)?; - Ok(()) - } - - /// Read a run's index + taps back, in index order. A missing run directory (no - /// `index.json`) is `NotFound`, not a panic; a malformed file is `Parse{file,..}`. - pub fn read(&self, name: &str) -> Result { - let run_dir = self.dir.join(name); - let index_path = run_dir.join("index.json"); - let index_text = match fs::read_to_string(&index_path) { - Ok(t) => t, - Err(e) if e.kind() == std::io::ErrorKind::NotFound => { - return Err(TraceStoreError::NotFound(name.to_string())) - } - Err(e) => return Err(TraceStoreError::Io(e)), - }; - let index: Index = serde_json::from_str(&index_text).map_err(|source| { - TraceStoreError::Parse { file: index_path.display().to_string(), source } - })?; - let mut taps = Vec::with_capacity(index.taps.len()); - for tap_name in &index.taps { - let path = run_dir.join(format!("{tap_name}.json")); - let text = fs::read_to_string(&path)?; - let tap: ColumnarTrace = serde_json::from_str(&text).map_err(|source| { - TraceStoreError::Parse { file: path.display().to_string(), source } - })?; - taps.push(tap); - } - Ok(RunTraces { manifest: index.manifest, taps }) - } -} - -/// What can go wrong reading or writing the trace store. -#[derive(Debug)] -pub enum TraceStoreError { - /// An I/O error reading or writing a trace file. - Io(std::io::Error), - /// A trace/index file did not parse (the offending file named). - Parse { file: String, source: serde_json::Error }, - /// No recorded run of this name (no `index.json` under `traces//`). - NotFound(String), -} - -impl fmt::Display for TraceStoreError { - fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { - match self { - TraceStoreError::Io(e) => write!(f, "trace store i/o: {e}"), - TraceStoreError::Parse { file, source } => { - write!(f, "trace store parse error in {file}: {source}") - } - TraceStoreError::NotFound(name) => { - write!(f, "no recorded run '{name}' under runs/traces") - } - } - } -} - -impl std::error::Error for TraceStoreError {} - -impl From for TraceStoreError { - fn from(e: std::io::Error) -> Self { - TraceStoreError::Io(e) - } -} -``` - -- [ ] **Step 5: Run the tests to verify they pass** - -Run: `cargo test -p aura-registry trace_store` -Expected: PASS — `write_then_read_round_trips`, `read_missing_run_is_not_found`, -`corrupt_tap_file_is_a_parse_error_naming_the_file` all green (3 passed). - ---- - -### Task 3: CLI persist wiring (aura-cli) - -This task is **one compile unit**: the `trace: Option<&str>` parameter on three run -fns and the `parse_real_args` 3→4-tuple widen break every caller (including nine -in-file test callers), so all caller-threading lands here before the build gate. The -RED test (Step 1) is an e2e that compiles against the *binary*, not the changed -signatures, so it fails cleanly before any code change. - -**Files:** -- Test: `crates/aura-cli/tests/cli_run.rs` (new e2e) -- Modify: `crates/aura-cli/src/main.rs` (imports :17–26, helper, :158, :193, :264, :823, USAGE :858, argv :867–870, test callers :1099–1100/:1180–1181/:1215/:1220/:1230–1239/:1248–1249) - -- [ ] **Step 1: Write the failing e2e test** - -Append to `crates/aura-cli/tests/cli_run.rs` (it already has `BIN` + `temp_cwd`): - -```rust -#[test] -fn run_trace_persists_taps_and_plain_run_writes_no_traces() { - let cwd = temp_cwd("run-trace"); - - // plain `aura run` (no --trace) persists nothing to disk. - let plain = Command::new(BIN).arg("run").current_dir(&cwd).output().expect("spawn aura run"); - assert!(plain.status.success(), "plain run exit: {:?}", plain.status); - assert!(!cwd.join("runs/traces").exists(), "plain run must write no trace files"); - - // `aura run --trace demo` persists the two taps + index, and still prints the report. - let traced = Command::new(BIN) - .args(["run", "--trace", "demo"]) - .current_dir(&cwd) - .output() - .expect("spawn aura run --trace"); - assert!(traced.status.success(), "traced run exit: {:?}", traced.status); - let stdout = String::from_utf8(traced.stdout).expect("utf-8 stdout"); - assert!(stdout.trim_end().starts_with("{\"manifest\":{"), "report still printed: {stdout:?}"); - assert!(cwd.join("runs/traces/demo/index.json").exists(), "index.json written"); - assert!(cwd.join("runs/traces/demo/equity.json").exists(), "equity tap written"); - assert!(cwd.join("runs/traces/demo/exposure.json").exists(), "exposure tap written"); - - let _ = std::fs::remove_dir_all(&cwd); -} -``` - -- [ ] **Step 2: Run the e2e to verify it fails** - -Run: `cargo test -p aura-cli --test cli_run run_trace_persists_taps_and_plain_run_writes_no_traces` -Expected: FAIL — `traced run exit` assertion fails: `["run","--trace","demo"]` has no argv arm yet, so it falls to the usage-error path (exit 2), and no trace files are written. (The tree still compiles — no source change yet.) - -- [ ] **Step 3: Extend the imports** - -In `crates/aura-cli/src/main.rs`, add `ColumnarTrace` to the `aura_engine::{…}` -import (`main.rs:18`) and `TraceStore` to the `aura_registry::{…}` import -(`main.rs:25`): - -```rust -use aura_engine::{ - f64_field, monte_carlo, param_stability, summarize, walk_forward, window_of, ColumnarTrace, - Composite, Edge, FlatGraph, GraphBuilder, Harness, McAggregate, McFamily, RollMode, - RunManifest, RunReport, SourceSpec, SweepFamily, SyntheticSpec, Target, VecSource, - WalkForwardResult, WindowBounds, WindowRoller, WindowRun, -}; -use aura_registry::{ - group_families, mc_member_reports, optimize, rank_by, sweep_member_reports, - walkforward_member_reports, FamilyKind, Registry, TraceStore, -}; -``` - -(Only `TraceStore` is needed this iteration; `persist_traces` Display-formats the -error inline, so `TraceStoreError` is not named here — importing it would be an -unused import under `-D warnings`. Iteration 2's `emit_chart` adds it.) - -- [ ] **Step 4: Add the `persist_traces` helper** - -Insert directly before `fn run_sample` (`main.rs:158`): - -```rust -/// Persist a run's drained taps to the on-disk trace store under `runs/traces//` -/// (beside the run registry's `runs/`). Shared by every run form — all drain the same -/// two f64 taps (equity off the broker, exposure off the strategy) and carry a -/// `RunManifest`. Pure wiring over `ColumnarTrace` + `TraceStore`; the engine is -/// untouched. An I/O failure is a usage-level error (stderr + exit 2), per the spec. -fn persist_traces( - name: &str, - manifest: &RunManifest, - eq_rows: &[(Timestamp, Vec)], - ex_rows: &[(Timestamp, Vec)], -) { - let taps = vec![ - ColumnarTrace::from_rows("equity", &[ScalarKind::F64], eq_rows), - ColumnarTrace::from_rows("exposure", &[ScalarKind::F64], ex_rows), - ]; - if let Err(e) = TraceStore::open("runs").write(name, manifest, &taps) { - eprintln!("aura: trace persist failed: {e}"); - std::process::exit(2); - } -} -``` - -- [ ] **Step 5: Thread `trace` through `run_sample`** - -Replace `fn run_sample` (`main.rs:158–184`) with (signature gains `trace`, manifest -hoisted before the persist call, fold inlined after): - -```rust -fn run_sample(trace: Option<&str>) -> RunReport { - let (mut h, rx_eq, rx_ex) = sample_harness(SYNTHETIC_PIP_SIZE); - let sources: Vec> = - vec![Box::new(VecSource::new(synthetic_prices()))]; - let window = window_of(&sources).expect("non-empty synthetic stream"); - h.run(sources); - - let eq_rows: Vec<(Timestamp, Vec)> = rx_eq.try_iter().collect(); - let ex_rows: Vec<(Timestamp, Vec)> = rx_ex.try_iter().collect(); - let manifest = sim_optimal_manifest( - vec![ - ("sma_fast".to_string(), Scalar::i64(2)), - ("sma_slow".to_string(), Scalar::i64(4)), - ("exposure_scale".to_string(), Scalar::f64(0.5)), - ], - window, - 0, - SYNTHETIC_PIP_SIZE, - ); - if let Some(name) = trace { - persist_traces(name, &manifest, &eq_rows, &ex_rows); - } - let metrics = summarize(&f64_field(&eq_rows, 0), &f64_field(&ex_rows, 0)); - RunReport { manifest, metrics } -} -``` - -- [ ] **Step 6: Thread `trace` through `run_sample_real`** - -In `fn run_sample_real` (`main.rs:193–257`): change the signature to -`fn run_sample_real(symbol: &str, from_ms: Option, to_ms: Option, trace: Option<&str>) -> RunReport`, -then replace the drain-to-RunReport tail (`main.rs:238–256`) with: - -```rust - let eq_rows: Vec<(Timestamp, Vec)> = rx_eq.try_iter().collect(); - let ex_rows: Vec<(Timestamp, Vec)> = rx_ex.try_iter().collect(); - let manifest = sim_optimal_manifest( - vec![ - ("sma_fast".to_string(), Scalar::i64(2)), - ("sma_slow".to_string(), Scalar::i64(4)), - ("exposure_scale".to_string(), Scalar::f64(0.5)), - ], - window, - 0, - spec.pip_size, - ); - if let Some(name) = trace { - persist_traces(name, &manifest, &eq_rows, &ex_rows); - } - let metrics = summarize(&f64_field(&eq_rows, 0), &f64_field(&ex_rows, 0)); - RunReport { manifest, metrics } -} -``` - -- [ ] **Step 7: Widen `parse_real_args` to a 4-tuple with a `--trace` tail flag** - -Replace `fn parse_real_args` (`main.rs:264–283`) with: - -```rust -fn parse_real_args( - rest: &[&str], -) -> Result<(String, Option, Option, Option), String> { - let usage = || "run --real [--from ] [--to ] [--trace ]".to_string(); - let (symbol, mut tail) = match rest.split_first() { - Some((sym, t)) if !sym.is_empty() => (*sym, t), - _ => return Err(usage()), - }; - let mut from: Option = None; - let mut to: Option = None; - let mut trace: Option = None; - while let Some((flag, t)) = tail.split_first() { - let (value, t) = t.split_first().ok_or_else(usage)?; - match *flag { - "--from" if from.is_none() => from = Some(value.parse().map_err(|_| usage())?), - "--to" if to.is_none() => to = Some(value.parse().map_err(|_| usage())?), - "--trace" if trace.is_none() => trace = Some((*value).to_string()), - _ => return Err(usage()), - } - tail = t; - } - Ok((symbol.to_string(), from, to, trace)) -} -``` - -- [ ] **Step 8: Thread `trace` through `run_macd`** - -In `fn run_macd` (`main.rs:823–856`): change the signature to -`fn run_macd(trace: Option<&str>) -> RunReport`, then replace the drain-to-RunReport -tail (`main.rs:836–855`) with: - -```rust - let eq_rows: Vec<(Timestamp, Vec)> = rx_eq.try_iter().collect(); - let ex_rows: Vec<(Timestamp, Vec)> = rx_ex.try_iter().collect(); - let manifest = sim_optimal_manifest( - vec![ - ("ema_fast".to_string(), Scalar::i64(2)), - ("ema_slow".to_string(), Scalar::i64(4)), - ("ema_signal".to_string(), Scalar::i64(3)), - ("exposure_scale".to_string(), Scalar::f64(0.5)), - ], - window, - 0, - SYNTHETIC_PIP_SIZE, - ); - if let Some(name) = trace { - persist_traces(name, &manifest, &eq_rows, &ex_rows); - } - let metrics = summarize(&f64_field(&eq_rows, 0), &f64_field(&ex_rows, 0)); - RunReport { manifest, metrics } -} -``` - -- [ ] **Step 9: Update the argv arms** - -Replace the three run arms (`main.rs:867–875` — `["run"]`, `["run","--macd"]`, the -`["run","--real", rest @ ..]` block) with: - -```rust - ["run"] => println!("{}", run_sample(None).to_json()), - ["run", "--macd"] => println!("{}", run_macd(None).to_json()), - ["run", "--trace", name] => println!("{}", run_sample(Some(name)).to_json()), - ["run", "--macd", "--trace", name] => println!("{}", run_macd(Some(name)).to_json()), - ["run", "--real", rest @ ..] => match parse_real_args(rest) { - Ok((sym, from, to, trace)) => { - println!("{}", run_sample_real(&sym, from, to, trace.as_deref()).to_json()) - } - Err(msg) => { - eprintln!("aura: {msg}"); - std::process::exit(2); - } - }, -``` - -- [ ] **Step 10: Update `USAGE`** - -Replace the `USAGE` const (`main.rs:858–859`) with (one line; adds `[--trace ]` -on both run forms — the `chart` verb is added in iteration 2): - -```rust -const USAGE: &str = - "usage: aura run [--macd] [--trace ] | aura run --real [--from ] [--to ] [--trace ] | aura graph | aura sweep [--name ] | aura mc [--name ] | aura walkforward [--name ] | aura runs families | aura runs family [rank ]"; -``` - -- [ ] **Step 11: Migrate the nine in-file test callers** - -These break on the signature/tuple changes; update each in place: - -- `main.rs:1099–1100`: `run_macd()` → `run_macd(None)` (both lines). -- `main.rs:1248–1249`: `run_sample()` → `run_sample(None)` (both lines). -- `main.rs:1180`, `1181`, `1215`, `1220`: append `, None` as the 4th arg to each - `run_sample_real(SYMBOL_OR_"GER40", Some(..), Some(..))` call, e.g. - `run_sample_real(SYMBOL, Some(GER40_SEP2024_FROM_MS), Some(GER40_SEP2024_TO_MS), None)`. -- `parse_real_args` test (`main.rs:1229–1244`): replace the body with the 4-tuple - shape + a new `--trace` case: - -```rust - fn parse_real_args_accepts_symbol_and_optional_window() { - assert_eq!(parse_real_args(&["EURUSD"]), Ok(("EURUSD".to_string(), None, None, None))); - assert_eq!( - parse_real_args(&["EURUSD", "--from", "100", "--to", "200"]), - Ok(("EURUSD".to_string(), Some(100), Some(200), None)) - ); - // flags in any order - assert_eq!( - parse_real_args(&["EURUSD", "--to", "200", "--from", "100"]), - Ok(("EURUSD".to_string(), Some(100), Some(200), None)) - ); - // the --trace tail flag carries a string name, parsed alongside the window - assert_eq!( - parse_real_args(&["EURUSD", "--from", "100", "--trace", "demo"]), - Ok(("EURUSD".to_string(), Some(100), None, Some("demo".to_string()))) - ); - // a flag without its value, an empty symbol, and a non-numeric ms all reject - assert!(parse_real_args(&["EURUSD", "--from"]).is_err()); - assert!(parse_real_args(&[]).is_err()); - assert!(parse_real_args(&["EURUSD", "--from", "notanumber"]).is_err()); - } -``` - -- [ ] **Step 12: Build gate — workspace compiles, full aura-cli suite green** - -Run: `cargo build --workspace` -Expected: PASS — 0 errors (all callers threaded; no unresolved signature/tuple breaks). - -Run: `cargo test -p aura-cli` -Expected: PASS — the new e2e `run_trace_persists_taps_and_plain_run_writes_no_traces` -passes, the updated `parse_real_args_accepts_symbol_and_optional_window` passes, and -the byte-for-byte default-run pins (`run_prints_json_and_exits_zero`, -`run_with_trailing_token_is_strict_and_exits_two`) stay green (default run unchanged). - -- [ ] **Step 13: Workspace regression gate** - -Run: `cargo test --workspace` -Expected: PASS — Task 1 (`columnar_trace_*`) + Task 2 (`trace_store::tests::*`) + the -CLI suite all green; no regression elsewhere. - -Run: `cargo clippy --workspace --all-targets -- -D warnings` -Expected: PASS — no warnings. (Watch for: an unused import if `TraceStore`/`ColumnarTrace` -is imported but a step is skipped; a `clippy::needless_range_loop` on `from_rows`'s -index loop — if flagged, the `.enumerate()` form already shown avoids it.) diff --git a/docs/plans/0056-trace-charts-serve.md b/docs/plans/0056-trace-charts-serve.md deleted file mode 100644 index 5c53e2f..0000000 --- a/docs/plans/0056-trace-charts-serve.md +++ /dev/null @@ -1,583 +0,0 @@ -# Trace charts — iteration 2 (serve) — Implementation Plan - -> **Parent spec:** `docs/specs/0056-trace-charts-persist-and-serve.md` -> -> **For agentic workers:** REQUIRED SUB-SKILL: use the `implement` skill to run -> this plan. Steps use `- [ ]` checkboxes for tracking. - -**Goal:** Read a persisted run's traces (`runs/traces//`) and serve them as a -self-contained uPlot HTML chart via `aura chart [--panels]` — feeds overlaid -on a shared timestamp axis (default) or in timestamp-aligned stacked panels. - -**Architecture:** Three layers on top of iteration 1's on-disk artifact. A -first-party `chart-viewer.js` exposes a pure `buildCharts(traces, mode)` (the -`AURA_TRACES` → uPlot-config transform, no DOM) plus a browser-only `mount()`. -`render_chart_html` (aura-cli/render.rs) assembles a self-contained page mirroring -`render_html`: vendored uPlot (already checked in, commit d534f10) + `chart-viewer.js` -inlined via `include_str!`, the data injected as `window.AURA_TRACES`. `emit_chart` -(aura-cli/main.rs) reads the `TraceStore`, builds `ChartData` by the spec-§6 3-step -union-spine `join_on_ts` alignment, and prints the page. The engine is untouched (C14). - -**Tech Stack:** Rust (`aura-cli`); the iteration-1 building blocks -`aura_engine::{ColumnarTrace, join_on_ts, JoinedRow}` + `aura_registry::{TraceStore, -RunTraces, TraceStoreError}`; vendored uPlot 1.6.32; `serde_json` (already an -aura-cli dep); a `node` `.mjs` DOM guard mirroring `viewer_dot.rs`. - ---- - -**Files this plan creates or modifies:** - -- Create: `crates/aura-cli/assets/chart-viewer.js` — first-party `AURA_TRACES` → - uPlot bootstrap (pure `buildCharts` + browser `mount`). -- Create: `crates/aura-cli/tests/chart_viewer.rs` — `.rs` node-driver (shells `node`). -- Create: `crates/aura-cli/tests/chart_viewer_build.mjs` — the DOM-less guard over - `buildCharts`. -- Create: `crates/aura-cli/tests/fixtures/sample-traces.json` — an `AURA_TRACES` - fixture for the guard. -- Modify: `crates/aura-cli/src/render.rs` — add `ChartMode`/`ChartData`/`Series` + - `render_chart_html` + a self-contained unit test (mirror `render.rs:92`). -- Modify: `crates/aura-cli/src/main.rs` — imports (`:17–26`), `emit_chart` + the - `build_chart_data` alignment helper, the `["chart", …]` argv arms (`:885–914`, - before `["graph"]` at `:899`), `USAGE` (`:877–878`). -- Test: `crates/aura-cli/tests/cli_run.rs` — a `chart` e2e sibling (HTML emitted; - missing run exits 2). - -uPlot is already vendored (commit d534f10: `assets/uPlot.iife.min.js` + -`uPlot.min.css` + the `refresh-assets.sh` line) — no fetch in this plan. - ---- - -### Task 1: `chart-viewer.js` + its DOM guard - -**Files:** -- Create: `crates/aura-cli/assets/chart-viewer.js` -- Create: `crates/aura-cli/tests/chart_viewer_build.mjs` -- Create: `crates/aura-cli/tests/chart_viewer.rs` -- Create: `crates/aura-cli/tests/fixtures/sample-traces.json` - -- [ ] **Step 1: Write the fixture** - -Create `crates/aura-cli/tests/fixtures/sample-traces.json` (an `AURA_TRACES` payload -with two co-cadenced single-column series, matching what `render_chart_html` injects): - -```json -{ - "mode": "overlay", - "manifest": { "commit": "test", "params": [], "window": [1, 3], "seed": 0, "broker": "sim-optimal(pip_size=0.0001)" }, - "taps": ["equity", "exposure"], - "xs": [1, 2, 3], - "series": [ - { "name": "equity", "y_scale_id": "y_0", "points": [0.0, 0.001, 0.004] }, - { "name": "exposure", "y_scale_id": "y_1", "points": [1.0, 1.0, -1.0] } - ] -} -``` - -- [ ] **Step 2: Write the failing guard (.mjs + .rs driver)** - -Create `crates/aura-cli/tests/chart_viewer_build.mjs`: - -```js -// Headless guard for the trace-chart viewer (chart-viewer.js). Property protected: -// buildCharts maps an AURA_TRACES payload to uPlot configs WITHOUT a DOM — overlay -// is one chart with every series on its own y-scale over the shared xs; panels is -// one chart per series, each on the SAME shared xs (timestamp-aligned). It loads the -// real viewer module (a fix/regression there is observed here) and drives the -// exported pure `buildCharts`; uPlot itself (which needs `document`) is never -// instantiated — mount() is browser-only and not called here. - -import { createRequire } from "node:module"; -import { readFileSync } from "node:fs"; -import { fileURLToPath } from "node:url"; -import { dirname, join } from "node:path"; - -const require = createRequire(import.meta.url); -const here = dirname(fileURLToPath(import.meta.url)); -const traces = JSON.parse(readFileSync(join(here, "fixtures", "sample-traces.json"), "utf8")); - -const fail = (msg) => { console.error("FAIL: " + msg); process.exit(1); }; -const eq = (a, b, msg) => { if (a !== b) fail(`${msg} (got ${a}, want ${b})`); }; -const sameArr = (a, b, msg) => { - if (!Array.isArray(a) || !Array.isArray(b) || a.length !== b.length || a.some((v, i) => v !== b[i])) fail(msg); -}; - -// No `document` global -> chart-viewer.js's browser mount is skipped; require gives -// us only the pure export (mirrors viewer_dot_ids.mjs stubbing window before require). -global.window = { AURA_TRACES: traces }; -const { buildCharts } = require(join(here, "..", "assets", "chart-viewer.js")); - -const n = traces.series.length; - -// overlay: ONE chart, all series superimposed on the shared xs, each on its own scale. -const overlay = buildCharts(traces, "overlay"); -eq(overlay.length, 1, "overlay is one chart"); -eq(overlay[0].data.length, 1 + n, "overlay data = xs + N series"); -eq(overlay[0].opts.series.length, 1 + n, "overlay uPlot series = x + N"); -sameArr(overlay[0].data[0], traces.xs, "overlay x-axis is the shared xs"); -const scaleIds = overlay[0].opts.series.slice(1).map((s) => s.scale); -eq(new Set(scaleIds).size, n, "each overlay series has a distinct y-scale"); - -// panels: ONE chart per series, all timestamp-aligned on the SAME shared xs. -const panels = buildCharts(traces, "panels"); -eq(panels.length, n, "panels = one chart per series"); -for (const p of panels) { - eq(p.data.length, 2, "each panel = xs + its one series"); - sameArr(p.data[0], traces.xs, "panel shares the same xs (timestamp-aligned)"); -} - -console.log(`OK — overlay 1 chart / ${n} series, panels ${n} charts, shared xs.`); -process.exit(0); -``` - -Create `crates/aura-cli/tests/chart_viewer.rs` (mirrors `viewer_dot.rs`): - -```rust -//! Integration guard for the `aura chart` viewer's data transform. -//! -//! `chart-viewer.js` builds the uPlot configs in JavaScript, so the property -//! "buildCharts maps AURA_TRACES to the right per-mode uPlot config shape" cannot -//! be checked from Rust. This shells out to `node`, running -//! `tests/chart_viewer_build.mjs`, which loads the real viewer module and asserts -//! the overlay/panels config shape over a fixture — without instantiating uPlot -//! (which needs a DOM). `node` is REQUIRED: if absent this test FAILS. - -use std::path::PathBuf; -use std::process::Command; - -#[test] -fn chart_viewer_builds_overlay_and_panels_configs() { - let script = PathBuf::from(env!("CARGO_MANIFEST_DIR")) - .join("tests") - .join("chart_viewer_build.mjs"); - assert!(script.exists(), "guard script missing at {}", script.display()); - - let out = match Command::new("node").arg(&script).output() { - Ok(out) => out, - Err(e) => panic!( - "node is required for the chart-viewer guard but could not be run \ - ({e}); install Node.js or ensure `node` is on PATH" - ), - }; - - let stdout = String::from_utf8_lossy(&out.stdout); - let stderr = String::from_utf8_lossy(&out.stderr); - assert!( - out.status.success(), - "chart-viewer guard failed (exit {:?}).\n--- node stdout ---\n{stdout}\n--- node stderr ---\n{stderr}", - out.status.code() - ); -} -``` - -- [ ] **Step 3: Run the guard to verify it fails** - -Run: `cargo test -p aura-cli --test chart_viewer chart_viewer_builds_overlay_and_panels_configs` -Expected: FAIL — node cannot `require` `assets/chart-viewer.js` (it does not exist yet), so the guard exits non-zero and the `.rs` assert reports the node stderr. - -- [ ] **Step 4: Write `chart-viewer.js`** - -Create `crates/aura-cli/assets/chart-viewer.js`: - -```js -// The `aura chart` viewer: turns the injected window.AURA_TRACES (xs + per-series -// Option points, already timestamp-union-aligned by the CLI) into uPlot charts. -// -// buildCharts is PURE (no DOM, no uPlot) and is the unit the headless guard drives. -// mount() is browser-only — it instantiates the vendored global `uPlot` per config. -(function () { - var PALETTE = ["#89b4fa", "#f38ba8", "#a6e3a1", "#f9e2af", "#cba6f7", "#94e2d5"]; - - // {xs, series:[{name, y_scale_id, points}]} + mode -> array of {opts, data} configs. - // overlay -> one chart, all series, each on its own auto-ranged y-scale (shared xs). - // panels -> one chart per series, each on the SAME shared xs (timestamp-aligned). - function buildCharts(traces, mode) { - var xs = traces.xs; - var series = traces.series; - if (mode === "panels") { - return series.map(function (s, i) { - return { - opts: { - title: s.name, - scales: { x: { time: false }, [s.y_scale_id]: {} }, - series: [{}, { label: s.name, scale: s.y_scale_id, stroke: PALETTE[i % PALETTE.length] }], - axes: [{}, { scale: s.y_scale_id }], - }, - data: [xs, s.points], - }; - }); - } - // overlay (default) - var scales = { x: { time: false } }; - var uSeries = [{}]; - var axes = [{}]; - series.forEach(function (s, i) { - scales[s.y_scale_id] = {}; - uSeries.push({ label: s.name, scale: s.y_scale_id, stroke: PALETTE[i % PALETTE.length] }); - // the first two series carry a drawn axis (left, then right); the rest keep - // their own auto-scale, read via the legend. - if (i < 2) axes.push({ scale: s.y_scale_id, side: i === 0 ? 3 : 1 }); - }); - return [{ opts: { scales: scales, series: uSeries, axes: axes }, data: [xs].concat(series.map(function (s) { return s.points; })) }]; - } - - // Browser-only: instantiate the vendored global uPlot per config into #stage. - function mount() { - var traces = window.AURA_TRACES; - var configs = buildCharts(traces, traces.mode); - var stage = document.getElementById("stage"); - var w = stage.clientWidth || 900; - var h = configs.length > 1 ? Math.max(160, Math.floor(440 / configs.length)) : 460; - configs.forEach(function (cfg) { - var opts = Object.assign({ width: w, height: h }, cfg.opts); - // eslint-disable-next-line no-undef - new uPlot(opts, cfg.data, stage); - }); - } - - if (typeof window !== "undefined" && typeof document !== "undefined" && window.AURA_TRACES) { - if (document.readyState === "loading") document.addEventListener("DOMContentLoaded", mount); - else mount(); - } - if (typeof module !== "undefined" && module.exports) module.exports = { buildCharts: buildCharts }; -})(); -``` - -- [ ] **Step 5: Run the guard to verify it passes** - -Run: `cargo test -p aura-cli --test chart_viewer chart_viewer_builds_overlay_and_panels_configs` -Expected: PASS — `OK — overlay 1 chart / 2 series, panels 2 charts, shared xs.` (1 passed). - ---- - -### Task 2: `render_chart_html` + chart types (render.rs) - -**Files:** -- Modify: `crates/aura-cli/src/render.rs` (types + fn after `render_html`; test in `mod tests`) - -- [ ] **Step 1: Write the failing self-contained test** - -Append inside the existing `#[cfg(test)] mod tests { … }` in `crates/aura-cli/src/render.rs`: - -```rust - fn sample_chart_data() -> ChartData { - ChartData { - manifest: aura_engine::RunManifest { - commit: "test".into(), - params: vec![], - window: (aura_engine::Timestamp(1), aura_engine::Timestamp(3)), - seed: 0, - broker: "sim-optimal(pip_size=0.0001)".into(), - }, - taps: vec!["equity".into(), "exposure".into()], - xs: vec![1, 2, 3], - series: vec![ - Series { name: "equity".into(), y_scale_id: "y_0".into(), points: vec![Some(0.0), Some(0.001), Some(0.004)] }, - Series { name: "exposure".into(), y_scale_id: "y_1".into(), points: vec![Some(1.0), Some(1.0), Some(-1.0)] }, - ], - } - } - - #[test] - fn render_chart_html_is_self_contained_and_injects_traces() { - let html = render_chart_html(&sample_chart_data(), ChartMode::Overlay); - assert!(html.starts_with(""), "not an HTML doc"); - assert!(html.trim_end().ends_with(""), "HTML doc not closed"); - // self-contained: no remote asset - assert!(!html.contains("\n\n\n\n\n"); - html -} -``` - -(`serde::Serialize` on `Series` lets `serde_json::json!` embed it; `RunManifest` -already derives `Serialize`. `serde_json` is already an aura-cli dependency — used by -`main.rs`'s `runs_family` JSON.) - -- [ ] **Step 4: Run the test to verify it passes** - -Run: `cargo test -p aura-cli --lib render_chart_html` -Expected: PASS — `render_chart_html_is_self_contained_and_injects_traces` + -`render_chart_html_reflects_panels_mode` green (2 passed). - ---- - -### Task 3: `emit_chart` + serve-time alignment + the `chart` verb (main.rs) - -This task is one compile unit (new `emit_chart`/`build_chart_data` + two argv arms + -imports + USAGE); it changes no existing signature, so there is no caller blast -radius. The RED e2e (Step 1) compiles against the binary and fails before the arms -exist. - -**Files:** -- Test: `crates/aura-cli/tests/cli_run.rs` (chart e2e sibling) -- Modify: `crates/aura-cli/src/main.rs` (imports `:17–26`, USAGE `:877–878`, argv - `:885–914`, new fns) - -- [ ] **Step 1: Write the failing e2e** - -Append to `crates/aura-cli/tests/cli_run.rs`: - -```rust -#[test] -fn chart_emits_self_contained_html_for_a_persisted_run() { - let cwd = temp_cwd("chart-serve"); - - // persist a run first (iteration 1), then chart it. - let traced = Command::new(BIN).args(["run", "--trace", "demo"]).current_dir(&cwd).output().expect("spawn run --trace"); - assert!(traced.status.success(), "run --trace exit: {:?}", traced.status); - - let chart = Command::new(BIN).args(["chart", "demo"]).current_dir(&cwd).output().expect("spawn chart"); - assert!(chart.status.success(), "chart exit: {:?}", chart.status); - let html = String::from_utf8(chart.stdout).expect("utf-8 html"); - assert!(html.starts_with(""), "not an HTML doc: {:?}", &html[..html.len().min(40)]); - assert!(html.contains("window.AURA_TRACES = {"), "AURA_TRACES not injected"); - assert!(html.contains("\"equity\""), "equity series missing from the chart"); - assert!(html.contains("uPlot=function"), "vendored uPlot not inlined"); - - // a missing run is a usage error (stderr + exit 2), not a panic. - let missing = Command::new(BIN).args(["chart", "nope"]).current_dir(&cwd).output().expect("spawn chart nope"); - assert_eq!(missing.status.code(), Some(2), "missing run must exit 2"); - assert!(!missing.status.success()); - - let _ = std::fs::remove_dir_all(&cwd); -} -``` - -- [ ] **Step 2: Run the e2e to verify it fails** - -Run: `cargo test -p aura-cli --test cli_run chart_emits_self_contained_html_for_a_persisted_run` -Expected: FAIL — `["chart","demo"]` has no argv arm yet, so it hits the usage-error path (exit 2) and prints no HTML. (Tree compiles — only a new test added.) - -- [ ] **Step 3: Extend the imports** - -In `crates/aura-cli/src/main.rs`, add `join_on_ts, JoinedRow` to the -`aura_engine::{…}` import (`main.rs:17–22`) and `RunTraces, TraceStoreError` to the -`aura_registry::{…}` import (`main.rs:23–26`). Also bring in the new render types: -the file already has `mod render;`, so reference them as `render::{ChartData, ChartMode, Series}` -at the call sites (no `use` needed) — OR add `use render::{ChartData, ChartMode, Series};` -after `mod render;` (`main.rs:14`). Use the explicit `use`: - -```rust -mod render; -use render::{ChartData, ChartMode, Series}; -``` - -and the two engine/registry import lists become (only the added names shown in context): - -```rust -use aura_engine::{ - f64_field, join_on_ts, monte_carlo, param_stability, summarize, walk_forward, window_of, - ColumnarTrace, Composite, Edge, FlatGraph, GraphBuilder, Harness, JoinedRow, McAggregate, - McFamily, RollMode, RunManifest, RunReport, SourceSpec, SweepFamily, SyntheticSpec, Target, - VecSource, WalkForwardResult, WindowBounds, WindowRoller, WindowRun, -}; -use aura_registry::{ - group_families, mc_member_reports, optimize, rank_by, sweep_member_reports, - walkforward_member_reports, FamilyKind, Registry, RunTraces, TraceStore, TraceStoreError, -}; -``` - -- [ ] **Step 4: Add the alignment helper + `emit_chart`** - -Insert after `persist_traces` (it ends near `main.rs:174`): - -```rust -/// Build the serve-ready `ChartData` from a run's read-back traces by the spec-§6 -/// 3-step union-spine alignment — no tap privileged, no point dropped: -/// (1) xs = the sorted, deduped union of every tap's timestamps; -/// (2) synthesize an empty-payload spine over xs and pass ALL taps (via -/// `ColumnarTrace::to_rows`, which yields uniformly-f64 rows) as symmetric sides -/// of `join_on_ts`, so no side row is dropped and none occupies the privileged -/// `JoinedRow.spine`; -/// (3) flatten each (tap, column) to a `Series` of `Option` over xs. -fn build_chart_data(traces: RunTraces) -> ChartData { - let mut xs: Vec = traces.taps.iter().flat_map(|t| t.ts.iter().copied()).collect(); - xs.sort_unstable(); - xs.dedup(); - - let spine: Vec<(Timestamp, Vec)> = xs.iter().map(|&t| (Timestamp(t), Vec::new())).collect(); - let tap_rows: Vec)>> = traces.taps.iter().map(|t| t.to_rows()).collect(); - let sides: Vec<&[(Timestamp, Vec)]> = tap_rows.iter().map(|r| r.as_slice()).collect(); - let joined: Vec = join_on_ts(&spine, &sides); - - let mut series: Vec = Vec::new(); - for (i, tap) in traces.taps.iter().enumerate() { - for c in 0..tap.columns.len() { - let name = if tap.columns.len() == 1 { tap.tap.clone() } else { format!("{}[{c}]", tap.tap) }; - let y_scale_id = format!("y_{}", series.len()); - let points: Vec> = - joined.iter().map(|r| r.sides[i].as_ref().map(|row| row[c].as_f64())).collect(); - series.push(Series { name, y_scale_id, points }); - } - } - - ChartData { - manifest: traces.manifest, - taps: traces.taps.iter().map(|t| t.tap.clone()).collect(), - xs, - series, - } -} - -/// `aura chart [--panels]`: read the persisted run's traces, align them, and -/// print the self-contained uPlot page. A missing run is a usage error (stderr + -/// exit 2), per the CLI's convention — never a panic. -fn emit_chart(name: &str, mode: ChartMode) { - let traces = match TraceStore::open("runs").read(name) { - Ok(t) => t, - Err(TraceStoreError::NotFound(n)) => { - eprintln!("aura: no recorded run '{n}' under runs/traces (run `aura run --trace {n}` first)"); - std::process::exit(2); - } - Err(e) => { - eprintln!("aura: {e}"); - std::process::exit(2); - } - }; - print!("{}", render::render_chart_html(&build_chart_data(traces), mode)); -} -``` - -- [ ] **Step 5: Add the argv arms** - -In `main()`'s argv match, insert the two chart arms directly before the `["graph"]` -arm (`main.rs:899`): - -```rust - ["chart", name] => emit_chart(name, ChartMode::Overlay), - ["chart", name, "--panels"] => emit_chart(name, ChartMode::Panels), -``` - -- [ ] **Step 6: Update `USAGE`** - -In the `USAGE` const (`main.rs:877–878`), add ` | aura chart [--panels]` after -the `aura graph` clause so every dispatched verb is advertised: - -```rust -const USAGE: &str = - "usage: aura run [--macd] [--trace ] | aura run --real [--from ] [--to ] [--trace ] | aura chart [--panels] | aura graph | aura sweep [--name ] | aura mc [--name ] | aura walkforward [--name ] | aura runs families | aura runs family [rank ]"; -``` - -- [ ] **Step 7: Run the e2e to verify it passes** - -Run: `cargo test -p aura-cli --test cli_run chart_emits_self_contained_html_for_a_persisted_run` -Expected: PASS — the persisted run charts to self-contained HTML carrying the equity -series + inlined uPlot; `aura chart nope` exits 2 (1 passed). - -- [ ] **Step 8: Build + lint + workspace gate** - -Run: `cargo build --workspace` -Expected: PASS — 0 errors. - -Run: `cargo test --workspace` -Expected: PASS — Task 1 (`chart_viewer`), Task 2 (`render_chart_html_*`), Task 3 -(`chart_emits_*`) green, and no regression (the iteration-1 trace tests + the -byte-for-byte default-run pins stay green). - -Run: `cargo clippy --workspace --all-targets -- -D warnings` -Expected: PASS — no warnings. (Watch for: a `clippy::needless_range_loop` on the -`for c in 0..tap.columns.len()` loop — if flagged, it reads `tap.columns[c]` only via -the parallel `joined[..].sides[i]` index, so rewrite as `.iter().enumerate()` over -`tap.columns` keeping `c`; an unused import if a name is mis-listed.) diff --git a/docs/specs/0055-fieldtest-runway-real-data-seam.md b/docs/specs/0055-fieldtest-runway-real-data-seam.md deleted file mode 100644 index 64e5bdd..0000000 --- a/docs/specs/0055-fieldtest-runway-real-data-seam.md +++ /dev/null @@ -1,173 +0,0 @@ -# Fieldtest — milestone Runway (real-data ergonomics & honesty hardening) — 2026-06-18 - -**Status:** Draft — awaiting orchestrator triage -**Author:** fieldtester (dispatched by fieldtest skill, milestone-close gate) - -## Scope - -Milestone-close fieldtest for **Runway — real-data ergonomics & honesty -hardening** (commit range `aedaa5d..HEAD`, HEAD `0d185fb`). The milestone's -promise: *a downstream consumer can run an honest real-data backtest end to -end* — open the real GER40 M1 archive via the canonical opener from -`aura-ingest` alone, run a strategy against real bars, get a PnL scaled by the -**correct per-instrument pip** (an index must not be mis-scaled as forex), be -**refused with a clear diagnostic** when the instrument has no vetted pip spec, -join multi-tap sink traces honestly on timestamp, and find the run-registry -surface coherent (the dead-end `runs list`/`rank` retired; `runs families`/ -`family` live; the two-store Registry discoverable from the public docs). - -Scenarios were derived **top-down from the promise**, not as a union of the -per-cycle axes. Public interface only: design ledger (`docs/design/INDEX.md`), -`docs/glossary.md`, `docs/project-layout.md`, `cargo doc` rustdoc, `git log`, -`CLAUDE.md`. No `crates/*/src` (incl. the engine's own -`crates/aura-ingest/examples/`, which sits under the forbidden code root) was -read. Built and ran HEAD from the working tree throughout: workspace built with -`cargo build --workspace`; every fixture/CLI run via `cargo run` so HEAD source -is always what executes. The two Rust fixtures path-dep the engine crates as a -real C16 project would; the CLI scenarios invoke the `aura` binary directly. - -## Examples - -### `fieldtests/milestone-runway/out_rw_1_pip_honesty.txt` — per-instrument pip honesty (#22) -- CLI invocations of `aura run --real` on a vetted index and on un-vetted - instruments. -- Fits the scope: this is the headline honesty fix — "symbol data carries no - pip metadata, so a real run guessed". A consumer must get the right pip on a - vetted symbol and a refusal on an un-vetted one. -- Outcome: ran. `GER40` → exit 0, manifest broker `sim-optimal(pip_size=1)` - (correct index point, not the FX `0.0001`), `total_pips=977.38`. `AAPL.US` - (un-specced) and `FOOBAR` (un-specced + no archive) → **exit 2**, stderr - diagnostic, **no stdout leak**. The refusal fires at the pip lookup *before* - data access (FOOBAR, which has no archive, gets the pip-refusal diagnostic, - not a data-not-found one). Matched expected. - -### `fieldtests/milestone-runway/rw_2_open_ohlc_real.rs` — real GER40 M1 ingestion via the canonical opener (#80/#81/#92) -- A consumer opens the real GER40 archive with `aura_ingest::{default_data_server, - DEFAULT_DATA_PATH, open_ohlc}`, wires the four OHLC sources into a 4-source-role - harness (SMA-cross-on-close → Exposure → SimBroker(pip 1.0)), and runs to - completion over real bars. -- Fits the scope: it exercises the "build a real-data source from `aura-ingest` - alone" promise (#81), the canonical 4-source merge order (#92), and the - timestamp-window open (#80) in one realistic task. -- Outcome: built (after one fix — see the `Cell` constructor friction) and ran. - `open_ohlc` returned exactly 4 sources O/H/L/C, each reporting `bounds()` - without materializing; 53173 real bars streamed on all four taps; OHLC - ordering (`low<=open<=high`) held on **53173/53173** bars; close-driven equity - produced (`+314.46` pips at pip 1.0). The crate has **no `data-server` - dependency** — the #81 crutch the prior milestone-18 fixture needed is gone. - Matched expected. - -### `fieldtests/milestone-runway/rw_3_multitap_ts_join.rs` — multi-tap trace honesty (#93) -- A consumer records two taps (exposure + cumulative equity) off the same real - GER40 run — the taps fire at **different cadences** — and fuses them with - `aura_engine::join_on_ts` / `JoinedRow`. -- Fits the scope: directly the #93 promise — "join multi-tap sink traces - honestly on timestamp, not by index." -- Outcome: built and ran. Exposure fired 27725×, equity 27744× (differ by 19); - `join_on_ts` produced exactly one `JoinedRow` per spine entry (27744); a - by-timestamp cross-check of every `Some`/`None` side against the exposure - stream found **0 mismatches**; the first rows are the warm-up bars where - exposure had not yet fired (`None`) — precisely the cardinality-misalignment a - positional zip would corrupt. Matched expected. - -### `fieldtests/milestone-runway/out_rw_4_runs_coherence.txt` — run-registry coherence (#73/#82) -- CLI invocations confirming `runs list`/`rank` are retired and - `runs families`/`runs family … rank` work, plus a rustdoc check of the - two-store `Registry::open`. -- Fits the scope: the registry-honesty half of the promise. -- Outcome: ran. `runs list` and `runs rank` → **exit 2** with the usage line - (clean diagnostic; both absent from the usage string). `runs families` → - exit 0 listing the family; `runs family rank total_pips` → exit 0 with - ranked members. The `Registry::open` rustdoc documents the two - directory-co-located stores and the per-directory-isolation consequence (#82). - Matched expected — with one edge observation (bogus family id, below). - -## Findings - -### [working] Per-instrument pip honesty is correct and refuses cleanly -- Example: `out_rw_1_pip_honesty.txt`. -- Vetted `GER40` emits `sim-optimal(pip_size=1)` in the manifest (index point, - not the FX `0.0001`); un-vetted `AAPL.US`/`FOOBAR` exit 2 with a stderr - diagnostic that names the symbol and the fix ("add it to the instrument - table") and **no stdout leak**. The refusal gates at the lookup before any - data access. This is the milestone's headline honesty win, working end to end. -- Recommended action: carry on. - -### [working] A real-data source builds from `aura-ingest` alone — the `data-server` crutch is gone -- Example: `rw_2_open_ohlc_real.rs` + its Cargo.toml. -- `default_data_server()`, `DEFAULT_DATA_PATH`, `DataServer` (re-export), and - `open_ohlc` are all reachable from `aura-ingest`; the fixture crate carries - **no** `data-server` dependency. The prior milestone-18 fixture had to pull - `data-server` as a git dep because `M1FieldSource::open` wanted - `&Arc` and aura-ingest re-exported neither it nor the path. #81 - closes that. `open_ohlc` vouches for the C4 O/H/L/C order so the consumer - never spells it out (#92); 53173 real bars streamed to completion. -- Recommended action: carry on. - -### [working] Multi-tap traces fuse honestly on timestamp -- Example: `rw_3_multitap_ts_join.rs`. -- Two real-data taps at genuinely different cadences (Δ19 rows) join to exactly - one `JoinedRow` per spine entry, every `Some`/`None` side aligning to the - source stream **by timestamp** (0 by-ts mismatches over 27744 rows). The - Option-per-side semantics surfaced the warm-up `None` rows exactly where a - zip-by-index would have silently shifted every later row. The `Recorder` - rustdoc states the cadence rule ("join on the recorded timestamp") at the - point of use, so a consumer is steered to `join_on_ts` rather than a zip. -- Recommended action: carry on. - -### [working] Run-registry surface is coherent; two-store behaviour is documented -- Example: `out_rw_4_runs_coherence.txt`. -- `runs list`/`rank` are gone from the usage string and exit 2; `runs families`/ - `runs family … rank` work and emit JSON. `Registry::open`'s rustdoc documents - the two directory-co-located stores (`runs.jsonl` + the fixed-name - `families.jsonl` sibling) and the per-directory (not per-filename) isolation - consequence at the bind point (#82) — exactly the "discoverable from the - public docs" half of the promise. -- Recommended action: carry on. - -### [friction] Hand-authoring a param-cell vector needs `Cell::from_i64`, not the obvious `Cell::I64` -- Example: `rw_2_open_ohlc_real.rs` (first compile attempt). -- A consumer bootstrapping a harness with literal params naturally reaches for - `Cell::I64(3)` / `Cell::F64(1.0)` (mirroring `Scalar::I64` / `Scalar::F64`, - which ARE tuple-style and used right beside it for grid values). `Cell` instead - exposes only `from_i64` / `from_f64` / `from_bool` / `from_ts` constructors, so - the first build failed with `no associated item 'I64' found for struct Cell`. - The compiler's suggestion was excellent (it listed `from_i64` etc.), so the fix - was a 5-second one — but the asymmetry with `Scalar`'s tuple variants is a real - papercut for hand-authored params (the prior fieldtests sidestepped it by only - ever feeding `Cell`s that came pre-built out of a `GridSpace`). Friction, not a - bug: the task completed and the diagnostic was clean. -- Recommended action: plan a tidy iteration — either add `Cell::I64`/`F64`/etc. - tuple-style constructors (or `From`/`From` impls) to match `Scalar`, - or document the `from_*` constructors on `Cell`'s rustdoc as the param-authoring - entry point. (Low priority; ergonomic only.) - -### [spec_gap] `runs family ` exits 0 with empty output, not a "no such family" diagnostic -- Example: `out_rw_4_runs_coherence.txt`, final block. -- `aura runs family nope-99` (a family id that does not exist) prints nothing and - exits **0**. The retired commands exit 2 with a usage line, and a real consumer - who typos a family id would reasonably expect a non-zero exit or a "no such - family `nope-99`" message rather than silent success that looks - indistinguishable from "a real but empty family". The top-level usage string, - the ledger (C18 realization note), and the glossary do not constrain what an - unknown-but-well-formed family id should do, so I cannot tell whether exit 0 + - empty is the intended contract (treat-as-empty, mirroring how a missing store - is `Ok(vec![])`) or an oversight. Two readings are equally plausible: - (a) **treat-as-empty** — consistent with `Registry::load` treating a missing - file as an empty registry, so an unknown id is "a family with zero members"; - (b) **not-found is an error** — consistent with the retired commands and with - typo-safety. I picked no behaviour to assert in a fixture; I recorded the - observation. -- Recommended action: ratify the intended behaviour into the C18 realization - note (and the CLI usage), or tighten it to a non-zero exit + diagnostic if (b). - -## Recommendation summary - -| Finding | Class | Action | -|---|---|---| -| Per-instrument pip honesty (vetted pip + refusal) | working | carry-on | -| Real-data source builds from `aura-ingest` alone (`open_ohlc`, re-exports) | working | carry-on | -| Multi-tap traces fuse on timestamp (`join_on_ts`) | working | carry-on | -| Run-registry coherence + two-store rustdoc | working | carry-on | -| `Cell` lacks tuple-style constructors (asymmetry with `Scalar`) | friction | plan (tidy iteration) | -| `runs family ` → exit 0, empty | spec_gap | ratify / tighten the design ledger | diff --git a/docs/specs/0056-trace-charts-persist-and-serve.md b/docs/specs/0056-trace-charts-persist-and-serve.md deleted file mode 100644 index 2c91201..0000000 --- a/docs/specs/0056-trace-charts-persist-and-serve.md +++ /dev/null @@ -1,432 +0,0 @@ -# Trace charts — persist recorder taps, serve as web charts — Design Spec - -**Date:** 2026-06-18 -**Status:** Draft — awaiting spec review (autonomous build; specify auto-sign gate) -**Authors:** orchestrator + Claude - -## Goal - -Make a backtest's recorded streams **visible**. Today a run's recorder taps -(`equity`, `exposure`, or any wired column) are drained in-memory, folded to three -summary numbers (`total_pips` / `max_drawdown` / `exposure_sign_flips`), and then -discarded — nothing persists the time series, and the only visual surface -(`aura graph`) shows graph *structure*, not run *data*. - -This cycle closes the gap end-to-end: **tap data from a graph → persist it to disk -→ serve it as a chart in the browser.** It is the first cut of aura's visual face -under the web-from-disk seam ratified this session into C14/C22 (commit `3b56efb`, -issue #101): the engine writes recorded traces to disk; a static self-contained -HTML page (the `render_html` precedent) charts them; multiple feeds are overlaid on -a shared timestamp axis, or aligned in stacked panels. - -In scope (first cut): - -1. A **columnar trace encoding** (`ColumnarTrace`) — the SoA form of one drained - tap, on disk as JSON (C7: one column = one series). -2. A **disk trace store** under `runs/traces//`, beside the run registry - (C18, `runs/`): write a run's taps, read them back. -3. `aura run --trace ` — persist the sample harness's taps (additive; the - default `aura run` is byte-for-byte unchanged). -4. `aura chart [--panels]` — read a persisted run's taps and emit a - self-contained HTML chart page (uPlot, vendored), feeds **overlaid** (default, - shared timestamp x via the existing `join_on_ts`) or in **stacked panels**. - -Out of scope (explicit follow-on, named so the boundary is auditable): families / -comparison across multiple runs (C21), a local HTTP server, level-of-detail / -"mipmapping" downsampling, an Arrow/Parquet columnar store, and live streaming -during a run. - -## Architecture - -Four layers, each landing where its existing siblings already live — no new crate, -no engine UI knowledge (C14 preserved): - -| Layer | Crate / file | Sibling precedent it joins | -|-------|--------------|----------------------------| -| Columnar encoding (pure) | `aura-engine` / `report.rs` | `summarize`, `f64_field`, `join_on_ts` — pure post-run reductions over recorded streams | -| Disk trace store (I/O) | `aura-registry` / new `trace_store.rs` | `Registry` owns `runs/` + directory-co-located sibling stores (`families.jsonl`) | -| Chart-page assembly (string) | `aura-cli` / `render.rs` | `render_html` — self-contained HTML, vendored JS assets | -| CLI verbs + serve-time align | `aura-cli` / `main.rs` | the exhaustive argv match; `run_sample` drain→fold | - -The crate-homes are dictated by the existing structure, not chosen: a columnar -encoding is the same family as `join_on_ts` (a pure reduction over -`&[(Timestamp, Vec)]`), the on-disk store is the same family as the -registry's directory-co-located stores, and the HTML assembly is the same family as -`render_html`. Engine purity holds: `aura-engine` does the *encoding* (struct → -JSON), never file I/O; the registry crate does the I/O; the CLI does stdout + -browser hand-off. No pixel/UI knowledge enters the engine (C14). - -Determinism (C1) and causality (C2/C3) are untouched: persistence is a **post-run** -reduction over already-recorded sink output, exactly like `summarize`. `join_on_ts` -stays a post-run reduction (C3: not an in-graph join), now invoked at *serve* time -rather than only in example code. - -## Concrete code shapes - -### 1. The user-facing flow (the criterion's empirical evidence) - -A researcher who just ran a backtest wants to *see* the equity curve and the -exposure, not read three numbers. The reach is two commands: - -```console -# persist the run's taps under runs/traces/demo/ (default run output unchanged) -$ aura run --trace demo -{"manifest":{...},"metrics":{"total_pips":...,"max_drawdown":...,"exposure_sign_flips":...}} - -$ ls runs/traces/demo/ -index.json equity.json exposure.json - -# serve them as a self-contained chart page (overlay: both feeds, shared ts axis) -$ aura chart demo > demo.html && xdg-open demo.html - -# or timestamp-aligned stacked panels (one per feed, shared time axis) -$ aura chart demo --panels > demo.html -``` - -`equity.json` — the columnar (SoA) form of one tap, chart-ready (one column = one -series), kind-tagged for honesty (C7): - -```json -{ - "tap": "equity", - "kinds": ["F64"], - "ts": [1, 2, 3, 4, 5, 6, 7], - "columns": [[0.0, 0.0, 0.0010, 0.0040, 0.0040, 0.0010, -0.0010]] -} -``` - -`index.json` — the tap list (deterministic read order) + the run manifest, so the -page header can show run context (commit / params / window / broker), mirroring -`aura graph`'s header: - -```json -{ "manifest": { "commit": "...", "params": [...], "window": [1,7], "seed": 0, - "broker": "sim-optimal(pip_size=0.0001)" }, - "taps": ["equity", "exposure"] } -``` - -The emitted page is self-contained (no remote fetch), injecting the aligned data as -`window.AURA_TRACES` and inlining vendored uPlot + a `chart-viewer.js` — exactly the -`render_html` pattern. - -### 2. `ColumnarTrace` (aura-engine / report.rs) — NEW - -```rust -/// One drained recorder tap in columnar (SoA) form: parallel arrays, one per -/// recorded column, plus the shared recorded-timestamp axis. Chart-ready (a column -/// is a series of numbers) and kind-tagged (C7: the true base type of each column -/// survives even though values are coerced to f64 for plotting). Pure: the same -/// rows always encode to the same struct (C1). -#[derive(Clone, Debug, PartialEq, serde::Serialize, serde::Deserialize)] -pub struct ColumnarTrace { - pub tap: String, - pub kinds: Vec, // "F64" | "I64" | "Bool" | "Timestamp" (no serde dep on ScalarKind) - pub ts: Vec, // recorded timestamps (Timestamp.0) - pub columns: Vec>, // columns[c][r] — coerced: f64 as-is, i64 as f64, bool 1/0, ts as f64 -} - -impl ColumnarTrace { - /// Transpose a drained tap (`(ts, row)` pairs) into columns. Empty rows -> empty - /// ts + empty columns sized to `kinds`. - pub fn from_rows(tap: &str, kinds: &[ScalarKind], rows: &[(Timestamp, Vec)]) -> Self { /* ... */ } - - /// Inverse for the serve/align path: rebuild `(ts, row)` pairs with each cell as - /// `Scalar::f64(columns[c][r])`. The on-disk store is already f64 columns, so the - /// rebuilt rows are **uniformly f64** (the original base type survives only in - /// `kinds` as display metadata) — which keeps the serve-side `as_f64` projection - /// (§6 step 3) total, never the F64-only panic path. A true value round-trip for - /// the shipped F64 taps; for other kinds the value round-trips through the f64 - /// store with `kinds` carrying the base type. - pub fn to_rows(&self) -> Vec<(Timestamp, Vec)> { /* ... */ } -} -``` - -### 3. `TraceStore` (aura-registry / trace_store.rs) — NEW - -```rust -/// A per-run directory of columnar tap files under `/traces//`, -/// beside the run registry's `runs.jsonl`. Directory-keyed (like the family -/// sibling store): one subdir per run name, one `.json` per tap, plus -/// `index.json` (tap order + manifest). -pub struct TraceStore { dir: PathBuf } // dir = /traces - -impl TraceStore { - pub fn open(runs_dir: impl AsRef) -> TraceStore { /* runs_dir/traces */ } - - /// Write one run's taps + index under traces//, creating dirs as needed. - pub fn write(&self, name: &str, manifest: &RunManifest, taps: &[ColumnarTrace]) -> Result<(), TraceStoreError> { /* ... */ } - - /// Read a run's index + taps back, in index order. A missing run dir is - /// `NotFound` (treat-as-absent, C18-consistent), not a panic. - pub fn read(&self, name: &str) -> Result { /* ... */ } -} - -pub struct RunTraces { pub manifest: RunManifest, pub taps: Vec } -``` - -### 4. Chart-page assembly (aura-cli / render.rs) — NEW, mirrors `render_html` - -```rust -const UPLOT_JS: &str = include_str!("../assets/uPlot.iife.min.js"); -const UPLOT_CSS: &str = include_str!("../assets/uPlot.min.css"); -const CHART_VIEWER_JS: &str = include_str!("../assets/chart-viewer.js"); // NEW asset - -pub enum ChartMode { Overlay, Panels } - -/// Self-contained chart page for one run's taps. `data` (a `ChartData`, §6) carries -/// the taps aligned on the shared union-spine x — one `Series` per (tap, column), -/// each with its own y-scale id so disparate magnitudes stay legible; `mode` selects -/// overlay vs panels over the *same* data. The page fetches nothing at view time. -pub fn render_chart_html(data: &ChartData, mode: ChartMode) -> String { - // SHELL_HEAD-style dark shell + run-context header - // + + - // + window.AURA_TRACES = {mode, manifest, taps, xs, series:[{name,y_scale_id,points}]} - // + // overlay: one uPlot, per-series y-scale; - // // panels: one uPlot per series, shared xs -} -``` - -### 5. CLI wiring (aura-cli / main.rs) — before → after - -`--trace ` is an **orthogonal modifier** that composes with **every** run form -— plain `run`, `run --macd`, and `run --real `. All three already drain the -identical two taps (`rx_eq` / `rx_ex`) and build a `RunManifest` (`run_sample`:159, -`run_macd`:836, `run_sample_real`:238), so one shared helper serves all three; the -persist step is inserted between drain and fold (additive; runs only when the trace -name is `Some`): - -```rust -// before: -let eq_rows: Vec<(Timestamp, Vec)> = rx_eq.try_iter().collect(); -let ex_rows: Vec<(Timestamp, Vec)> = rx_ex.try_iter().collect(); -let equity = f64_field(&eq_rows, 0); -let exposure = f64_field(&ex_rows, 0); -let metrics = summarize(&equity, &exposure); -// ... RunReport - -// shared helper (aura-cli), called by run_sample / run_macd / run_sample_real: -fn persist_traces(name: &str, manifest: &RunManifest, - eq_rows: &[(Timestamp, Vec)], - ex_rows: &[(Timestamp, Vec)]) -> Result<(), TraceStoreError> { - let taps = vec![ - ColumnarTrace::from_rows("equity", &[ScalarKind::F64], eq_rows), - ColumnarTrace::from_rows("exposure", &[ScalarKind::F64], ex_rows), - ]; - TraceStore::open("runs").write(name, manifest, &taps) -} - -// after, in each run path (manifest hoisted before the fold; trace: Option<&str>): -let eq_rows = rx_eq.try_iter().collect::>(); -let ex_rows = rx_ex.try_iter().collect::>(); -if let Some(name) = trace { persist_traces(name, &manifest, &eq_rows, &ex_rows)?; } -let metrics = summarize(&f64_field(&eq_rows, 0), &f64_field(&ex_rows, 0)); -// ... RunReport unchanged; when trace is None, stdout + behaviour are byte-for-byte as before -``` - -`main()` argv match. `--trace` composes with **every** run form; `run_sample` / -`run_macd` / `run_sample_real` each gain a `trace: Option<&str>` parameter — the -existing no-trace arms pass `None`, so their stdout stays byte-for-byte unchanged. -Every combination the new `USAGE` advertises has an explicit arm (no advertised form -falls through to the usage error): - -```rust -// before: -["run"] => println!("{}", run_sample().to_json()), -["run", "--macd"] => println!("{}", run_macd().to_json()), -["run", "--real", rest @ ..] => match parse_real_args(rest) { Ok((sym, from, to)) => ... }, - -// after — existing arms pass None (unchanged stdout); new arms thread the trace name: -["run"] => println!("{}", run_sample(None).to_json()), -["run", "--macd"] => println!("{}", run_macd(None).to_json()), -["run", "--trace", name] => println!("{}", run_sample(Some(name)).to_json()), -["run", "--macd", "--trace", name] => println!("{}", run_macd(Some(name)).to_json()), -// parse_real_args -> 4-tuple (symbol, from, to, trace: Option); the existing -// ["run","--real", rest @ ..] arm threads it: run_sample_real(&sym, from, to, trace.as_deref()) -["chart", name] => emit_chart(name, ChartMode::Overlay), -["chart", name, "--panels"] => emit_chart(name, ChartMode::Panels), -``` - -`parse_real_args` gains one tail flag — `--trace ` — alongside the existing -`--from`/`--to` pairs (same once-only, value-required grammar), returning a fourth -`Option` element; `run_sample_real` takes `trace: Option<&str>` and persists -the same two taps when it is `Some`. `emit_chart` reads the store and aligns -**all** taps as symmetric sides of a synthetic empty-payload union spine via -`join_on_ts` (the exact three-step mechanism in §6; no tap privileged, no points -dropped), producing one `ChartData` that feeds **both** modes (overlay superimposes -the series, panels stack one plot per series on that shared x), then prints the page; a -missing run name is a usage error (stderr + `exit 2`), consistent with the rest of the -CLI. `USAGE` becomes -`aura run [--macd] [--trace ] | aura run --real [--from ] [--to ] [--trace ] | aura chart [--panels] | …` -— every advertised `[--macd] [--trace]` combination has a matching arm above, so the -usage string and the dispatch cannot disagree. - -### 6. Overlay & panel chart semantics (shared timestamp x; the y-axis choice) - -**Both modes are timestamp-aligned on one shared x-axis** — the **sorted union of all -taps' recorded timestamps**, aligned via `join_on_ts` with the union as the spine -(each series null-filled where its tap did not fire — uPlot renders the gap; a *union* -spine drops no tap's points). This is exactly the user's "mehrere Feeds in einem -Chart, **oder aligned nach Timestamps in Panels**": the shared timestamp x is what -makes a vertical time-cursor line up across feeds in **both** modes (issue #101, -Fork A — timestamp-aligned panels). - -The two modes differ only in **render** and in the **y-axis**, which is load-bearing -whenever feeds differ in magnitude (pip-equity ~±0.004 vs exposure ±1.0 — a single -shared y-scale would flatten the equity curve to a line): - -- **Overlay** (default) — **one** plot, all series superimposed, **each on its own - auto-ranged y-scale** (uPlot's multi-scale: a named scale per series, independently - min/max-ranged). The first two series carry a drawn y-axis (left, right); any - further series keep their own auto-scale, read via the legend/tooltip. Suits a few - feeds you want superimposed in one frame. -- **Panels** (`--panels`) — **one stacked uPlot per series**, each on its own y-scale, - **all sharing the common union-spine timestamp x-axis** so a vertical time-cursor - lines up across the stacked panels (the "aligned nach Timestamps in Panels" the user - asked for). The clean choice when feeds are numerous or wildly different in - magnitude. - -**Serve-time alignment (both modes), the exact mechanism — no privileged tap.** -`join_on_ts(spine, sides)` keys `sides` off a caller-supplied `spine` of -`(Timestamp, Vec)` rows and drops any side row whose ts is absent from the -spine (report.rs:498); it does **not** itself compute a union. `emit_chart` therefore -builds the alignment in three concrete steps, putting **no tap in the privileged -`JoinedRow.spine` slot**: - -1. **Union x.** Concatenate every tap's `ts` array, sort, dedup → `xs: Vec` — - the shared x-axis. -2. **Symmetric join.** Synthesize a spine of *empty-payload* rows - `xs.iter().map(|&t| (Timestamp(t), Vec::new()))`, and pass **all** taps (each - rebuilt via `ColumnarTrace::to_rows`) as the `sides` of - `join_on_ts(&spine, &all_tap_rows)`. Because the spine covers every tap's - timestamps, no side row is dropped; the synthetic spine's own empty payload is - never read. Every tap is a symmetric side — none occupies the never-gap-filled - `JoinedRow.spine`. -3. **Flatten to series.** Each tap contributes **one chart series per recorded - column** (the shipped `equity`/`exposure` taps are single-column → one series - each). Tap *i*'s column *c* maps to - `joined.iter().map(|r| r.sides[i].as_ref().map(|row| row[c].as_f64()))` — the - value where tap *i* fired at that x, `None` (a uPlot gap) otherwise. The result is - `ChartData { xs: Vec, series: Vec, manifest, taps }`, where a - `Series = { name: String, y_scale_id: String, points: Vec> }` (one - per (tap, column); the `y_scale_id` is what gives overlay its per-series y-scale). - Both modes consume the *same* `ChartData`: overlay draws all `series` on one plot - over `xs`; panels draw one stacked uPlot per `series`, all over the same `xs`. - -For the first-cut sample harness the two taps (`equity`, `exposure`) are -**co-cadenced** (both fire every cycle), so the union spine is exactly their shared -timestamps; the per-series y-scales are what keep the ±0.004 equity legible beside the -±1.0 exposure. The **timestamp-aligned shared x (both modes)** is the source-mandated -contract (issue #101, Fork A); the **per-series y-scale** and the **union (vs. -one-tap) spine** are documented naive-viewer defaults settled under the -autonomous-build mandate and recorded on issue #101 — not user-picked design forks, -but documented orchestrator decisions within that mandate. - -## Components - -- **`ColumnarTrace`** (aura-engine): transpose/round-trip of one tap. Pure. -- **`TraceStore` + `RunTraces` + `TraceStoreError`** (aura-registry): the on-disk - per-run trace directory; write + read; missing = `NotFound`. -- **`render_chart_html` + `ChartMode` + `ChartData`** (aura-cli): page assembly. -- **Vendored assets** (aura-cli/assets): `uPlot.iife.min.js`, `uPlot.min.css` - (third-party, pinned + recorded in `refresh-assets.sh` like the existing - Graphviz-WASM/pan-zoom blobs), and `chart-viewer.js` (first-party, the - `AURA_TRACES` → uPlot bootstrap). -- **CLI** (aura-cli/main.rs): the shared `persist_traces` helper threaded into - `run_sample` / `run_macd` / `run_sample_real` (each `trace: Option<&str>`), - `emit_chart`, the new argv arms, and the serve-time union-spine `join_on_ts` - alignment shared by both chart modes. - -## Data flow - -``` -aura run --trace demo - harness.run() -> recorder taps drained -> Vec<(Timestamp, Vec)> per tap - | | - | ColumnarTrace::from_rows (transpose, coerce, kind-tag) - | v - | TraceStore::write -> runs/traces/demo/{index,equity,exposure}.json - v - summarize -> RunReport -> stdout (UNCHANGED) - -aura chart demo [--panels] - TraceStore::read(demo) -> RunTraces { manifest, taps: [ColumnarTrace] } - | | - | synthetic empty union-spine + ALL taps as sides -> join_on_ts -> ChartData - | { xs, series:[{name,y_scale_id,points: Option per x}] } (null = gap, no drop) - | overlay: ONE plot, all series, per-series y-scale - | panels: ONE stacked plot per series, all on the shared xs (timestamp-aligned) - v - render_chart_html(ChartData, mode) -> self-contained HTML -> stdout -> browser (uPlot) -``` - -## Error handling - -- **Unknown run name** (`aura chart ` with no `runs/traces//`): - `TraceStore::read` returns `NotFound`; the CLI prints `aura: no recorded run - '' under runs/traces` to stderr and `exit 2` — usage-error parity with the - rest of the CLI, never a panic. (Consistent with C18's treat-missing-as-absent.) -- **Corrupt / unreadable trace file**: `TraceStoreError::{Io, Parse{file,source}}` - surfaced like `RegistryError` (file named, propagated), `exit 2`. -- **`--trace` write failure** (I/O): propagated from the run path as a usage-level - error (stderr + `exit 2`); the run still computed its metrics, so the message - names the persistence step, not the run. -- **Empty taps** (a run that recorded nothing): `from_rows` yields empty arrays; - `render_chart_html` emits a page with an empty chart and a "no recorded samples" - note — no panic, no division-by-zero. -- **Bool/ts coercion**: documented and total (bool→1.0/0.0, ts→f64); `kinds` - preserves the real base type so the store stays honest (C7). - -## Testing strategy - -- **`ColumnarTrace` round-trip** (aura-engine unit): `from_rows` then `to_rows` - reproduces the row set; columnar shape asserted (ts length, column count = - kinds, per-kind coercion); empty-rows case. -- **`TraceStore` write→read** (aura-registry unit, temp dir): a written run reads - back equal (manifest + taps in index order); missing run = `NotFound`; a - corrupt tap file = `Parse{file,..}` with the file named. Temp-dir isolation - mirrors the registry's `temp_family_dir` pattern. -- **`render_chart_html` self-contained** (aura-cli unit, mirrors - `render_html_is_self_contained_and_embeds_the_model`): starts ``, - closes ``, no remote `