diff --git a/docs/plans/0056-trace-charts-serve.md b/docs/plans/0056-trace-charts-serve.md new file mode 100644 index 0000000..5c53e2f --- /dev/null +++ b/docs/plans/0056-trace-charts-serve.md @@ -0,0 +1,583 @@ +# Trace charts — iteration 2 (serve) — Implementation Plan + +> **Parent spec:** `docs/specs/0056-trace-charts-persist-and-serve.md` +> +> **For agentic workers:** REQUIRED SUB-SKILL: use the `implement` skill to run +> this plan. Steps use `- [ ]` checkboxes for tracking. + +**Goal:** Read a persisted run's traces (`runs/traces//`) and serve them as a +self-contained uPlot HTML chart via `aura chart [--panels]` — feeds overlaid +on a shared timestamp axis (default) or in timestamp-aligned stacked panels. + +**Architecture:** Three layers on top of iteration 1's on-disk artifact. A +first-party `chart-viewer.js` exposes a pure `buildCharts(traces, mode)` (the +`AURA_TRACES` → uPlot-config transform, no DOM) plus a browser-only `mount()`. +`render_chart_html` (aura-cli/render.rs) assembles a self-contained page mirroring +`render_html`: vendored uPlot (already checked in, commit d534f10) + `chart-viewer.js` +inlined via `include_str!`, the data injected as `window.AURA_TRACES`. `emit_chart` +(aura-cli/main.rs) reads the `TraceStore`, builds `ChartData` by the spec-§6 3-step +union-spine `join_on_ts` alignment, and prints the page. The engine is untouched (C14). + +**Tech Stack:** Rust (`aura-cli`); the iteration-1 building blocks +`aura_engine::{ColumnarTrace, join_on_ts, JoinedRow}` + `aura_registry::{TraceStore, +RunTraces, TraceStoreError}`; vendored uPlot 1.6.32; `serde_json` (already an +aura-cli dep); a `node` `.mjs` DOM guard mirroring `viewer_dot.rs`. + +--- + +**Files this plan creates or modifies:** + +- Create: `crates/aura-cli/assets/chart-viewer.js` — first-party `AURA_TRACES` → + uPlot bootstrap (pure `buildCharts` + browser `mount`). +- Create: `crates/aura-cli/tests/chart_viewer.rs` — `.rs` node-driver (shells `node`). +- Create: `crates/aura-cli/tests/chart_viewer_build.mjs` — the DOM-less guard over + `buildCharts`. +- Create: `crates/aura-cli/tests/fixtures/sample-traces.json` — an `AURA_TRACES` + fixture for the guard. +- Modify: `crates/aura-cli/src/render.rs` — add `ChartMode`/`ChartData`/`Series` + + `render_chart_html` + a self-contained unit test (mirror `render.rs:92`). +- Modify: `crates/aura-cli/src/main.rs` — imports (`:17–26`), `emit_chart` + the + `build_chart_data` alignment helper, the `["chart", …]` argv arms (`:885–914`, + before `["graph"]` at `:899`), `USAGE` (`:877–878`). +- Test: `crates/aura-cli/tests/cli_run.rs` — a `chart` e2e sibling (HTML emitted; + missing run exits 2). + +uPlot is already vendored (commit d534f10: `assets/uPlot.iife.min.js` + +`uPlot.min.css` + the `refresh-assets.sh` line) — no fetch in this plan. + +--- + +### Task 1: `chart-viewer.js` + its DOM guard + +**Files:** +- Create: `crates/aura-cli/assets/chart-viewer.js` +- Create: `crates/aura-cli/tests/chart_viewer_build.mjs` +- Create: `crates/aura-cli/tests/chart_viewer.rs` +- Create: `crates/aura-cli/tests/fixtures/sample-traces.json` + +- [ ] **Step 1: Write the fixture** + +Create `crates/aura-cli/tests/fixtures/sample-traces.json` (an `AURA_TRACES` payload +with two co-cadenced single-column series, matching what `render_chart_html` injects): + +```json +{ + "mode": "overlay", + "manifest": { "commit": "test", "params": [], "window": [1, 3], "seed": 0, "broker": "sim-optimal(pip_size=0.0001)" }, + "taps": ["equity", "exposure"], + "xs": [1, 2, 3], + "series": [ + { "name": "equity", "y_scale_id": "y_0", "points": [0.0, 0.001, 0.004] }, + { "name": "exposure", "y_scale_id": "y_1", "points": [1.0, 1.0, -1.0] } + ] +} +``` + +- [ ] **Step 2: Write the failing guard (.mjs + .rs driver)** + +Create `crates/aura-cli/tests/chart_viewer_build.mjs`: + +```js +// Headless guard for the trace-chart viewer (chart-viewer.js). Property protected: +// buildCharts maps an AURA_TRACES payload to uPlot configs WITHOUT a DOM — overlay +// is one chart with every series on its own y-scale over the shared xs; panels is +// one chart per series, each on the SAME shared xs (timestamp-aligned). It loads the +// real viewer module (a fix/regression there is observed here) and drives the +// exported pure `buildCharts`; uPlot itself (which needs `document`) is never +// instantiated — mount() is browser-only and not called here. + +import { createRequire } from "node:module"; +import { readFileSync } from "node:fs"; +import { fileURLToPath } from "node:url"; +import { dirname, join } from "node:path"; + +const require = createRequire(import.meta.url); +const here = dirname(fileURLToPath(import.meta.url)); +const traces = JSON.parse(readFileSync(join(here, "fixtures", "sample-traces.json"), "utf8")); + +const fail = (msg) => { console.error("FAIL: " + msg); process.exit(1); }; +const eq = (a, b, msg) => { if (a !== b) fail(`${msg} (got ${a}, want ${b})`); }; +const sameArr = (a, b, msg) => { + if (!Array.isArray(a) || !Array.isArray(b) || a.length !== b.length || a.some((v, i) => v !== b[i])) fail(msg); +}; + +// No `document` global -> chart-viewer.js's browser mount is skipped; require gives +// us only the pure export (mirrors viewer_dot_ids.mjs stubbing window before require). +global.window = { AURA_TRACES: traces }; +const { buildCharts } = require(join(here, "..", "assets", "chart-viewer.js")); + +const n = traces.series.length; + +// overlay: ONE chart, all series superimposed on the shared xs, each on its own scale. +const overlay = buildCharts(traces, "overlay"); +eq(overlay.length, 1, "overlay is one chart"); +eq(overlay[0].data.length, 1 + n, "overlay data = xs + N series"); +eq(overlay[0].opts.series.length, 1 + n, "overlay uPlot series = x + N"); +sameArr(overlay[0].data[0], traces.xs, "overlay x-axis is the shared xs"); +const scaleIds = overlay[0].opts.series.slice(1).map((s) => s.scale); +eq(new Set(scaleIds).size, n, "each overlay series has a distinct y-scale"); + +// panels: ONE chart per series, all timestamp-aligned on the SAME shared xs. +const panels = buildCharts(traces, "panels"); +eq(panels.length, n, "panels = one chart per series"); +for (const p of panels) { + eq(p.data.length, 2, "each panel = xs + its one series"); + sameArr(p.data[0], traces.xs, "panel shares the same xs (timestamp-aligned)"); +} + +console.log(`OK — overlay 1 chart / ${n} series, panels ${n} charts, shared xs.`); +process.exit(0); +``` + +Create `crates/aura-cli/tests/chart_viewer.rs` (mirrors `viewer_dot.rs`): + +```rust +//! Integration guard for the `aura chart` viewer's data transform. +//! +//! `chart-viewer.js` builds the uPlot configs in JavaScript, so the property +//! "buildCharts maps AURA_TRACES to the right per-mode uPlot config shape" cannot +//! be checked from Rust. This shells out to `node`, running +//! `tests/chart_viewer_build.mjs`, which loads the real viewer module and asserts +//! the overlay/panels config shape over a fixture — without instantiating uPlot +//! (which needs a DOM). `node` is REQUIRED: if absent this test FAILS. + +use std::path::PathBuf; +use std::process::Command; + +#[test] +fn chart_viewer_builds_overlay_and_panels_configs() { + let script = PathBuf::from(env!("CARGO_MANIFEST_DIR")) + .join("tests") + .join("chart_viewer_build.mjs"); + assert!(script.exists(), "guard script missing at {}", script.display()); + + let out = match Command::new("node").arg(&script).output() { + Ok(out) => out, + Err(e) => panic!( + "node is required for the chart-viewer guard but could not be run \ + ({e}); install Node.js or ensure `node` is on PATH" + ), + }; + + let stdout = String::from_utf8_lossy(&out.stdout); + let stderr = String::from_utf8_lossy(&out.stderr); + assert!( + out.status.success(), + "chart-viewer guard failed (exit {:?}).\n--- node stdout ---\n{stdout}\n--- node stderr ---\n{stderr}", + out.status.code() + ); +} +``` + +- [ ] **Step 3: Run the guard to verify it fails** + +Run: `cargo test -p aura-cli --test chart_viewer chart_viewer_builds_overlay_and_panels_configs` +Expected: FAIL — node cannot `require` `assets/chart-viewer.js` (it does not exist yet), so the guard exits non-zero and the `.rs` assert reports the node stderr. + +- [ ] **Step 4: Write `chart-viewer.js`** + +Create `crates/aura-cli/assets/chart-viewer.js`: + +```js +// The `aura chart` viewer: turns the injected window.AURA_TRACES (xs + per-series +// Option points, already timestamp-union-aligned by the CLI) into uPlot charts. +// +// buildCharts is PURE (no DOM, no uPlot) and is the unit the headless guard drives. +// mount() is browser-only — it instantiates the vendored global `uPlot` per config. +(function () { + var PALETTE = ["#89b4fa", "#f38ba8", "#a6e3a1", "#f9e2af", "#cba6f7", "#94e2d5"]; + + // {xs, series:[{name, y_scale_id, points}]} + mode -> array of {opts, data} configs. + // overlay -> one chart, all series, each on its own auto-ranged y-scale (shared xs). + // panels -> one chart per series, each on the SAME shared xs (timestamp-aligned). + function buildCharts(traces, mode) { + var xs = traces.xs; + var series = traces.series; + if (mode === "panels") { + return series.map(function (s, i) { + return { + opts: { + title: s.name, + scales: { x: { time: false }, [s.y_scale_id]: {} }, + series: [{}, { label: s.name, scale: s.y_scale_id, stroke: PALETTE[i % PALETTE.length] }], + axes: [{}, { scale: s.y_scale_id }], + }, + data: [xs, s.points], + }; + }); + } + // overlay (default) + var scales = { x: { time: false } }; + var uSeries = [{}]; + var axes = [{}]; + series.forEach(function (s, i) { + scales[s.y_scale_id] = {}; + uSeries.push({ label: s.name, scale: s.y_scale_id, stroke: PALETTE[i % PALETTE.length] }); + // the first two series carry a drawn axis (left, then right); the rest keep + // their own auto-scale, read via the legend. + if (i < 2) axes.push({ scale: s.y_scale_id, side: i === 0 ? 3 : 1 }); + }); + return [{ opts: { scales: scales, series: uSeries, axes: axes }, data: [xs].concat(series.map(function (s) { return s.points; })) }]; + } + + // Browser-only: instantiate the vendored global uPlot per config into #stage. + function mount() { + var traces = window.AURA_TRACES; + var configs = buildCharts(traces, traces.mode); + var stage = document.getElementById("stage"); + var w = stage.clientWidth || 900; + var h = configs.length > 1 ? Math.max(160, Math.floor(440 / configs.length)) : 460; + configs.forEach(function (cfg) { + var opts = Object.assign({ width: w, height: h }, cfg.opts); + // eslint-disable-next-line no-undef + new uPlot(opts, cfg.data, stage); + }); + } + + if (typeof window !== "undefined" && typeof document !== "undefined" && window.AURA_TRACES) { + if (document.readyState === "loading") document.addEventListener("DOMContentLoaded", mount); + else mount(); + } + if (typeof module !== "undefined" && module.exports) module.exports = { buildCharts: buildCharts }; +})(); +``` + +- [ ] **Step 5: Run the guard to verify it passes** + +Run: `cargo test -p aura-cli --test chart_viewer chart_viewer_builds_overlay_and_panels_configs` +Expected: PASS — `OK — overlay 1 chart / 2 series, panels 2 charts, shared xs.` (1 passed). + +--- + +### Task 2: `render_chart_html` + chart types (render.rs) + +**Files:** +- Modify: `crates/aura-cli/src/render.rs` (types + fn after `render_html`; test in `mod tests`) + +- [ ] **Step 1: Write the failing self-contained test** + +Append inside the existing `#[cfg(test)] mod tests { … }` in `crates/aura-cli/src/render.rs`: + +```rust + fn sample_chart_data() -> ChartData { + ChartData { + manifest: aura_engine::RunManifest { + commit: "test".into(), + params: vec![], + window: (aura_engine::Timestamp(1), aura_engine::Timestamp(3)), + seed: 0, + broker: "sim-optimal(pip_size=0.0001)".into(), + }, + taps: vec!["equity".into(), "exposure".into()], + xs: vec![1, 2, 3], + series: vec![ + Series { name: "equity".into(), y_scale_id: "y_0".into(), points: vec![Some(0.0), Some(0.001), Some(0.004)] }, + Series { name: "exposure".into(), y_scale_id: "y_1".into(), points: vec![Some(1.0), Some(1.0), Some(-1.0)] }, + ], + } + } + + #[test] + fn render_chart_html_is_self_contained_and_injects_traces() { + let html = render_chart_html(&sample_chart_data(), ChartMode::Overlay); + assert!(html.starts_with(""), "not an HTML doc"); + assert!(html.trim_end().ends_with(""), "HTML doc not closed"); + // self-contained: no remote asset + assert!(!html.contains("\n\n\n\n\n"); + html +} +``` + +(`serde::Serialize` on `Series` lets `serde_json::json!` embed it; `RunManifest` +already derives `Serialize`. `serde_json` is already an aura-cli dependency — used by +`main.rs`'s `runs_family` JSON.) + +- [ ] **Step 4: Run the test to verify it passes** + +Run: `cargo test -p aura-cli --lib render_chart_html` +Expected: PASS — `render_chart_html_is_self_contained_and_injects_traces` + +`render_chart_html_reflects_panels_mode` green (2 passed). + +--- + +### Task 3: `emit_chart` + serve-time alignment + the `chart` verb (main.rs) + +This task is one compile unit (new `emit_chart`/`build_chart_data` + two argv arms + +imports + USAGE); it changes no existing signature, so there is no caller blast +radius. The RED e2e (Step 1) compiles against the binary and fails before the arms +exist. + +**Files:** +- Test: `crates/aura-cli/tests/cli_run.rs` (chart e2e sibling) +- Modify: `crates/aura-cli/src/main.rs` (imports `:17–26`, USAGE `:877–878`, argv + `:885–914`, new fns) + +- [ ] **Step 1: Write the failing e2e** + +Append to `crates/aura-cli/tests/cli_run.rs`: + +```rust +#[test] +fn chart_emits_self_contained_html_for_a_persisted_run() { + let cwd = temp_cwd("chart-serve"); + + // persist a run first (iteration 1), then chart it. + let traced = Command::new(BIN).args(["run", "--trace", "demo"]).current_dir(&cwd).output().expect("spawn run --trace"); + assert!(traced.status.success(), "run --trace exit: {:?}", traced.status); + + let chart = Command::new(BIN).args(["chart", "demo"]).current_dir(&cwd).output().expect("spawn chart"); + assert!(chart.status.success(), "chart exit: {:?}", chart.status); + let html = String::from_utf8(chart.stdout).expect("utf-8 html"); + assert!(html.starts_with(""), "not an HTML doc: {:?}", &html[..html.len().min(40)]); + assert!(html.contains("window.AURA_TRACES = {"), "AURA_TRACES not injected"); + assert!(html.contains("\"equity\""), "equity series missing from the chart"); + assert!(html.contains("uPlot=function"), "vendored uPlot not inlined"); + + // a missing run is a usage error (stderr + exit 2), not a panic. + let missing = Command::new(BIN).args(["chart", "nope"]).current_dir(&cwd).output().expect("spawn chart nope"); + assert_eq!(missing.status.code(), Some(2), "missing run must exit 2"); + assert!(!missing.status.success()); + + let _ = std::fs::remove_dir_all(&cwd); +} +``` + +- [ ] **Step 2: Run the e2e to verify it fails** + +Run: `cargo test -p aura-cli --test cli_run chart_emits_self_contained_html_for_a_persisted_run` +Expected: FAIL — `["chart","demo"]` has no argv arm yet, so it hits the usage-error path (exit 2) and prints no HTML. (Tree compiles — only a new test added.) + +- [ ] **Step 3: Extend the imports** + +In `crates/aura-cli/src/main.rs`, add `join_on_ts, JoinedRow` to the +`aura_engine::{…}` import (`main.rs:17–22`) and `RunTraces, TraceStoreError` to the +`aura_registry::{…}` import (`main.rs:23–26`). Also bring in the new render types: +the file already has `mod render;`, so reference them as `render::{ChartData, ChartMode, Series}` +at the call sites (no `use` needed) — OR add `use render::{ChartData, ChartMode, Series};` +after `mod render;` (`main.rs:14`). Use the explicit `use`: + +```rust +mod render; +use render::{ChartData, ChartMode, Series}; +``` + +and the two engine/registry import lists become (only the added names shown in context): + +```rust +use aura_engine::{ + f64_field, join_on_ts, monte_carlo, param_stability, summarize, walk_forward, window_of, + ColumnarTrace, Composite, Edge, FlatGraph, GraphBuilder, Harness, JoinedRow, McAggregate, + McFamily, RollMode, RunManifest, RunReport, SourceSpec, SweepFamily, SyntheticSpec, Target, + VecSource, WalkForwardResult, WindowBounds, WindowRoller, WindowRun, +}; +use aura_registry::{ + group_families, mc_member_reports, optimize, rank_by, sweep_member_reports, + walkforward_member_reports, FamilyKind, Registry, RunTraces, TraceStore, TraceStoreError, +}; +``` + +- [ ] **Step 4: Add the alignment helper + `emit_chart`** + +Insert after `persist_traces` (it ends near `main.rs:174`): + +```rust +/// Build the serve-ready `ChartData` from a run's read-back traces by the spec-§6 +/// 3-step union-spine alignment — no tap privileged, no point dropped: +/// (1) xs = the sorted, deduped union of every tap's timestamps; +/// (2) synthesize an empty-payload spine over xs and pass ALL taps (via +/// `ColumnarTrace::to_rows`, which yields uniformly-f64 rows) as symmetric sides +/// of `join_on_ts`, so no side row is dropped and none occupies the privileged +/// `JoinedRow.spine`; +/// (3) flatten each (tap, column) to a `Series` of `Option` over xs. +fn build_chart_data(traces: RunTraces) -> ChartData { + let mut xs: Vec = traces.taps.iter().flat_map(|t| t.ts.iter().copied()).collect(); + xs.sort_unstable(); + xs.dedup(); + + let spine: Vec<(Timestamp, Vec)> = xs.iter().map(|&t| (Timestamp(t), Vec::new())).collect(); + let tap_rows: Vec)>> = traces.taps.iter().map(|t| t.to_rows()).collect(); + let sides: Vec<&[(Timestamp, Vec)]> = tap_rows.iter().map(|r| r.as_slice()).collect(); + let joined: Vec = join_on_ts(&spine, &sides); + + let mut series: Vec = Vec::new(); + for (i, tap) in traces.taps.iter().enumerate() { + for c in 0..tap.columns.len() { + let name = if tap.columns.len() == 1 { tap.tap.clone() } else { format!("{}[{c}]", tap.tap) }; + let y_scale_id = format!("y_{}", series.len()); + let points: Vec> = + joined.iter().map(|r| r.sides[i].as_ref().map(|row| row[c].as_f64())).collect(); + series.push(Series { name, y_scale_id, points }); + } + } + + ChartData { + manifest: traces.manifest, + taps: traces.taps.iter().map(|t| t.tap.clone()).collect(), + xs, + series, + } +} + +/// `aura chart [--panels]`: read the persisted run's traces, align them, and +/// print the self-contained uPlot page. A missing run is a usage error (stderr + +/// exit 2), per the CLI's convention — never a panic. +fn emit_chart(name: &str, mode: ChartMode) { + let traces = match TraceStore::open("runs").read(name) { + Ok(t) => t, + Err(TraceStoreError::NotFound(n)) => { + eprintln!("aura: no recorded run '{n}' under runs/traces (run `aura run --trace {n}` first)"); + std::process::exit(2); + } + Err(e) => { + eprintln!("aura: {e}"); + std::process::exit(2); + } + }; + print!("{}", render::render_chart_html(&build_chart_data(traces), mode)); +} +``` + +- [ ] **Step 5: Add the argv arms** + +In `main()`'s argv match, insert the two chart arms directly before the `["graph"]` +arm (`main.rs:899`): + +```rust + ["chart", name] => emit_chart(name, ChartMode::Overlay), + ["chart", name, "--panels"] => emit_chart(name, ChartMode::Panels), +``` + +- [ ] **Step 6: Update `USAGE`** + +In the `USAGE` const (`main.rs:877–878`), add ` | aura chart [--panels]` after +the `aura graph` clause so every dispatched verb is advertised: + +```rust +const USAGE: &str = + "usage: aura run [--macd] [--trace ] | aura run --real [--from ] [--to ] [--trace ] | aura chart [--panels] | aura graph | aura sweep [--name ] | aura mc [--name ] | aura walkforward [--name ] | aura runs families | aura runs family [rank ]"; +``` + +- [ ] **Step 7: Run the e2e to verify it passes** + +Run: `cargo test -p aura-cli --test cli_run chart_emits_self_contained_html_for_a_persisted_run` +Expected: PASS — the persisted run charts to self-contained HTML carrying the equity +series + inlined uPlot; `aura chart nope` exits 2 (1 passed). + +- [ ] **Step 8: Build + lint + workspace gate** + +Run: `cargo build --workspace` +Expected: PASS — 0 errors. + +Run: `cargo test --workspace` +Expected: PASS — Task 1 (`chart_viewer`), Task 2 (`render_chart_html_*`), Task 3 +(`chart_emits_*`) green, and no regression (the iteration-1 trace tests + the +byte-for-byte default-run pins stay green). + +Run: `cargo clippy --workspace --all-targets -- -D warnings` +Expected: PASS — no warnings. (Watch for: a `clippy::needless_range_loop` on the +`for c in 0..tap.columns.len()` loop — if flagged, it reads `tap.columns[c]` only via +the parallel `joined[..].sides[i]` index, so rewrite as `.iter().enumerate()` over +`tap.columns` keeping `c`; an unused import if a name is mis-listed.)