diff --git a/docs/specs/0056-trace-charts-persist-and-serve.md b/docs/specs/0056-trace-charts-persist-and-serve.md new file mode 100644 index 0000000..2c91201 --- /dev/null +++ b/docs/specs/0056-trace-charts-persist-and-serve.md @@ -0,0 +1,432 @@ +# Trace charts — persist recorder taps, serve as web charts — Design Spec + +**Date:** 2026-06-18 +**Status:** Draft — awaiting spec review (autonomous build; specify auto-sign gate) +**Authors:** orchestrator + Claude + +## Goal + +Make a backtest's recorded streams **visible**. Today a run's recorder taps +(`equity`, `exposure`, or any wired column) are drained in-memory, folded to three +summary numbers (`total_pips` / `max_drawdown` / `exposure_sign_flips`), and then +discarded — nothing persists the time series, and the only visual surface +(`aura graph`) shows graph *structure*, not run *data*. + +This cycle closes the gap end-to-end: **tap data from a graph → persist it to disk +→ serve it as a chart in the browser.** It is the first cut of aura's visual face +under the web-from-disk seam ratified this session into C14/C22 (commit `3b56efb`, +issue #101): the engine writes recorded traces to disk; a static self-contained +HTML page (the `render_html` precedent) charts them; multiple feeds are overlaid on +a shared timestamp axis, or aligned in stacked panels. + +In scope (first cut): + +1. A **columnar trace encoding** (`ColumnarTrace`) — the SoA form of one drained + tap, on disk as JSON (C7: one column = one series). +2. A **disk trace store** under `runs/traces//`, beside the run registry + (C18, `runs/`): write a run's taps, read them back. +3. `aura run --trace ` — persist the sample harness's taps (additive; the + default `aura run` is byte-for-byte unchanged). +4. `aura chart [--panels]` — read a persisted run's taps and emit a + self-contained HTML chart page (uPlot, vendored), feeds **overlaid** (default, + shared timestamp x via the existing `join_on_ts`) or in **stacked panels**. + +Out of scope (explicit follow-on, named so the boundary is auditable): families / +comparison across multiple runs (C21), a local HTTP server, level-of-detail / +"mipmapping" downsampling, an Arrow/Parquet columnar store, and live streaming +during a run. + +## Architecture + +Four layers, each landing where its existing siblings already live — no new crate, +no engine UI knowledge (C14 preserved): + +| Layer | Crate / file | Sibling precedent it joins | +|-------|--------------|----------------------------| +| Columnar encoding (pure) | `aura-engine` / `report.rs` | `summarize`, `f64_field`, `join_on_ts` — pure post-run reductions over recorded streams | +| Disk trace store (I/O) | `aura-registry` / new `trace_store.rs` | `Registry` owns `runs/` + directory-co-located sibling stores (`families.jsonl`) | +| Chart-page assembly (string) | `aura-cli` / `render.rs` | `render_html` — self-contained HTML, vendored JS assets | +| CLI verbs + serve-time align | `aura-cli` / `main.rs` | the exhaustive argv match; `run_sample` drain→fold | + +The crate-homes are dictated by the existing structure, not chosen: a columnar +encoding is the same family as `join_on_ts` (a pure reduction over +`&[(Timestamp, Vec)]`), the on-disk store is the same family as the +registry's directory-co-located stores, and the HTML assembly is the same family as +`render_html`. Engine purity holds: `aura-engine` does the *encoding* (struct → +JSON), never file I/O; the registry crate does the I/O; the CLI does stdout + +browser hand-off. No pixel/UI knowledge enters the engine (C14). + +Determinism (C1) and causality (C2/C3) are untouched: persistence is a **post-run** +reduction over already-recorded sink output, exactly like `summarize`. `join_on_ts` +stays a post-run reduction (C3: not an in-graph join), now invoked at *serve* time +rather than only in example code. + +## Concrete code shapes + +### 1. The user-facing flow (the criterion's empirical evidence) + +A researcher who just ran a backtest wants to *see* the equity curve and the +exposure, not read three numbers. The reach is two commands: + +```console +# persist the run's taps under runs/traces/demo/ (default run output unchanged) +$ aura run --trace demo +{"manifest":{...},"metrics":{"total_pips":...,"max_drawdown":...,"exposure_sign_flips":...}} + +$ ls runs/traces/demo/ +index.json equity.json exposure.json + +# serve them as a self-contained chart page (overlay: both feeds, shared ts axis) +$ aura chart demo > demo.html && xdg-open demo.html + +# or timestamp-aligned stacked panels (one per feed, shared time axis) +$ aura chart demo --panels > demo.html +``` + +`equity.json` — the columnar (SoA) form of one tap, chart-ready (one column = one +series), kind-tagged for honesty (C7): + +```json +{ + "tap": "equity", + "kinds": ["F64"], + "ts": [1, 2, 3, 4, 5, 6, 7], + "columns": [[0.0, 0.0, 0.0010, 0.0040, 0.0040, 0.0010, -0.0010]] +} +``` + +`index.json` — the tap list (deterministic read order) + the run manifest, so the +page header can show run context (commit / params / window / broker), mirroring +`aura graph`'s header: + +```json +{ "manifest": { "commit": "...", "params": [...], "window": [1,7], "seed": 0, + "broker": "sim-optimal(pip_size=0.0001)" }, + "taps": ["equity", "exposure"] } +``` + +The emitted page is self-contained (no remote fetch), injecting the aligned data as +`window.AURA_TRACES` and inlining vendored uPlot + a `chart-viewer.js` — exactly the +`render_html` pattern. + +### 2. `ColumnarTrace` (aura-engine / report.rs) — NEW + +```rust +/// One drained recorder tap in columnar (SoA) form: parallel arrays, one per +/// recorded column, plus the shared recorded-timestamp axis. Chart-ready (a column +/// is a series of numbers) and kind-tagged (C7: the true base type of each column +/// survives even though values are coerced to f64 for plotting). Pure: the same +/// rows always encode to the same struct (C1). +#[derive(Clone, Debug, PartialEq, serde::Serialize, serde::Deserialize)] +pub struct ColumnarTrace { + pub tap: String, + pub kinds: Vec, // "F64" | "I64" | "Bool" | "Timestamp" (no serde dep on ScalarKind) + pub ts: Vec, // recorded timestamps (Timestamp.0) + pub columns: Vec>, // columns[c][r] — coerced: f64 as-is, i64 as f64, bool 1/0, ts as f64 +} + +impl ColumnarTrace { + /// Transpose a drained tap (`(ts, row)` pairs) into columns. Empty rows -> empty + /// ts + empty columns sized to `kinds`. + pub fn from_rows(tap: &str, kinds: &[ScalarKind], rows: &[(Timestamp, Vec)]) -> Self { /* ... */ } + + /// Inverse for the serve/align path: rebuild `(ts, row)` pairs with each cell as + /// `Scalar::f64(columns[c][r])`. The on-disk store is already f64 columns, so the + /// rebuilt rows are **uniformly f64** (the original base type survives only in + /// `kinds` as display metadata) — which keeps the serve-side `as_f64` projection + /// (§6 step 3) total, never the F64-only panic path. A true value round-trip for + /// the shipped F64 taps; for other kinds the value round-trips through the f64 + /// store with `kinds` carrying the base type. + pub fn to_rows(&self) -> Vec<(Timestamp, Vec)> { /* ... */ } +} +``` + +### 3. `TraceStore` (aura-registry / trace_store.rs) — NEW + +```rust +/// A per-run directory of columnar tap files under `/traces//`, +/// beside the run registry's `runs.jsonl`. Directory-keyed (like the family +/// sibling store): one subdir per run name, one `.json` per tap, plus +/// `index.json` (tap order + manifest). +pub struct TraceStore { dir: PathBuf } // dir = /traces + +impl TraceStore { + pub fn open(runs_dir: impl AsRef) -> TraceStore { /* runs_dir/traces */ } + + /// Write one run's taps + index under traces//, creating dirs as needed. + pub fn write(&self, name: &str, manifest: &RunManifest, taps: &[ColumnarTrace]) -> Result<(), TraceStoreError> { /* ... */ } + + /// Read a run's index + taps back, in index order. A missing run dir is + /// `NotFound` (treat-as-absent, C18-consistent), not a panic. + pub fn read(&self, name: &str) -> Result { /* ... */ } +} + +pub struct RunTraces { pub manifest: RunManifest, pub taps: Vec } +``` + +### 4. Chart-page assembly (aura-cli / render.rs) — NEW, mirrors `render_html` + +```rust +const UPLOT_JS: &str = include_str!("../assets/uPlot.iife.min.js"); +const UPLOT_CSS: &str = include_str!("../assets/uPlot.min.css"); +const CHART_VIEWER_JS: &str = include_str!("../assets/chart-viewer.js"); // NEW asset + +pub enum ChartMode { Overlay, Panels } + +/// Self-contained chart page for one run's taps. `data` (a `ChartData`, §6) carries +/// the taps aligned on the shared union-spine x — one `Series` per (tap, column), +/// each with its own y-scale id so disparate magnitudes stay legible; `mode` selects +/// overlay vs panels over the *same* data. The page fetches nothing at view time. +pub fn render_chart_html(data: &ChartData, mode: ChartMode) -> String { + // SHELL_HEAD-style dark shell + run-context header + // + + + // + window.AURA_TRACES = {mode, manifest, taps, xs, series:[{name,y_scale_id,points}]} + // + // overlay: one uPlot, per-series y-scale; + // // panels: one uPlot per series, shared xs +} +``` + +### 5. CLI wiring (aura-cli / main.rs) — before → after + +`--trace ` is an **orthogonal modifier** that composes with **every** run form +— plain `run`, `run --macd`, and `run --real `. All three already drain the +identical two taps (`rx_eq` / `rx_ex`) and build a `RunManifest` (`run_sample`:159, +`run_macd`:836, `run_sample_real`:238), so one shared helper serves all three; the +persist step is inserted between drain and fold (additive; runs only when the trace +name is `Some`): + +```rust +// before: +let eq_rows: Vec<(Timestamp, Vec)> = rx_eq.try_iter().collect(); +let ex_rows: Vec<(Timestamp, Vec)> = rx_ex.try_iter().collect(); +let equity = f64_field(&eq_rows, 0); +let exposure = f64_field(&ex_rows, 0); +let metrics = summarize(&equity, &exposure); +// ... RunReport + +// shared helper (aura-cli), called by run_sample / run_macd / run_sample_real: +fn persist_traces(name: &str, manifest: &RunManifest, + eq_rows: &[(Timestamp, Vec)], + ex_rows: &[(Timestamp, Vec)]) -> Result<(), TraceStoreError> { + let taps = vec![ + ColumnarTrace::from_rows("equity", &[ScalarKind::F64], eq_rows), + ColumnarTrace::from_rows("exposure", &[ScalarKind::F64], ex_rows), + ]; + TraceStore::open("runs").write(name, manifest, &taps) +} + +// after, in each run path (manifest hoisted before the fold; trace: Option<&str>): +let eq_rows = rx_eq.try_iter().collect::>(); +let ex_rows = rx_ex.try_iter().collect::>(); +if let Some(name) = trace { persist_traces(name, &manifest, &eq_rows, &ex_rows)?; } +let metrics = summarize(&f64_field(&eq_rows, 0), &f64_field(&ex_rows, 0)); +// ... RunReport unchanged; when trace is None, stdout + behaviour are byte-for-byte as before +``` + +`main()` argv match. `--trace` composes with **every** run form; `run_sample` / +`run_macd` / `run_sample_real` each gain a `trace: Option<&str>` parameter — the +existing no-trace arms pass `None`, so their stdout stays byte-for-byte unchanged. +Every combination the new `USAGE` advertises has an explicit arm (no advertised form +falls through to the usage error): + +```rust +// before: +["run"] => println!("{}", run_sample().to_json()), +["run", "--macd"] => println!("{}", run_macd().to_json()), +["run", "--real", rest @ ..] => match parse_real_args(rest) { Ok((sym, from, to)) => ... }, + +// after — existing arms pass None (unchanged stdout); new arms thread the trace name: +["run"] => println!("{}", run_sample(None).to_json()), +["run", "--macd"] => println!("{}", run_macd(None).to_json()), +["run", "--trace", name] => println!("{}", run_sample(Some(name)).to_json()), +["run", "--macd", "--trace", name] => println!("{}", run_macd(Some(name)).to_json()), +// parse_real_args -> 4-tuple (symbol, from, to, trace: Option); the existing +// ["run","--real", rest @ ..] arm threads it: run_sample_real(&sym, from, to, trace.as_deref()) +["chart", name] => emit_chart(name, ChartMode::Overlay), +["chart", name, "--panels"] => emit_chart(name, ChartMode::Panels), +``` + +`parse_real_args` gains one tail flag — `--trace ` — alongside the existing +`--from`/`--to` pairs (same once-only, value-required grammar), returning a fourth +`Option` element; `run_sample_real` takes `trace: Option<&str>` and persists +the same two taps when it is `Some`. `emit_chart` reads the store and aligns +**all** taps as symmetric sides of a synthetic empty-payload union spine via +`join_on_ts` (the exact three-step mechanism in §6; no tap privileged, no points +dropped), producing one `ChartData` that feeds **both** modes (overlay superimposes +the series, panels stack one plot per series on that shared x), then prints the page; a +missing run name is a usage error (stderr + `exit 2`), consistent with the rest of the +CLI. `USAGE` becomes +`aura run [--macd] [--trace ] | aura run --real [--from ] [--to ] [--trace ] | aura chart [--panels] | …` +— every advertised `[--macd] [--trace]` combination has a matching arm above, so the +usage string and the dispatch cannot disagree. + +### 6. Overlay & panel chart semantics (shared timestamp x; the y-axis choice) + +**Both modes are timestamp-aligned on one shared x-axis** — the **sorted union of all +taps' recorded timestamps**, aligned via `join_on_ts` with the union as the spine +(each series null-filled where its tap did not fire — uPlot renders the gap; a *union* +spine drops no tap's points). This is exactly the user's "mehrere Feeds in einem +Chart, **oder aligned nach Timestamps in Panels**": the shared timestamp x is what +makes a vertical time-cursor line up across feeds in **both** modes (issue #101, +Fork A — timestamp-aligned panels). + +The two modes differ only in **render** and in the **y-axis**, which is load-bearing +whenever feeds differ in magnitude (pip-equity ~±0.004 vs exposure ±1.0 — a single +shared y-scale would flatten the equity curve to a line): + +- **Overlay** (default) — **one** plot, all series superimposed, **each on its own + auto-ranged y-scale** (uPlot's multi-scale: a named scale per series, independently + min/max-ranged). The first two series carry a drawn y-axis (left, right); any + further series keep their own auto-scale, read via the legend/tooltip. Suits a few + feeds you want superimposed in one frame. +- **Panels** (`--panels`) — **one stacked uPlot per series**, each on its own y-scale, + **all sharing the common union-spine timestamp x-axis** so a vertical time-cursor + lines up across the stacked panels (the "aligned nach Timestamps in Panels" the user + asked for). The clean choice when feeds are numerous or wildly different in + magnitude. + +**Serve-time alignment (both modes), the exact mechanism — no privileged tap.** +`join_on_ts(spine, sides)` keys `sides` off a caller-supplied `spine` of +`(Timestamp, Vec)` rows and drops any side row whose ts is absent from the +spine (report.rs:498); it does **not** itself compute a union. `emit_chart` therefore +builds the alignment in three concrete steps, putting **no tap in the privileged +`JoinedRow.spine` slot**: + +1. **Union x.** Concatenate every tap's `ts` array, sort, dedup → `xs: Vec` — + the shared x-axis. +2. **Symmetric join.** Synthesize a spine of *empty-payload* rows + `xs.iter().map(|&t| (Timestamp(t), Vec::new()))`, and pass **all** taps (each + rebuilt via `ColumnarTrace::to_rows`) as the `sides` of + `join_on_ts(&spine, &all_tap_rows)`. Because the spine covers every tap's + timestamps, no side row is dropped; the synthetic spine's own empty payload is + never read. Every tap is a symmetric side — none occupies the never-gap-filled + `JoinedRow.spine`. +3. **Flatten to series.** Each tap contributes **one chart series per recorded + column** (the shipped `equity`/`exposure` taps are single-column → one series + each). Tap *i*'s column *c* maps to + `joined.iter().map(|r| r.sides[i].as_ref().map(|row| row[c].as_f64()))` — the + value where tap *i* fired at that x, `None` (a uPlot gap) otherwise. The result is + `ChartData { xs: Vec, series: Vec, manifest, taps }`, where a + `Series = { name: String, y_scale_id: String, points: Vec> }` (one + per (tap, column); the `y_scale_id` is what gives overlay its per-series y-scale). + Both modes consume the *same* `ChartData`: overlay draws all `series` on one plot + over `xs`; panels draw one stacked uPlot per `series`, all over the same `xs`. + +For the first-cut sample harness the two taps (`equity`, `exposure`) are +**co-cadenced** (both fire every cycle), so the union spine is exactly their shared +timestamps; the per-series y-scales are what keep the ±0.004 equity legible beside the +±1.0 exposure. The **timestamp-aligned shared x (both modes)** is the source-mandated +contract (issue #101, Fork A); the **per-series y-scale** and the **union (vs. +one-tap) spine** are documented naive-viewer defaults settled under the +autonomous-build mandate and recorded on issue #101 — not user-picked design forks, +but documented orchestrator decisions within that mandate. + +## Components + +- **`ColumnarTrace`** (aura-engine): transpose/round-trip of one tap. Pure. +- **`TraceStore` + `RunTraces` + `TraceStoreError`** (aura-registry): the on-disk + per-run trace directory; write + read; missing = `NotFound`. +- **`render_chart_html` + `ChartMode` + `ChartData`** (aura-cli): page assembly. +- **Vendored assets** (aura-cli/assets): `uPlot.iife.min.js`, `uPlot.min.css` + (third-party, pinned + recorded in `refresh-assets.sh` like the existing + Graphviz-WASM/pan-zoom blobs), and `chart-viewer.js` (first-party, the + `AURA_TRACES` → uPlot bootstrap). +- **CLI** (aura-cli/main.rs): the shared `persist_traces` helper threaded into + `run_sample` / `run_macd` / `run_sample_real` (each `trace: Option<&str>`), + `emit_chart`, the new argv arms, and the serve-time union-spine `join_on_ts` + alignment shared by both chart modes. + +## Data flow + +``` +aura run --trace demo + harness.run() -> recorder taps drained -> Vec<(Timestamp, Vec)> per tap + | | + | ColumnarTrace::from_rows (transpose, coerce, kind-tag) + | v + | TraceStore::write -> runs/traces/demo/{index,equity,exposure}.json + v + summarize -> RunReport -> stdout (UNCHANGED) + +aura chart demo [--panels] + TraceStore::read(demo) -> RunTraces { manifest, taps: [ColumnarTrace] } + | | + | synthetic empty union-spine + ALL taps as sides -> join_on_ts -> ChartData + | { xs, series:[{name,y_scale_id,points: Option per x}] } (null = gap, no drop) + | overlay: ONE plot, all series, per-series y-scale + | panels: ONE stacked plot per series, all on the shared xs (timestamp-aligned) + v + render_chart_html(ChartData, mode) -> self-contained HTML -> stdout -> browser (uPlot) +``` + +## Error handling + +- **Unknown run name** (`aura chart ` with no `runs/traces//`): + `TraceStore::read` returns `NotFound`; the CLI prints `aura: no recorded run + '' under runs/traces` to stderr and `exit 2` — usage-error parity with the + rest of the CLI, never a panic. (Consistent with C18's treat-missing-as-absent.) +- **Corrupt / unreadable trace file**: `TraceStoreError::{Io, Parse{file,source}}` + surfaced like `RegistryError` (file named, propagated), `exit 2`. +- **`--trace` write failure** (I/O): propagated from the run path as a usage-level + error (stderr + `exit 2`); the run still computed its metrics, so the message + names the persistence step, not the run. +- **Empty taps** (a run that recorded nothing): `from_rows` yields empty arrays; + `render_chart_html` emits a page with an empty chart and a "no recorded samples" + note — no panic, no division-by-zero. +- **Bool/ts coercion**: documented and total (bool→1.0/0.0, ts→f64); `kinds` + preserves the real base type so the store stays honest (C7). + +## Testing strategy + +- **`ColumnarTrace` round-trip** (aura-engine unit): `from_rows` then `to_rows` + reproduces the row set; columnar shape asserted (ts length, column count = + kinds, per-kind coercion); empty-rows case. +- **`TraceStore` write→read** (aura-registry unit, temp dir): a written run reads + back equal (manifest + taps in index order); missing run = `NotFound`; a + corrupt tap file = `Parse{file,..}` with the file named. Temp-dir isolation + mirrors the registry's `temp_family_dir` pattern. +- **`render_chart_html` self-contained** (aura-cli unit, mirrors + `render_html_is_self_contained_and_embeds_the_model`): starts ``, + closes ``, no remote `