From a8e11e7733e6e9f1611c4647d86c921a8b17c76a Mon Sep 17 00:00:00 2001 From: Brummel Date: Mon, 22 Jun 2026 11:36:22 +0200 Subject: [PATCH] =?UTF-8?q?spec:=200062=20visual-world-cut-2=20=E2=80=94?= =?UTF-8?q?=20decimation=20+=20run-context=20header=20(boss-signed)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Served-page hardening for the families-comparison chart, bundling two issues that share one code locus (the aura-cli render path) and one purpose (C22/C14: make the #107 view openable + legible on real multi-year families): - #108 — serve-time min-max decimation as a pure ChartData->ChartData transform in emit_chart (both single-run and family paths), bounding the served page to a few thousand points regardless of the underlying M1 point count; full data stays on disk; the WFO null-fill page collapses for free. Engine/registry untouched. - #102 — a ChartMeta threaded from RunManifest into window.AURA_TRACES, rendered by a new pure buildHeader in chart-viewer.js (headless .mjs guarded), showing name/window/broker/commit/taps (+ member count for a family). Signed under /boss on the Step-5 grounding-check PASS (10/10 load-bearing assumptions ratified by green tests). refs #108 refs #102 --- docs/specs/0062-visual-world-cut-2.md | 336 ++++++++++++++++++++++++++ 1 file changed, 336 insertions(+) create mode 100644 docs/specs/0062-visual-world-cut-2.md diff --git a/docs/specs/0062-visual-world-cut-2.md b/docs/specs/0062-visual-world-cut-2.md new file mode 100644 index 0000000..840488c --- /dev/null +++ b/docs/specs/0062-visual-world-cut-2.md @@ -0,0 +1,336 @@ +# Visual World cut 2 — served-page hardening (decimation + run-context header) — Design Spec + +**Date:** 2026-06-22 +**Status:** Draft — awaiting user spec review +**Authors:** orchestrator + Claude + +Bundles two tracker issues into one cut on the chart served page: +- **#108** — serve-time point decimation / LOD (multi-year real-data pages reach + 100s of MB and are browser-unopenable). +- **#102** — surface a run-context header (manifest / window / broker / taps) in + the served page. + +Both touch one code locus — `ChartData` / `render_chart_html` in +`crates/aura-cli/src/render.rs`, `build_chart_data` / `build_comparison_chart_data` +/ `emit_chart` in `crates/aura-cli/src/main.rs`, and `chart-viewer.js` — and one +purpose under C22/C14: turn the #107 families-comparison first cut into a page a +(usually remote) researcher can actually open and read on REAL multi-year families. + +The derived fork decisions for this cut are recorded on the reference issue +(Brummel/Aura #108, "Design reconciliation (specify)") and are not re-litigated +here. + +## Goal + +1. **Decimation (#108).** A charted page's size must stop growing with the + underlying point count. A 5y M1 family (1.6M points × 4 members = 148 MB today) + must render a few-thousand-point page with its visual shape (min/max envelope) + preserved. Full recorded data stays on disk; only the *served page* is thinned. +2. **Run-context header (#102).** The served page must say *what it is* — name, + broker, data window, commit, charted taps (and member count for a family) — + instead of an anonymous set of curves. The manifest is already persisted and + read back; today it is deliberately dropped from the page. Re-include it and + render it client-side. + +Non-goals: an Arrow/Parquet streaming/range-query store (the deferred "scale +path"); a `--width`/`--max-points` CLI knob (a fixed viewport-scale default this +cut, a clean future add); the page chrome fix (the chart page already wears its +own `aura chart` title/header since the label split — only the run-context is +missing). Engine + registry are untouched (C14). + +## Architecture + +Both halves live entirely in the CLI render path (C14: no UI / pixel knowledge in +the engine). The pipeline in `emit_chart` becomes: + +``` +read traces ─► build ChartData (xs + series + meta) ─► [--tap filter] ─► decimate ─► render_chart_html + │ │ + meta from RunManifest pure ChartData->ChartData transform + (#102, both paths) (#108, both paths), meta passed through +``` + +- **#108** is a pure, deterministic (C1) transform `decimate(ChartData, buckets) + -> ChartData` applied to the *already-aligned* `ChartData` (shared `xs` + + per-series `Option`) right before rendering, on both the single-run and the + family path. Operating after the union-join means a null-only bucket stays null, + so the walk-forward null-fill blow-up collapses on the page for free (output is + bounded to N × O(buckets) for sweep / MC / WFO alike). Transient generation + memory (the full join before decimation) is unchanged and out of scope — the + runs are "fine" per #108; only the page is the cliff. +- **#102** adds a `meta: ChartMeta` field to `ChartData`, populated from the + `RunManifest` (single run) or the shared family manifest (family), serialized + into `window.AURA_TRACES.meta`, and rendered by a new pure `buildHeader(meta)` + in `chart-viewer.js` — mirroring the pure `buildCharts` + headless `.mjs` guard + architecture. + +## Concrete code shapes + +### The user-facing program (the acceptance criterion's evidence) + +```console +# A researcher charts a 5-year GER40 sweep family (4 members, ~1.6M M1 pts each), +# on a remote session (charts served over python3 -m http.server): +$ aura sweep --real GER40 …window… --trace ger40-5y # produces a 4-member family +$ aura chart ger40-5y > page.html + +# BEFORE this cut: page.html ≈ 148 MB — the browser stalls / cannot open it. +# AFTER: page.html ≈ a few MB; opens instantly; the min/max envelope of +# every member curve is preserved (no spike dropped). + +# AND the page is no longer an anonymous set of curves — it carries a header: +# aura chart family: ger40-5y · members 4 · tap equity · +# broker sim-optimal(pip_size=1) · window 2019-01-01→2023-12-31 · commit 4c64feb +``` + +A single run reads the same way, with its bound params instead of a member count: + +```console +$ aura run --real EURUSD --trace eur-demo && aura chart eur-demo > page.html +# aura chart run: eur-demo · tap equity, exposure · +# broker sim-optimal(pip_size=0.0001) · window 2024-01-01→2024-12-31 · commit 4c64feb +``` + +### #102 — `ChartMeta` and `ChartData` (render.rs) + +```rust +// NEW: the run-context for the page header (#102). Serialized into +// window.AURA_TRACES.meta and rendered client-side by chart-viewer.js's pure +// buildHeader. Single run: the run's manifest fields + bound params. Family: the +// shared context (commit/window/broker) + member count; per-member identity is the +// series label (member.key), not repeated here. +#[derive(serde::Serialize, Default)] +pub struct ChartMeta { + pub kind: String, // "run" | "family" + pub name: String, // run name or family id + pub commit: String, // git commit (C18); buildHeader shows the short form + pub window: (i64, i64), // inclusive (from, to) epoch-ns + pub broker: String, // e.g. "sim-optimal(pip_size=1)" + pub seed: u64, // 0 for a seed-free synthetic run + pub taps: Vec, // the charted taps (single: all/--tap; family: the one compared) + pub members: Option, // family member count; None for a single run + pub params: Vec<(String, String)>, // single run: bound params name->display; empty for a family +} + +// BEFORE +pub struct ChartData { + pub xs: Vec, + pub series: Vec, +} +// AFTER +pub struct ChartData { + pub xs: Vec, + pub series: Vec, + pub meta: ChartMeta, +} +``` + +`render_chart_html` injects it (the only change to the function body): + +```rust +// BEFORE +let traces = serde_json::json!({ "mode": mode_str, "xs": data.xs, "series": data.series }).to_string(); +// AFTER +let traces = serde_json::json!({ "mode": mode_str, "xs": data.xs, "series": data.series, "meta": data.meta }).to_string(); +``` + +The `ChartData` doc comment (render.rs:135-139, "The run's manifest stays on disk +… it is not carried into the served page") is reworded: the manifest context is +now carried as `meta` and read by `buildHeader`. + +### #108 — `decimate` (main.rs, beside `build_chart_data`) + +```rust +/// Default decimation budget: target horizontal buckets. ~2000 buckets ⇒ ≤ ~4000 +/// spine slots (min+max per bucket) — a few-thousand-point page regardless of the +/// underlying multi-year M1 point count. +const CHART_DECIMATE_BUCKETS: usize = 2000; + +/// Serve-time min-max decimation on the aligned `ChartData` (#108). Partition the +/// shared `xs` into at most `buckets` contiguous index ranges; per bucket emit the +/// bucket's first and last timestamp as two shared spine slots, and for each series +/// the min and max of its non-null values in that range (min at the first slot, max +/// at the second — sub-pixel ordering, so the drawn vertical extent of the column is +/// preserved). A bucket spanning one timestamp collapses to a single slot; an +/// all-null bucket emits null. `meta` passes through unchanged. Deterministic (C1): +/// same input ⇒ same output. Full data stays on disk; only the served page is +/// thinned. No-op (returns `data` unchanged) when `xs.len() <= 2 * buckets`. +fn decimate(data: ChartData, buckets: usize) -> ChartData { + // xs is sorted + deduped (strictly increasing) ⇒ bucket boundary timestamps are + // strictly increasing across buckets, so the decimated spine stays monotonic + // (uPlot requires it). Per series, fold each bucket's non-null Option to + // (min, max); place min then max at the bucket's two boundary slots. + // … exact bytes are the planner's. +} +``` + +Applied in `emit_chart` on both arms (the join is untouched): + +```rust +// Run arm (BEFORE): let mut data = build_chart_data(traces); +// if let Some(t) = tap { data = filter_to_tap(data, t)?; } +// print!("{}", render::render_chart_html(&data, mode)); +// Run arm (AFTER): let mut data = build_chart_data(name, traces); // name -> meta +// if let Some(t) = tap { data = filter_to_tap(data, t)?; } // narrows meta.taps too +// let data = decimate(data, CHART_DECIMATE_BUCKETS); +// print!("{}", render::render_chart_html(&data, mode)); + +// Family arm (AFTER): let data = build_comparison_chart_data(name, &members, tap.unwrap_or("equity"))?; +// let data = decimate(data, CHART_DECIMATE_BUCKETS); +// print!("{}", render::render_chart_html(&data, mode)); +``` + +### #102 — meta construction (main.rs) + +`build_chart_data` gains the run `name` and builds `meta` from `traces.manifest` +(it currently drops the manifest); `filter_to_tap` narrows `meta.taps` to the +picked tap; `build_comparison_chart_data` gains `name` and builds `meta` from the +shared manifest of `members[0]` plus the member count and the compared tap. + +```rust +// build_chart_data (single run): manifest -> ChartMeta +let m = &traces.manifest; +let meta = ChartMeta { + kind: "run".into(), + name: name.to_string(), + commit: m.commit.clone(), + window: (m.window.0.0, m.window.1.0), + broker: m.broker.clone(), + seed: m.seed, + taps: traces.taps.iter().map(|t| t.tap.clone()).collect(), + members: None, + params: m.params.iter().map(|(k, v)| (k.clone(), scalar_display(v))).collect(), +}; +// … xs/series built as today; ChartData { xs, series, meta } + +// build_comparison_chart_data (family): shared manifest -> ChartMeta +let m = &members[0].traces.manifest; +let meta = ChartMeta { + kind: "family".into(), + name: name.to_string(), + commit: m.commit.clone(), + window: (m.window.0.0, m.window.1.0), + broker: m.broker.clone(), + seed: m.seed, + taps: vec![tap.to_string()], + members: Some(members.len()), + params: Vec::new(), // per-member params are the series labels +}; +``` + +`scalar_display` renders a `Scalar` to a compact string for the header (an +existing display path may be reused; the planner picks it). + +### #102 — `buildHeader` (chart-viewer.js, pure, headless-guarded) + +```js +// {kind,name,commit,window:[from,to],broker,seed,taps,members,params} -> array of +// {label, value} chips. PURE (no DOM), the unit the headless guard drives; mount() +// renders the chips into the header's #ctx slot. Empty meta -> []. +function buildHeader(meta) { + if (!meta) return []; + var fmtDay = function (ns) { return new Date(ns / 1e6).toISOString().slice(0, 10); }; + var chips = []; + chips.push({ label: meta.kind === "family" ? "family" : "run", value: meta.name }); + if (meta.members != null) chips.push({ label: "members", value: String(meta.members) }); + if (meta.taps && meta.taps.length) chips.push({ label: "tap", value: meta.taps.join(", ") }); + if (meta.broker) chips.push({ label: "broker", value: meta.broker }); + if (meta.window) chips.push({ label: "window", value: fmtDay(meta.window[0]) + "→" + fmtDay(meta.window[1]) }); + if (meta.seed) chips.push({ label: "seed", value: String(meta.seed) }); + if (meta.commit) chips.push({ label: "commit", value: meta.commit.slice(0, 7) }); + if (meta.params && meta.params.length) + chips.push({ label: "params", value: meta.params.map(function (p) { return p[0] + "=" + p[1]; }).join(" ") }); + return chips; +} +// exported beside buildCharts/keyXRange; mount() reads window.AURA_TRACES.meta and +// fills with the chips. +``` + +`CHART_HEAD` (render.rs:65-72) gains a `` slot +between the `aura chart` label and the `#xmode-toggle` button (the toggle keeps +`margin-left:auto`, so it stays right). + +## Components + +| Component | File | Change | +|---|---|---| +| `ChartMeta` | `crates/aura-cli/src/render.rs` | NEW serializable run-context struct | +| `ChartData` | `crates/aura-cli/src/render.rs` | + `meta: ChartMeta` field; doc reworded | +| `render_chart_html` | `crates/aura-cli/src/render.rs` | inject `"meta"` into `window.AURA_TRACES` | +| `CHART_HEAD` | `crates/aura-cli/src/render.rs` | + `#ctx` header slot | +| `decimate` + `CHART_DECIMATE_BUCKETS` | `crates/aura-cli/src/main.rs` | NEW pure transform + default | +| `build_chart_data` | `crates/aura-cli/src/main.rs` | + `name` param; build `meta` from manifest | +| `filter_to_tap` | `crates/aura-cli/src/main.rs` | narrow `meta.taps` to the picked tap | +| `build_comparison_chart_data` | `crates/aura-cli/src/main.rs` | + `name` param; build family `meta` | +| `emit_chart` | `crates/aura-cli/src/main.rs` | apply `decimate` on both arms | +| `buildHeader` + `mount` | `crates/aura-cli/assets/chart-viewer.js` | NEW pure header builder; render into `#ctx` | + +## Data flow + +1. `emit_chart(name, tap, mode)` classifies the name (`name_kind`) — unchanged. +2. **Run:** `read(name)` → `build_chart_data(name, traces)` builds `xs`/`series` + (union-join, unchanged) and `meta` from `traces.manifest`; optional + `filter_to_tap` narrows series + `meta.taps`. + **Family:** `read_family(name)` → `build_comparison_chart_data(name, members, + tap)` builds one series per member on a shared y-scale (unchanged) and `meta` + from the shared manifest + member count. +3. `decimate(data, CHART_DECIMATE_BUCKETS)` thins `xs`+`series` (meta passes + through). No-op when already within budget (every existing small fixture). +4. `render_chart_html` injects `{mode, xs, series, meta}` as `window.AURA_TRACES`. +5. In the browser, `mount()` calls `buildCharts` (unchanged) and `buildHeader`, + filling `#ctx` with the run-context chips. + +## Error handling + +- `decimate` is total: any `ChartData` in, a valid `ChartData` out; empty `xs` → + unchanged; `buckets == 0` is never passed (the const is non-zero), but a + defensive `buckets.max(1)` keeps it total. +- No new exit paths. The existing refuse-don't-guess errors (`NotFound`, no-such-tap + → stderr + exit 2) are unchanged. +- `buildHeader(undefined)` → `[]` (a page built by an older CLI without `meta` + still renders its charts; the header is simply empty). + +## Testing strategy + +- **`decimate` unit tests** (`crates/aura-cli`): (a) **bounds** — a 10 000-point + `ChartData` decimated to 2000 buckets yields `xs.len() <= 4000`; (b) **min/max + preserved** — a series with a single tall spike inside a bucket keeps that spike + value in the output (max survives); (c) **nulls preserved** — an all-null bucket + emits null; (d) **no-op under budget** — `xs.len() <= 2*buckets` returns the data + unchanged; (e) **meta passthrough** — `meta` is identical in/out; (f) + **monotonic** — output `xs` strictly increasing. +- **`render.rs` render tests:** **flip** `render_chart_html_injects_only_what_the_viewer_reads` + (asserts `!contains("manifest")` / `!contains("taps")`) → a positive + `render_chart_html_injects_run_context` asserting the page carries `"meta"`, the + commit, the broker, and `"kind":"run"`. Update `sample_chart_data()` to construct + a `ChartMeta`. Keep `…is_self_contained_and_injects_traces`, + `…wears_its_own_chart_label`, `…reflects_panels_mode`. +- **`buildHeader` headless guard:** new `tests/chart_viewer_header.mjs` (loads the + real `chart-viewer.js`, drives `buildHeader`) + `tests/chart_viewer_header.rs` + (shells out to `node`, mirroring `chart_viewer.rs`). Asserts: a run meta yields + chips for name/tap/broker/window/commit; a family meta adds a `members` chip; + `buildHeader(undefined)` → `[]`. +- **Existing E2E** (`tests/cli_run.rs` family + single chart tests) stays green: + the fixtures are small, so `decimate` is a no-op, and `meta` is additive. If any + asserts exact injected bytes, extend it to tolerate the new `meta` key. + +## Acceptance criteria + +Against aura's feature-acceptance criterion (`CLAUDE.md`): + +- **The audience naturally reaches for it.** A researcher charting a multi-year + real-data family today gets a browser-unopenable 148 MB page; after this cut the + page opens and tells them which run/window/broker they are looking at — the exact + friction surfaced when probing the #107 view on real data. +- **Measurably improves correctness / removes a failure.** The page size decouples + from the point count (≤ ~4000 spine slots/series vs millions); the WFO null-fill + page collapses; the dropped run-context is restored. The min/max envelope is + preserved (no spike lost). +- **Reintroduces no failure class.** Engine + registry untouched (C14: the view + layer owns pixels, not the engine). `decimate` is a pure deterministic transform + (C1). Full recorded data stays on disk unchanged — decimation is view-only, never + a data mutation. No look-ahead / merge / streaming invariant is in scope (view + layer). +- **Concrete evidence:** the worked `aura chart ger40-5y` program above (page-size + before/after + the header line) is the empirical check. +```