Aura/docs/specs/0062-visual-world-cut-2.md

Visual World cut 2 — served-page hardening (decimation + run-context header) — Design Spec

Date: 2026-06-22 Status: Draft — awaiting user spec review Authors: orchestrator + Claude

Bundles two tracker issues into one cut on the chart served page:

• #108 — serve-time point decimation / LOD (multi-year real-data pages reach 100s of MB and are browser-unopenable).
• #102 — surface a run-context header (manifest / window / broker / taps) in the served page.

Both touch one code locus — ChartData / render_chart_html in crates/aura-cli/src/render.rs, build_chart_data / build_comparison_chart_data / emit_chart in crates/aura-cli/src/main.rs, and chart-viewer.js — and one purpose under C22/C14: turn the #107 families-comparison first cut into a page a (usually remote) researcher can actually open and read on REAL multi-year families.

The derived fork decisions for this cut are recorded on the reference issue (Brummel/Aura #108, "Design reconciliation (specify)") and are not re-litigated here.

Goal

1. Decimation (#108). A charted page's size must stop growing with the underlying point count. A 5y M1 family (1.6M points × 4 members = 148 MB today) must render a few-thousand-point page with its visual shape (min/max envelope) preserved. Full recorded data stays on disk; only the served page is thinned.
2. Run-context header (#102). The served page must say what it is — name, broker, data window, commit, charted taps (and member count for a family) — instead of an anonymous set of curves. The manifest is already persisted and read back; today it is deliberately dropped from the page. Re-include it and render it client-side.

Non-goals: an Arrow/Parquet streaming/range-query store (the deferred "scale path"); a --width/--max-points CLI knob (a fixed viewport-scale default this cut, a clean future add); the page chrome fix (the chart page already wears its own aura chart title/header since the label split — only the run-context is missing). Engine + registry are untouched (C14).

Architecture

Both halves live entirely in the CLI render path (C14: no UI / pixel knowledge in the engine). The pipeline in emit_chart becomes:

read traces ─► build ChartData (xs + series + meta)  ─► [--tap filter] ─► decimate ─► render_chart_html
                         │                                                    │
                  meta from RunManifest                       pure ChartData->ChartData transform
                  (#102, both paths)                          (#108, both paths), meta passed through

• #108 is a pure, deterministic (C1) transform decimate(ChartData, buckets) -> ChartData applied to the already-aligned ChartData (shared xs + per-series Option<f64>) right before rendering, on both the single-run and the family path. Operating after the union-join means a null-only bucket stays null, so the walk-forward null-fill blow-up collapses on the page for free (output is bounded to N × O(buckets) for sweep / MC / WFO alike). Transient generation memory (the full join before decimation) is unchanged and out of scope — the runs are "fine" per #108; only the page is the cliff.
• #102 adds a meta: ChartMeta field to ChartData, populated from the RunManifest (single run) or the shared family manifest (family), serialized into window.AURA_TRACES.meta, and rendered by a new pure buildHeader(meta) in chart-viewer.js — mirroring the pure buildCharts + headless .mjs guard architecture.

Concrete code shapes

The user-facing program (the acceptance criterion's evidence)

# A researcher charts a 5-year GER40 sweep family (4 members, ~1.6M M1 pts each),
# on a remote session (charts served over python3 -m http.server):
$ aura sweep --real GER40 …window…  --trace ger40-5y   # produces a 4-member family
$ aura chart ger40-5y > page.html

# BEFORE this cut: page.html ≈ 148 MB — the browser stalls / cannot open it.
# AFTER:           page.html ≈ a few MB; opens instantly; the min/max envelope of
#                  every member curve is preserved (no spike dropped).

# AND the page is no longer an anonymous set of curves — it carries a header:
#   aura chart   family: ger40-5y · members 4 · tap equity ·
#                broker sim-optimal(pip_size=1) · window 2019-01-01→2023-12-31 · commit 4c64feb

A single run reads the same way, with its bound params instead of a member count:

$ aura run --real EURUSD --trace eur-demo && aura chart eur-demo > page.html
#   aura chart   run: eur-demo · tap equity, exposure ·
#                broker sim-optimal(pip_size=0.0001) · window 2024-01-01→2024-12-31 · commit 4c64feb

#102 — ChartMeta and ChartData (render.rs)

// NEW: the run-context for the page header (#102). Serialized into
// window.AURA_TRACES.meta and rendered client-side by chart-viewer.js's pure
// buildHeader. Single run: the run's manifest fields + bound params. Family: the
// shared context (commit/window/broker) + member count; per-member identity is the
// series label (member.key), not repeated here.
#[derive(serde::Serialize, Default)]
pub struct ChartMeta {
    pub kind: String,                  // "run" | "family"
    pub name: String,                  // run name or family id
    pub commit: String,                // git commit (C18); buildHeader shows the short form
    pub window: (i64, i64),            // inclusive (from, to) epoch-ns
    pub broker: String,                // e.g. "sim-optimal(pip_size=1)"
    pub seed: u64,                     // 0 for a seed-free synthetic run
    pub taps: Vec<String>,             // the charted taps (single: all/--tap; family: the one compared)
    pub members: Option<usize>,        // family member count; None for a single run
    pub params: Vec<(String, String)>, // single run: bound params name->display; empty for a family
}

// BEFORE
pub struct ChartData {
    pub xs: Vec<i64>,
    pub series: Vec<Series>,
}
// AFTER
pub struct ChartData {
    pub xs: Vec<i64>,
    pub series: Vec<Series>,
    pub meta: ChartMeta,
}

render_chart_html injects it (the only change to the function body):

// BEFORE
let traces = serde_json::json!({ "mode": mode_str, "xs": data.xs, "series": data.series }).to_string();
// AFTER
let traces = serde_json::json!({ "mode": mode_str, "xs": data.xs, "series": data.series, "meta": data.meta }).to_string();

The ChartData doc comment (render.rs:135-139, "The run's manifest stays on disk … it is not carried into the served page") is reworded: the manifest context is now carried as meta and read by buildHeader.

#108 — decimate (main.rs, beside build_chart_data)

/// Default decimation budget: target horizontal buckets. ~2000 buckets ⇒ ≤ ~4000
/// spine slots (min+max per bucket) — a few-thousand-point page regardless of the
/// underlying multi-year M1 point count.
const CHART_DECIMATE_BUCKETS: usize = 2000;

/// Serve-time min-max decimation on the aligned `ChartData` (#108). Partition the
/// shared `xs` into at most `buckets` contiguous index ranges; per bucket emit the
/// bucket's first and last timestamp as two shared spine slots, and for each series
/// the min and max of its non-null values in that range (min at the first slot, max
/// at the second — sub-pixel ordering, so the drawn vertical extent of the column is
/// preserved). A bucket spanning one timestamp collapses to a single slot; an
/// all-null bucket emits null. `meta` passes through unchanged. Deterministic (C1):
/// same input ⇒ same output. Full data stays on disk; only the served page is
/// thinned. No-op (returns `data` unchanged) when `xs.len() <= 2 * buckets`.
fn decimate(data: ChartData, buckets: usize) -> ChartData {
    // xs is sorted + deduped (strictly increasing) ⇒ bucket boundary timestamps are
    // strictly increasing across buckets, so the decimated spine stays monotonic
    // (uPlot requires it). Per series, fold each bucket's non-null Option<f64> to
    // (min, max); place min then max at the bucket's two boundary slots.
    // … exact bytes are the planner's.
}

Applied in emit_chart on both arms (the join is untouched):

// Run arm (BEFORE):  let mut data = build_chart_data(traces);
//                    if let Some(t) = tap { data = filter_to_tap(data, t)?; }
//                    print!("{}", render::render_chart_html(&data, mode));
// Run arm (AFTER):   let mut data = build_chart_data(name, traces);          // name -> meta
//                    if let Some(t) = tap { data = filter_to_tap(data, t)?; } // narrows meta.taps too
//                    let data = decimate(data, CHART_DECIMATE_BUCKETS);
//                    print!("{}", render::render_chart_html(&data, mode));

// Family arm (AFTER): let data = build_comparison_chart_data(name, &members, tap.unwrap_or("equity"))?;
//                     let data = decimate(data, CHART_DECIMATE_BUCKETS);
//                     print!("{}", render::render_chart_html(&data, mode));

#102 — meta construction (main.rs)

build_chart_data gains the run name and builds meta from traces.manifest (it currently drops the manifest); filter_to_tap narrows meta.taps to the picked tap; build_comparison_chart_data gains name and builds meta from the shared manifest of members[0] plus the member count and the compared tap.

// build_chart_data (single run): manifest -> ChartMeta
let m = &traces.manifest;
let meta = ChartMeta {
    kind: "run".into(),
    name: name.to_string(),
    commit: m.commit.clone(),
    window: (m.window.0.0, m.window.1.0),
    broker: m.broker.clone(),
    seed: m.seed,
    taps: traces.taps.iter().map(|t| t.tap.clone()).collect(),
    members: None,
    params: m.params.iter().map(|(k, v)| (k.clone(), scalar_display(v))).collect(),
};
// … xs/series built as today; ChartData { xs, series, meta }

// build_comparison_chart_data (family): shared manifest -> ChartMeta
let m = &members[0].traces.manifest;
let meta = ChartMeta {
    kind: "family".into(),
    name: name.to_string(),
    commit: m.commit.clone(),
    window: (m.window.0.0, m.window.1.0),
    broker: m.broker.clone(),
    seed: m.seed,
    taps: vec![tap.to_string()],
    members: Some(members.len()),
    params: Vec::new(), // per-member params are the series labels
};

scalar_display renders a Scalar to a compact string for the header (an existing display path may be reused; the planner picks it).

#102 — buildHeader (chart-viewer.js, pure, headless-guarded)

// {kind,name,commit,window:[from,to],broker,seed,taps,members,params} -> array of
// {label, value} chips. PURE (no DOM), the unit the headless guard drives; mount()
// renders the chips into the header's #ctx slot. Empty meta -> [].
function buildHeader(meta) {
  if (!meta) return [];
  var fmtDay = function (ns) { return new Date(ns / 1e6).toISOString().slice(0, 10); };
  var chips = [];
  chips.push({ label: meta.kind === "family" ? "family" : "run", value: meta.name });
  if (meta.members != null) chips.push({ label: "members", value: String(meta.members) });
  if (meta.taps && meta.taps.length) chips.push({ label: "tap", value: meta.taps.join(", ") });
  if (meta.broker) chips.push({ label: "broker", value: meta.broker });
  if (meta.window) chips.push({ label: "window", value: fmtDay(meta.window[0]) + "→" + fmtDay(meta.window[1]) });
  if (meta.seed) chips.push({ label: "seed", value: String(meta.seed) });
  if (meta.commit) chips.push({ label: "commit", value: meta.commit.slice(0, 7) });
  if (meta.params && meta.params.length)
    chips.push({ label: "params", value: meta.params.map(function (p) { return p[0] + "=" + p[1]; }).join(" ") });
  return chips;
}
// exported beside buildCharts/keyXRange; mount() reads window.AURA_TRACES.meta and
// fills <span id="ctx"> with the chips.

CHART_HEAD (render.rs:65-72) gains a <span id="ctx" class="sub"></span> slot between the aura chart label and the #xmode-toggle button (the toggle keeps margin-left:auto, so it stays right).

Components

Component	File	Change
ChartMeta	crates/aura-cli/src/render.rs	NEW serializable run-context struct
ChartData	crates/aura-cli/src/render.rs	+ meta: ChartMeta field; doc reworded
render_chart_html	crates/aura-cli/src/render.rs	inject "meta" into window.AURA_TRACES
CHART_HEAD	crates/aura-cli/src/render.rs	+ #ctx header slot
decimate + CHART_DECIMATE_BUCKETS	crates/aura-cli/src/main.rs	NEW pure transform + default
build_chart_data	crates/aura-cli/src/main.rs	+ name param; build meta from manifest
filter_to_tap	crates/aura-cli/src/main.rs	narrow meta.taps to the picked tap
build_comparison_chart_data	crates/aura-cli/src/main.rs	+ name param; build family meta
emit_chart	crates/aura-cli/src/main.rs	apply decimate on both arms
buildHeader + mount	crates/aura-cli/assets/chart-viewer.js	NEW pure header builder; render into #ctx

Data flow

1. emit_chart(name, tap, mode) classifies the name (name_kind) — unchanged.
2. Run: read(name) → build_chart_data(name, traces) builds xs/series (union-join, unchanged) and meta from traces.manifest; optional filter_to_tap narrows series + meta.taps. Family: read_family(name) → build_comparison_chart_data(name, members, tap) builds one series per member on a shared y-scale (unchanged) and meta from the shared manifest + member count.
3. decimate(data, CHART_DECIMATE_BUCKETS) thins xs+series (meta passes through). No-op when already within budget (every existing small fixture).
4. render_chart_html injects {mode, xs, series, meta} as window.AURA_TRACES.
5. In the browser, mount() calls buildCharts (unchanged) and buildHeader, filling #ctx with the run-context chips.

Error handling

• decimate is total: any ChartData in, a valid ChartData out; empty xs → unchanged; buckets == 0 is never passed (the const is non-zero), but a defensive buckets.max(1) keeps it total.
• No new exit paths. The existing refuse-don't-guess errors (NotFound, no-such-tap → stderr + exit 2) are unchanged.
• buildHeader(undefined) → [] (a page built by an older CLI without meta still renders its charts; the header is simply empty).

Testing strategy

• decimate unit tests (crates/aura-cli): (a) bounds — a 10 000-point ChartData decimated to 2000 buckets yields xs.len() <= 4000; (b) min/max preserved — a series with a single tall spike inside a bucket keeps that spike value in the output (max survives); (c) nulls preserved — an all-null bucket emits null; (d) no-op under budget — xs.len() <= 2*buckets returns the data unchanged; (e) meta passthrough — meta is identical in/out; (f) monotonic — output xs strictly increasing.
• render.rs render tests: flip render_chart_html_injects_only_what_the_viewer_reads (asserts !contains("manifest") / !contains("taps")) → a positive render_chart_html_injects_run_context asserting the page carries "meta", the commit, the broker, and "kind":"run". Update sample_chart_data() to construct a ChartMeta. Keep …is_self_contained_and_injects_traces, …wears_its_own_chart_label, …reflects_panels_mode.
• buildHeader headless guard: new tests/chart_viewer_header.mjs (loads the real chart-viewer.js, drives buildHeader) + tests/chart_viewer_header.rs (shells out to node, mirroring chart_viewer.rs). Asserts: a run meta yields chips for name/tap/broker/window/commit; a family meta adds a members chip; buildHeader(undefined) → [].
• Existing E2E (tests/cli_run.rs family + single chart tests) stays green: the fixtures are small, so decimate is a no-op, and meta is additive. If any asserts exact injected bytes, extend it to tolerate the new meta key.

Acceptance criteria

Against aura's feature-acceptance criterion (CLAUDE.md):

• The audience naturally reaches for it. A researcher charting a multi-year real-data family today gets a browser-unopenable 148 MB page; after this cut the page opens and tells them which run/window/broker they are looking at — the exact friction surfaced when probing the #107 view on real data.
• Measurably improves correctness / removes a failure. The page size decouples from the point count (≤ ~4000 spine slots/series vs millions); the WFO null-fill page collapses; the dropped run-context is restored. The min/max envelope is preserved (no spike lost).
• Reintroduces no failure class. Engine + registry untouched (C14: the view layer owns pixels, not the engine). decimate is a pure deterministic transform (C1). Full recorded data stays on disk unchanged — decimation is view-only, never a data mutation. No look-ahead / merge / streaming invariant is in scope (view layer).
• Concrete evidence: the worked aura chart ger40-5y program above (page-size before/after + the header line) is the empirical check.