Cycle-close tidy for visual-world-cut-2 (decimation #108 + run-context header #102).
Architect drift review (53eeba5..HEAD): the only drift was a STALE LEDGER — the C22
section still listed #102 (run-context header) and #108 (decimation) as deferred/open
after they shipped in 476342d. Resolved in this commit:
- the #101 first-cut note no longer claims "not yet a run-context header";
- a new "served-page hardening — cut 2" amendment records #102 (ChartMeta +
buildHeader, family window = span across members) and #108 (the pure decimate
min-max transform) as landed, both CLI-side (C14);
- the "Open architectural threads" list drops the now-landed run-context header.
Substantive gates (self-verified): C14 honoured — all decimation/header/pixel logic
is in aura-cli (render.rs/main.rs/chart-viewer.js); the only engine touch in range is
the one-line #100 cell.rs doc. C1 preserved — decimate is pure, deterministic
bucketing over the sorted-deduped spine, no-op under budget, monotonic spine (pinned
by a unit test). Regression: cargo test --workspace = 446 passed / 0 failed; cargo
clippy --workspace --all-targets -D warnings clean. No separate baseline script.
Residual debt filed as #110 (idea): the decimation budget is a fixed const (no
--width flag), and the min-then-max sub-pixel ordering is a real ts-displacement in
the non-default continuous x-mode. Both are local aura-cli refinements, neither a bug.
Ephemeral spec/plan removed at cycle close (docs/specs/0062, docs/plans/0062).
Cycle 0062 tidy: drift-clean after this ledger amendment.
This commit is contained in:
@@ -1,336 +0,0 @@
|
||||
# Visual World cut 2 — served-page hardening (decimation + run-context header) — Design Spec
|
||||
|
||||
**Date:** 2026-06-22
|
||||
**Status:** Draft — awaiting user spec review
|
||||
**Authors:** orchestrator + Claude
|
||||
|
||||
Bundles two tracker issues into one cut on the chart served page:
|
||||
- **#108** — serve-time point decimation / LOD (multi-year real-data pages reach
|
||||
100s of MB and are browser-unopenable).
|
||||
- **#102** — surface a run-context header (manifest / window / broker / taps) in
|
||||
the served page.
|
||||
|
||||
Both touch one code locus — `ChartData` / `render_chart_html` in
|
||||
`crates/aura-cli/src/render.rs`, `build_chart_data` / `build_comparison_chart_data`
|
||||
/ `emit_chart` in `crates/aura-cli/src/main.rs`, and `chart-viewer.js` — and one
|
||||
purpose under C22/C14: turn the #107 families-comparison first cut into a page a
|
||||
(usually remote) researcher can actually open and read on REAL multi-year families.
|
||||
|
||||
The derived fork decisions for this cut are recorded on the reference issue
|
||||
(Brummel/Aura #108, "Design reconciliation (specify)") and are not re-litigated
|
||||
here.
|
||||
|
||||
## Goal
|
||||
|
||||
1. **Decimation (#108).** A charted page's size must stop growing with the
|
||||
underlying point count. A 5y M1 family (1.6M points × 4 members = 148 MB today)
|
||||
must render a few-thousand-point page with its visual shape (min/max envelope)
|
||||
preserved. Full recorded data stays on disk; only the *served page* is thinned.
|
||||
2. **Run-context header (#102).** The served page must say *what it is* — name,
|
||||
broker, data window, commit, charted taps (and member count for a family) —
|
||||
instead of an anonymous set of curves. The manifest is already persisted and
|
||||
read back; today it is deliberately dropped from the page. Re-include it and
|
||||
render it client-side.
|
||||
|
||||
Non-goals: an Arrow/Parquet streaming/range-query store (the deferred "scale
|
||||
path"); a `--width`/`--max-points` CLI knob (a fixed viewport-scale default this
|
||||
cut, a clean future add); the page chrome fix (the chart page already wears its
|
||||
own `aura chart` title/header since the label split — only the run-context is
|
||||
missing). Engine + registry are untouched (C14).
|
||||
|
||||
## Architecture
|
||||
|
||||
Both halves live entirely in the CLI render path (C14: no UI / pixel knowledge in
|
||||
the engine). The pipeline in `emit_chart` becomes:
|
||||
|
||||
```
|
||||
read traces ─► build ChartData (xs + series + meta) ─► [--tap filter] ─► decimate ─► render_chart_html
|
||||
│ │
|
||||
meta from RunManifest pure ChartData->ChartData transform
|
||||
(#102, both paths) (#108, both paths), meta passed through
|
||||
```
|
||||
|
||||
- **#108** is a pure, deterministic (C1) transform `decimate(ChartData, buckets)
|
||||
-> ChartData` applied to the *already-aligned* `ChartData` (shared `xs` +
|
||||
per-series `Option<f64>`) right before rendering, on both the single-run and the
|
||||
family path. Operating after the union-join means a null-only bucket stays null,
|
||||
so the walk-forward null-fill blow-up collapses on the page for free (output is
|
||||
bounded to N × O(buckets) for sweep / MC / WFO alike). Transient generation
|
||||
memory (the full join before decimation) is unchanged and out of scope — the
|
||||
runs are "fine" per #108; only the page is the cliff.
|
||||
- **#102** adds a `meta: ChartMeta` field to `ChartData`, populated from the
|
||||
`RunManifest` (single run) or the shared family manifest (family), serialized
|
||||
into `window.AURA_TRACES.meta`, and rendered by a new pure `buildHeader(meta)`
|
||||
in `chart-viewer.js` — mirroring the pure `buildCharts` + headless `.mjs` guard
|
||||
architecture.
|
||||
|
||||
## Concrete code shapes
|
||||
|
||||
### The user-facing program (the acceptance criterion's evidence)
|
||||
|
||||
```console
|
||||
# A researcher charts a 5-year GER40 sweep family (4 members, ~1.6M M1 pts each),
|
||||
# on a remote session (charts served over python3 -m http.server):
|
||||
$ aura sweep --real GER40 …window… --trace ger40-5y # produces a 4-member family
|
||||
$ aura chart ger40-5y > page.html
|
||||
|
||||
# BEFORE this cut: page.html ≈ 148 MB — the browser stalls / cannot open it.
|
||||
# AFTER: page.html ≈ a few MB; opens instantly; the min/max envelope of
|
||||
# every member curve is preserved (no spike dropped).
|
||||
|
||||
# AND the page is no longer an anonymous set of curves — it carries a header:
|
||||
# aura chart family: ger40-5y · members 4 · tap equity ·
|
||||
# broker sim-optimal(pip_size=1) · window 2019-01-01→2023-12-31 · commit 4c64feb
|
||||
```
|
||||
|
||||
A single run reads the same way, with its bound params instead of a member count:
|
||||
|
||||
```console
|
||||
$ aura run --real EURUSD --trace eur-demo && aura chart eur-demo > page.html
|
||||
# aura chart run: eur-demo · tap equity, exposure ·
|
||||
# broker sim-optimal(pip_size=0.0001) · window 2024-01-01→2024-12-31 · commit 4c64feb
|
||||
```
|
||||
|
||||
### #102 — `ChartMeta` and `ChartData` (render.rs)
|
||||
|
||||
```rust
|
||||
// NEW: the run-context for the page header (#102). Serialized into
|
||||
// window.AURA_TRACES.meta and rendered client-side by chart-viewer.js's pure
|
||||
// buildHeader. Single run: the run's manifest fields + bound params. Family: the
|
||||
// shared context (commit/window/broker) + member count; per-member identity is the
|
||||
// series label (member.key), not repeated here.
|
||||
#[derive(serde::Serialize, Default)]
|
||||
pub struct ChartMeta {
|
||||
pub kind: String, // "run" | "family"
|
||||
pub name: String, // run name or family id
|
||||
pub commit: String, // git commit (C18); buildHeader shows the short form
|
||||
pub window: (i64, i64), // inclusive (from, to) epoch-ns
|
||||
pub broker: String, // e.g. "sim-optimal(pip_size=1)"
|
||||
pub seed: u64, // 0 for a seed-free synthetic run
|
||||
pub taps: Vec<String>, // the charted taps (single: all/--tap; family: the one compared)
|
||||
pub members: Option<usize>, // family member count; None for a single run
|
||||
pub params: Vec<(String, String)>, // single run: bound params name->display; empty for a family
|
||||
}
|
||||
|
||||
// BEFORE
|
||||
pub struct ChartData {
|
||||
pub xs: Vec<i64>,
|
||||
pub series: Vec<Series>,
|
||||
}
|
||||
// AFTER
|
||||
pub struct ChartData {
|
||||
pub xs: Vec<i64>,
|
||||
pub series: Vec<Series>,
|
||||
pub meta: ChartMeta,
|
||||
}
|
||||
```
|
||||
|
||||
`render_chart_html` injects it (the only change to the function body):
|
||||
|
||||
```rust
|
||||
// BEFORE
|
||||
let traces = serde_json::json!({ "mode": mode_str, "xs": data.xs, "series": data.series }).to_string();
|
||||
// AFTER
|
||||
let traces = serde_json::json!({ "mode": mode_str, "xs": data.xs, "series": data.series, "meta": data.meta }).to_string();
|
||||
```
|
||||
|
||||
The `ChartData` doc comment (render.rs:135-139, "The run's manifest stays on disk
|
||||
… it is not carried into the served page") is reworded: the manifest context is
|
||||
now carried as `meta` and read by `buildHeader`.
|
||||
|
||||
### #108 — `decimate` (main.rs, beside `build_chart_data`)
|
||||
|
||||
```rust
|
||||
/// Default decimation budget: target horizontal buckets. ~2000 buckets ⇒ ≤ ~4000
|
||||
/// spine slots (min+max per bucket) — a few-thousand-point page regardless of the
|
||||
/// underlying multi-year M1 point count.
|
||||
const CHART_DECIMATE_BUCKETS: usize = 2000;
|
||||
|
||||
/// Serve-time min-max decimation on the aligned `ChartData` (#108). Partition the
|
||||
/// shared `xs` into at most `buckets` contiguous index ranges; per bucket emit the
|
||||
/// bucket's first and last timestamp as two shared spine slots, and for each series
|
||||
/// the min and max of its non-null values in that range (min at the first slot, max
|
||||
/// at the second — sub-pixel ordering, so the drawn vertical extent of the column is
|
||||
/// preserved). A bucket spanning one timestamp collapses to a single slot; an
|
||||
/// all-null bucket emits null. `meta` passes through unchanged. Deterministic (C1):
|
||||
/// same input ⇒ same output. Full data stays on disk; only the served page is
|
||||
/// thinned. No-op (returns `data` unchanged) when `xs.len() <= 2 * buckets`.
|
||||
fn decimate(data: ChartData, buckets: usize) -> ChartData {
|
||||
// xs is sorted + deduped (strictly increasing) ⇒ bucket boundary timestamps are
|
||||
// strictly increasing across buckets, so the decimated spine stays monotonic
|
||||
// (uPlot requires it). Per series, fold each bucket's non-null Option<f64> to
|
||||
// (min, max); place min then max at the bucket's two boundary slots.
|
||||
// … exact bytes are the planner's.
|
||||
}
|
||||
```
|
||||
|
||||
Applied in `emit_chart` on both arms (the join is untouched):
|
||||
|
||||
```rust
|
||||
// Run arm (BEFORE): let mut data = build_chart_data(traces);
|
||||
// if let Some(t) = tap { data = filter_to_tap(data, t)?; }
|
||||
// print!("{}", render::render_chart_html(&data, mode));
|
||||
// Run arm (AFTER): let mut data = build_chart_data(name, traces); // name -> meta
|
||||
// if let Some(t) = tap { data = filter_to_tap(data, t)?; } // narrows meta.taps too
|
||||
// let data = decimate(data, CHART_DECIMATE_BUCKETS);
|
||||
// print!("{}", render::render_chart_html(&data, mode));
|
||||
|
||||
// Family arm (AFTER): let data = build_comparison_chart_data(name, &members, tap.unwrap_or("equity"))?;
|
||||
// let data = decimate(data, CHART_DECIMATE_BUCKETS);
|
||||
// print!("{}", render::render_chart_html(&data, mode));
|
||||
```
|
||||
|
||||
### #102 — meta construction (main.rs)
|
||||
|
||||
`build_chart_data` gains the run `name` and builds `meta` from `traces.manifest`
|
||||
(it currently drops the manifest); `filter_to_tap` narrows `meta.taps` to the
|
||||
picked tap; `build_comparison_chart_data` gains `name` and builds `meta` from the
|
||||
shared manifest of `members[0]` plus the member count and the compared tap.
|
||||
|
||||
```rust
|
||||
// build_chart_data (single run): manifest -> ChartMeta
|
||||
let m = &traces.manifest;
|
||||
let meta = ChartMeta {
|
||||
kind: "run".into(),
|
||||
name: name.to_string(),
|
||||
commit: m.commit.clone(),
|
||||
window: (m.window.0.0, m.window.1.0),
|
||||
broker: m.broker.clone(),
|
||||
seed: m.seed,
|
||||
taps: traces.taps.iter().map(|t| t.tap.clone()).collect(),
|
||||
members: None,
|
||||
params: m.params.iter().map(|(k, v)| (k.clone(), scalar_display(v))).collect(),
|
||||
};
|
||||
// … xs/series built as today; ChartData { xs, series, meta }
|
||||
|
||||
// build_comparison_chart_data (family): shared manifest -> ChartMeta
|
||||
let m = &members[0].traces.manifest;
|
||||
let meta = ChartMeta {
|
||||
kind: "family".into(),
|
||||
name: name.to_string(),
|
||||
commit: m.commit.clone(),
|
||||
window: (m.window.0.0, m.window.1.0),
|
||||
broker: m.broker.clone(),
|
||||
seed: m.seed,
|
||||
taps: vec![tap.to_string()],
|
||||
members: Some(members.len()),
|
||||
params: Vec::new(), // per-member params are the series labels
|
||||
};
|
||||
```
|
||||
|
||||
`scalar_display` renders a `Scalar` to a compact string for the header (an
|
||||
existing display path may be reused; the planner picks it).
|
||||
|
||||
### #102 — `buildHeader` (chart-viewer.js, pure, headless-guarded)
|
||||
|
||||
```js
|
||||
// {kind,name,commit,window:[from,to],broker,seed,taps,members,params} -> array of
|
||||
// {label, value} chips. PURE (no DOM), the unit the headless guard drives; mount()
|
||||
// renders the chips into the header's #ctx slot. Empty meta -> [].
|
||||
function buildHeader(meta) {
|
||||
if (!meta) return [];
|
||||
var fmtDay = function (ns) { return new Date(ns / 1e6).toISOString().slice(0, 10); };
|
||||
var chips = [];
|
||||
chips.push({ label: meta.kind === "family" ? "family" : "run", value: meta.name });
|
||||
if (meta.members != null) chips.push({ label: "members", value: String(meta.members) });
|
||||
if (meta.taps && meta.taps.length) chips.push({ label: "tap", value: meta.taps.join(", ") });
|
||||
if (meta.broker) chips.push({ label: "broker", value: meta.broker });
|
||||
if (meta.window) chips.push({ label: "window", value: fmtDay(meta.window[0]) + "→" + fmtDay(meta.window[1]) });
|
||||
if (meta.seed) chips.push({ label: "seed", value: String(meta.seed) });
|
||||
if (meta.commit) chips.push({ label: "commit", value: meta.commit.slice(0, 7) });
|
||||
if (meta.params && meta.params.length)
|
||||
chips.push({ label: "params", value: meta.params.map(function (p) { return p[0] + "=" + p[1]; }).join(" ") });
|
||||
return chips;
|
||||
}
|
||||
// exported beside buildCharts/keyXRange; mount() reads window.AURA_TRACES.meta and
|
||||
// fills <span id="ctx"> with the chips.
|
||||
```
|
||||
|
||||
`CHART_HEAD` (render.rs:65-72) gains a `<span id="ctx" class="sub"></span>` slot
|
||||
between the `aura chart` label and the `#xmode-toggle` button (the toggle keeps
|
||||
`margin-left:auto`, so it stays right).
|
||||
|
||||
## Components
|
||||
|
||||
| Component | File | Change |
|
||||
|---|---|---|
|
||||
| `ChartMeta` | `crates/aura-cli/src/render.rs` | NEW serializable run-context struct |
|
||||
| `ChartData` | `crates/aura-cli/src/render.rs` | + `meta: ChartMeta` field; doc reworded |
|
||||
| `render_chart_html` | `crates/aura-cli/src/render.rs` | inject `"meta"` into `window.AURA_TRACES` |
|
||||
| `CHART_HEAD` | `crates/aura-cli/src/render.rs` | + `#ctx` header slot |
|
||||
| `decimate` + `CHART_DECIMATE_BUCKETS` | `crates/aura-cli/src/main.rs` | NEW pure transform + default |
|
||||
| `build_chart_data` | `crates/aura-cli/src/main.rs` | + `name` param; build `meta` from manifest |
|
||||
| `filter_to_tap` | `crates/aura-cli/src/main.rs` | narrow `meta.taps` to the picked tap |
|
||||
| `build_comparison_chart_data` | `crates/aura-cli/src/main.rs` | + `name` param; build family `meta` |
|
||||
| `emit_chart` | `crates/aura-cli/src/main.rs` | apply `decimate` on both arms |
|
||||
| `buildHeader` + `mount` | `crates/aura-cli/assets/chart-viewer.js` | NEW pure header builder; render into `#ctx` |
|
||||
|
||||
## Data flow
|
||||
|
||||
1. `emit_chart(name, tap, mode)` classifies the name (`name_kind`) — unchanged.
|
||||
2. **Run:** `read(name)` → `build_chart_data(name, traces)` builds `xs`/`series`
|
||||
(union-join, unchanged) and `meta` from `traces.manifest`; optional
|
||||
`filter_to_tap` narrows series + `meta.taps`.
|
||||
**Family:** `read_family(name)` → `build_comparison_chart_data(name, members,
|
||||
tap)` builds one series per member on a shared y-scale (unchanged) and `meta`
|
||||
from the shared manifest + member count.
|
||||
3. `decimate(data, CHART_DECIMATE_BUCKETS)` thins `xs`+`series` (meta passes
|
||||
through). No-op when already within budget (every existing small fixture).
|
||||
4. `render_chart_html` injects `{mode, xs, series, meta}` as `window.AURA_TRACES`.
|
||||
5. In the browser, `mount()` calls `buildCharts` (unchanged) and `buildHeader`,
|
||||
filling `#ctx` with the run-context chips.
|
||||
|
||||
## Error handling
|
||||
|
||||
- `decimate` is total: any `ChartData` in, a valid `ChartData` out; empty `xs` →
|
||||
unchanged; `buckets == 0` is never passed (the const is non-zero), but a
|
||||
defensive `buckets.max(1)` keeps it total.
|
||||
- No new exit paths. The existing refuse-don't-guess errors (`NotFound`, no-such-tap
|
||||
→ stderr + exit 2) are unchanged.
|
||||
- `buildHeader(undefined)` → `[]` (a page built by an older CLI without `meta`
|
||||
still renders its charts; the header is simply empty).
|
||||
|
||||
## Testing strategy
|
||||
|
||||
- **`decimate` unit tests** (`crates/aura-cli`): (a) **bounds** — a 10 000-point
|
||||
`ChartData` decimated to 2000 buckets yields `xs.len() <= 4000`; (b) **min/max
|
||||
preserved** — a series with a single tall spike inside a bucket keeps that spike
|
||||
value in the output (max survives); (c) **nulls preserved** — an all-null bucket
|
||||
emits null; (d) **no-op under budget** — `xs.len() <= 2*buckets` returns the data
|
||||
unchanged; (e) **meta passthrough** — `meta` is identical in/out; (f)
|
||||
**monotonic** — output `xs` strictly increasing.
|
||||
- **`render.rs` render tests:** **flip** `render_chart_html_injects_only_what_the_viewer_reads`
|
||||
(asserts `!contains("manifest")` / `!contains("taps")`) → a positive
|
||||
`render_chart_html_injects_run_context` asserting the page carries `"meta"`, the
|
||||
commit, the broker, and `"kind":"run"`. Update `sample_chart_data()` to construct
|
||||
a `ChartMeta`. Keep `…is_self_contained_and_injects_traces`,
|
||||
`…wears_its_own_chart_label`, `…reflects_panels_mode`.
|
||||
- **`buildHeader` headless guard:** new `tests/chart_viewer_header.mjs` (loads the
|
||||
real `chart-viewer.js`, drives `buildHeader`) + `tests/chart_viewer_header.rs`
|
||||
(shells out to `node`, mirroring `chart_viewer.rs`). Asserts: a run meta yields
|
||||
chips for name/tap/broker/window/commit; a family meta adds a `members` chip;
|
||||
`buildHeader(undefined)` → `[]`.
|
||||
- **Existing E2E** (`tests/cli_run.rs` family + single chart tests) stays green:
|
||||
the fixtures are small, so `decimate` is a no-op, and `meta` is additive. If any
|
||||
asserts exact injected bytes, extend it to tolerate the new `meta` key.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
Against aura's feature-acceptance criterion (`CLAUDE.md`):
|
||||
|
||||
- **The audience naturally reaches for it.** A researcher charting a multi-year
|
||||
real-data family today gets a browser-unopenable 148 MB page; after this cut the
|
||||
page opens and tells them which run/window/broker they are looking at — the exact
|
||||
friction surfaced when probing the #107 view on real data.
|
||||
- **Measurably improves correctness / removes a failure.** The page size decouples
|
||||
from the point count (≤ ~4000 spine slots/series vs millions); the WFO null-fill
|
||||
page collapses; the dropped run-context is restored. The min/max envelope is
|
||||
preserved (no spike lost).
|
||||
- **Reintroduces no failure class.** Engine + registry untouched (C14: the view
|
||||
layer owns pixels, not the engine). `decimate` is a pure deterministic transform
|
||||
(C1). Full recorded data stays on disk unchanged — decimation is view-only, never
|
||||
a data mutation. No look-ahead / merge / streaming invariant is in scope (view
|
||||
layer).
|
||||
- **Concrete evidence:** the worked `aura chart ger40-5y` program above (page-size
|
||||
before/after + the header line) is the empirical check.
|
||||
```
|
||||
Reference in New Issue
Block a user