Brummel/Aura
1
0
Fork 0
Code Issues 31 Pull Requests Actions Packages Projects Releases Wiki Activity

Families-comparison chart: point decimation/LOD for multi-year data (real 5y sweep = 148MB, 2y WFO = 76MB) #108

New Issue
Closed
opened 2026-06-22 10:32:54 +02:00 by Brummel · 2 comments
Brummel commented 2026-06-22 10:32:54 +02:00
Owner
Copy Link
Copy Source

Families-comparison chart has no point decimation — multi-year real-data pages reach 100s of MB

The first-cut families-comparison view (#107) inlines every point of every
member into one self-contained HTML page (the #101 "no mipmapping/LOD" non-goal
carried over). Functionally it works at any size — aura chart <family> exits 0
and renders all members — but page size and load cost grow linearly with the point
count, so a multi-year M1 family is impractical-to-unopenable in a browser.

Measured on REAL GER40 M1 data (built-in --real demo strategy, M1 cadence,
~324k points/year/member):

run window members gen time chart page
sweep --real GER40 1y (2023) 4 3 s 29 MB
sweep --real GER40 5y (2019–23) 4 (1.62M pts ea.) 12 s 148 MB
walkforward --real GER40 2y (2022–23) 21 OOS 2 s 76 MB
walkforward --real GER40 5y (extrapolated) ~57 OOS — ~400 MB+

The runs are fine — the engine streams (O(one chunk) resident, C12) and a 5y M1
sweep finishes in 12 s. The cliff is the view: the serializer materializes raw
points with no thinning.

Two amplifiers:

  1. Real floats are long. Full-precision values (215.9676784206947) make real
    pages bigger than rounded synthetic ones — the 5y sweep is 148 MB for 1.62M×4
    points (~23 bytes/point), worse than a rounded estimate.
  2. Walk-forward is the worst case (null-fill sparse matrix). OOS members are
    disjoint windows, so the union-ts spine is the full span and each of the N
    series is null outside its own segment: the 2y WFO page is 76 MB for only
    35 MB of on-disk data     — N series × union-length, mostly nulls. At 5y / ~57
    windows this dominates.

A resampling strategy (e.g. the GER40 breakout → 15m) stays ~7 MB over 5 years;
the heavy case is an M1-cadence strategy (the built-in --real demo fires per
M1 bar).

Direction (not prescriptive): serve-time point decimation / LOD — min-max buckets
per pixel column is the standard for equity curves (a chart never needs more points
than horizontal pixels; uPlot draws the decimated series, full data stays on disk).
For the walk-forward case this also collapses the null-fill blowup, since each
member is decimated to ~viewport width before union. A streaming/range-query store
(Arrow/Parquet — the #101 "scale path") is the heavier alternative.

context: surfaced answering "do sweep / MC / WFO families chart over multiple years
of real data?" MC over real data is separately unsupported (no aura mc --real,
#106 Fork A — it exits 2). Real sweep/WFO runs work; only the view needs this.
depends on: nothing; the data path + the view are present.

## Families-comparison chart has no point decimation — multi-year real-data pages reach 100s of MB The first-cut families-comparison view (#107) inlines **every** point of every member into one self-contained HTML page (the #101 "no mipmapping/LOD" non-goal carried over). Functionally it works at any size — `aura chart <family>` exits 0 and renders all members — but page size and load cost grow linearly with the point count, so a multi-year M1 family is impractical-to-unopenable in a browser. **Measured on REAL GER40 M1 data** (built-in `--real` demo strategy, M1 cadence, ~324k points/year/member): | run | window | members | gen time | chart page | |---|---|---|---|---| | `sweep --real GER40` | 1y (2023) | 4 | 3 s | **29 MB** | | `sweep --real GER40` | 5y (2019–23) | 4 (1.62M pts ea.) | 12 s | **148 MB** | | `walkforward --real GER40` | 2y (2022–23) | 21 OOS | 2 s | **76 MB** | | `walkforward --real GER40` | 5y (extrapolated) | ~57 OOS | — | **~400 MB+** | The **runs** are fine — the engine streams (O(one chunk) resident, C12) and a 5y M1 sweep finishes in 12 s. The cliff is the **view**: the serializer materializes raw points with no thinning. Two amplifiers: 1. **Real floats are long.** Full-precision values (`215.9676784206947`) make real pages bigger than rounded synthetic ones — the 5y sweep is 148 MB for 1.62M×4 points (~23 bytes/point), worse than a rounded estimate. 2. **Walk-forward is the worst case (null-fill sparse matrix).** OOS members are *disjoint* windows, so the union-ts spine is the full span and each of the N series is `null` outside its own segment: the 2y WFO page is **76 MB for only 35 MB of on-disk data** — N series × union-length, mostly nulls. At 5y / ~57 windows this dominates. A **resampling** strategy (e.g. the GER40 breakout → 15m) stays ~7 MB over 5 years; the heavy case is an **M1-cadence** strategy (the built-in `--real` demo fires per M1 bar). Direction (not prescriptive): serve-time point decimation / LOD — min-max buckets per pixel column is the standard for equity curves (a chart never needs more points than horizontal pixels; uPlot draws the decimated series, full data stays on disk). For the walk-forward case this also collapses the null-fill blowup, since each member is decimated to ~viewport width before union. A streaming/range-query store (Arrow/Parquet — the #101 "scale path") is the heavier alternative. context: surfaced answering "do sweep / MC / WFO families chart over multiple years of real data?" MC over real data is separately unsupported (no `aura mc --real`, #106 Fork A — it exits 2). Real sweep/WFO runs work; only the view needs this. depends on: nothing; the data path + the view are present.
Brummel added the idea label 2026-06-22 10:32:54 +02:00
Brummel commented 2026-06-22 11:25:05 +02:00
Author
Owner
Copy Link
Copy Source

Reference issue for a /boss cycle — "visual World cut 2" (served-page hardening).

This cycle bundles #108 (this issue, decimation/LOD) with #102 (run-context
header) into one served-page-hardening cut, and closes both. Rationale (from a
tracker-grounding sweep): the two share one code locus — render_chart_html /
ChartData / chart-viewer.js / build_comparison_chart_data in aura-cli —
and one purpose under C22/C14: turn the #107 families-comparison first cut into a
chart page a (usually remote) user can actually open and read on REAL multi-year
families. #108 is the load-bearing anchor (measured 148 MB / 5y M1 sweep, 76 MB /
2y WFO = functionally unopenable); #102 rides cheaply on the same page edit (the
manifest is already on disk, just re-inject + render a header).

Scope:

  • #108 — serve-time min-max-per-pixel-column decimation in the chart render
    path, applied per series before the union-ts join (so the WFO null-fill
    blow-up collapses too); full data stays on disk; engine/registry untouched
    (C14). Out of scope: the Arrow/Parquet streaming store (the deferred "scale
    path").
  • #102 — thread RunManifest (+ the tap list) back into ChartData and the
    injected window.AURA_TRACES, render a run-context header in chart-viewer.js
    (single run: commit/params/window/seed/broker; family: shared window/broker +
    per-member identity). Flip the negative-assertion test that currently pins the
    manifest's absence from the page to a positive one.

Excluded from this cut by substance, not effort: #47 (graph-render coverage
gap, different surface, no live consumer — stays a standalone quick-win) and
#72 (web-shell, design-fork behind #109).

**Reference issue for a `/boss` cycle — "visual World cut 2" (served-page hardening).** This cycle bundles **#108 (this issue, decimation/LOD)** with **#102 (run-context header)** into one served-page-hardening cut, and closes both. Rationale (from a tracker-grounding sweep): the two share one code locus — `render_chart_html` / `ChartData` / `chart-viewer.js` / `build_comparison_chart_data` in `aura-cli` — and one purpose under C22/C14: turn the #107 families-comparison first cut into a chart page a (usually remote) user can actually open and read on REAL multi-year families. #108 is the load-bearing anchor (measured 148 MB / 5y M1 sweep, 76 MB / 2y WFO = functionally unopenable); #102 rides cheaply on the same page edit (the manifest is already on disk, just re-inject + render a header). Scope: - **#108** — serve-time min-max-per-pixel-column decimation in the chart render path, applied per series *before* the union-ts join (so the WFO null-fill blow-up collapses too); full data stays on disk; engine/registry untouched (C14). Out of scope: the Arrow/Parquet streaming store (the deferred "scale path"). - **#102** — thread `RunManifest` (+ the tap list) back into `ChartData` and the injected `window.AURA_TRACES`, render a run-context header in `chart-viewer.js` (single run: commit/params/window/seed/broker; family: shared window/broker + per-member identity). Flip the negative-assertion test that currently pins the manifest's *absence* from the page to a positive one. Excluded from this cut by substance, not effort: **#47** (graph-render coverage gap, different surface, no live consumer — stays a standalone quick-win) and **#72** (web-shell, design-fork behind #109).
Brummel commented 2026-06-22 11:30:22 +02:00
Author
Owner
Copy Link
Copy Source

Design reconciliation (specify) — spec 0062 visual-world-cut-2

The forks below are derived orchestrator decisions (rationale, not user
provenance), recorded before the spec is written. Running under /boss.

  • Fork: decimation locus — before-join vs after-join. → after-join.
    Basis (derived): decimate the already-aligned ChartData (shared xs +
    per-series Option<f64>) as a pure transform, not the pre-join member rows.
    It is the smallest blast radius (one pure decimate(ChartData, n) -> ChartData
    in the render path, the join untouched), and it fixes exactly the issue's
    stated cliff — page size ("the cliff is the view"). For a family the output
    is bounded to N×O(buckets) for sweep/MC/WFO alike (a null-only bucket stays
    null, so the WFO null-fill collapses on the page). Transient generation memory
    (the full join before decimation) is unchanged — the same path that already
    generates the 2y-WFO / 5y-sweep pages today — and is out of scope (the runs
    are "fine" per the issue; only the page is the problem).

  • Fork: algorithm — min-max-per-bucket vs plain downsample. → min-max.
    Basis (derived): the issue names min-max-per-pixel-column as the equity-curve
    standard; it preserves spikes a plain stride/last-in-bucket would drop. Each
    bucket contributes a series' min and max sample (placed at the bucket's two
    boundary timestamps on the shared spine — sub-pixel ordering, so the drawn
    vertical extent per column is preserved). A bucket with a single sample emits
    one point; an all-null bucket emits null.

  • Fork: target width — CLI flag vs fixed const. → fixed const (~2000
    buckets ⇒ ≤~4000 spine points). Basis (derived): the issue asks for decimation,
    not a knob; a fixed viewport-scale default keeps the cut tight (YAGNI). A
    --width/--max-points flag is a clean future add, noted, not built.

  • Fork: header render — server-side (Rust) vs client-side (JS). →
    client-side, a pure buildHeader(meta) in chart-viewer.js mirroring
    buildCharts. Basis (derived): the issue states "render a small run-context
    header in chart-viewer.js" and "a node .mjs guard for the header-build is cheap
    (pure, like buildCharts)"; this matches the existing pure-function + headless
    .mjs guard architecture.

  • Fork: the "instrument" field. → render what the manifest has; do not
    invent an instrument field.     Basis (derived): RunManifest (report.rs:36-50)
    carries commit/params/window/seed/broker only — no instrument. The broker label
    carries pip_size, not the symbol, and the symbol is not persisted. The header
    shows commit (short) · window · seed · broker · params (+ the charted taps); it
    does not claim an instrument it cannot ground.

  • Fork: family header content. → shared context once + per-member identity
    via the series label.     Basis (derived): each FamilyMember has its own
    manifest, but commit/window/broker are shared across a family; params differ per
    member and are already the series label (member.key). So the family header
    shows kind=family · name · member count · tap · shared commit/window/broker; it
    does not repeat per-member params (they are the legend).

Out of scope (substance, not effort): #47 (graph-render coverage, a different
surface) and #72 (web-shell, design-fork behind #109).

## Design reconciliation (specify) — spec 0062 visual-world-cut-2 The forks below are derived orchestrator decisions (rationale, not user provenance), recorded before the spec is written. Running under `/boss`. - **Fork: decimation locus — before-join vs after-join.** → **after-join**. Basis (derived): decimate the already-aligned `ChartData` (shared `xs` + per-series `Option<f64>`) as a pure transform, not the pre-join member rows. It is the smallest blast radius (one pure `decimate(ChartData, n) -> ChartData` in the render path, the join untouched), and it fixes exactly the issue's stated cliff — *page size* ("the cliff is the view"). For a family the output is bounded to N×O(buckets) for sweep/MC/WFO alike (a null-only bucket stays null, so the WFO null-fill collapses on the page). Transient generation memory (the full join before decimation) is unchanged — the same path that already generates the 2y-WFO / 5y-sweep pages today — and is out of scope (the runs are "fine" per the issue; only the page is the problem). - **Fork: algorithm — min-max-per-bucket vs plain downsample.** → **min-max**. Basis (derived): the issue names min-max-per-pixel-column as the equity-curve standard; it preserves spikes a plain stride/last-in-bucket would drop. Each bucket contributes a series' min and max sample (placed at the bucket's two boundary timestamps on the shared spine — sub-pixel ordering, so the drawn vertical extent per column is preserved). A bucket with a single sample emits one point; an all-null bucket emits null. - **Fork: target width — CLI flag vs fixed const.** → **fixed const** (~2000 buckets ⇒ ≤~4000 spine points). Basis (derived): the issue asks for decimation, not a knob; a fixed viewport-scale default keeps the cut tight (YAGNI). A `--width`/`--max-points` flag is a clean future add, noted, not built. - **Fork: header render — server-side (Rust) vs client-side (JS).** → **client-side**, a pure `buildHeader(meta)` in `chart-viewer.js` mirroring `buildCharts`. Basis (derived): the issue states "render a small run-context header in chart-viewer.js" and "a node .mjs guard for the header-build is cheap (pure, like buildCharts)"; this matches the existing pure-function + headless `.mjs` guard architecture. - **Fork: the "instrument" field.** → **render what the manifest has; do not invent an instrument field.** Basis (derived): `RunManifest` (report.rs:36-50) carries commit/params/window/seed/broker only — no instrument. The broker label carries `pip_size`, not the symbol, and the symbol is not persisted. The header shows commit (short) · window · seed · broker · params (+ the charted taps); it does not claim an instrument it cannot ground. - **Fork: family header content.** → **shared context once + per-member identity via the series label.** Basis (derived): each `FamilyMember` has its own manifest, but commit/window/broker are shared across a family; params differ per member and are already the series label (`member.key`). So the family header shows kind=family · name · member count · tap · shared commit/window/broker; it does not repeat per-member params (they are the legend). Out of scope (substance, not effort): #47 (graph-render coverage, a different surface) and #72 (web-shell, design-fork behind #109).
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
BLOCKER
Core functionality blocked; preempts other selection
 bug
Observable misbehaviour
 feature
Substantial extension; no full spec required
 idea
Droppable; no commitment
No Label idea
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
claude (Claude) Brummel
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Brummel/Aura#108
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?