load_m1_window double-materializes (AoS buffer -> SoA columns) with an un-presized growing Vec #78

New Issue

2026-06-17T09:53:09+02:00

Brummel commented

2026-06-17 09:53:09 +02:00

load_m1_window double-materializes (AoS buffer → SoA columns) with an un-presized growing Vec

Found during a code-level performance audit (load/ingest edge, not the per-cycle hot path).

Problem

crates/aura-ingest/src/lib.rs:118-131:

let mut bars: Vec<M1Parsed> = Vec::new();          // no with_capacity → realloc growth
while let Some(chunk) = it.next_chunk() { bars.extend_from_slice(&chunk); }
Some(transpose_m1(&bars))                            // second full materialization

Chunks → AoS buffer (full copy, growing by reallocation) → SoA columns (second full
copy). That is ~2× peak memory + realloc churn at load time. One-time per window
(or per sweep point if reloaded), so not on the per-cycle hot path — but it is a
clean, risk-free memory win. The lazy M1FieldSource (lib.rs:164-244) already
handles the streaming case well (O(one chunk) resident).

Fix

Transpose directly from the chunk iterator into the SoA columns, dropping the
intermediate Vec<M1Parsed> entirely (one fewer full materialization, halved peak).
Where the iterator can report a size hint/count, pre-size the columns.

This is behaviour-preserving: the resulting M1Columns is byte-identical to the
current path (same pushes, same order); transpose_m1 stays as-is for the existing
unit tests.

Acceptance

load_m1_window no longer allocates the intermediate AoS Vec.
Output M1Columns identical to today on the existing fixtures.
cargo test --workspace green.

`load_m1_window` double-materializes (AoS buffer → SoA columns) with an un-presized growing `Vec` Found during a code-level performance audit (load/ingest edge, not the per-cycle hot path). ## Problem `crates/aura-ingest/src/lib.rs:118-131`: ```rust let mut bars: Vec<M1Parsed> = Vec::new(); // no with_capacity → realloc growth while let Some(chunk) = it.next_chunk() { bars.extend_from_slice(&chunk); } Some(transpose_m1(&bars)) // second full materialization ``` Chunks → AoS buffer (full copy, growing by reallocation) → SoA columns (second full copy). That is ~2× peak memory + realloc churn at load time. One-time per window (or per sweep point if reloaded), so not on the per-cycle hot path — but it is a clean, risk-free memory win. The lazy `M1FieldSource` (`lib.rs:164-244`) already handles the streaming case well (O(one chunk) resident). ## Fix Transpose directly from the chunk iterator into the SoA columns, dropping the intermediate `Vec<M1Parsed>` entirely (one fewer full materialization, halved peak). Where the iterator can report a size hint/count, pre-size the columns. This is **behaviour-preserving**: the resulting `M1Columns` is byte-identical to the current path (same pushes, same order); `transpose_m1` stays as-is for the existing unit tests. ## Acceptance - `load_m1_window` no longer allocates the intermediate AoS `Vec`. - Output `M1Columns` identical to today on the existing fixtures. - `cargo test --workspace` green.

Brummel referenced this issue from a commit

2026-06-17 10:19:30 +02:00

refactor(aura-ingest): chunk-direct M1 transpose, drop the AoS intermediate

Brummel closed this issue

2026-06-17 10:19:30 +02:00

Brummel referenced this issue

2026-06-17 10:49:27 +02:00

Rename Recorder -> Probe (the edge tap); decide whether a Probe must heap-allocate a record per cycle #77

Sign in to join this conversation.