Aura/docs/specs/fieldtest-0011-ingest.md

Fieldtest — cycle-0011 — 2026-06-04

Status: Draft — awaiting orchestrator triage Author: fieldtester (dispatched by fieldtest skill)

Scope

Cycle 0011 (Walking-skeleton milestone, #7) shipped a NEW public crate aura-ingest — aura's first real data-source ingestion boundary and the workspace's external-dependency firewall (it alone links data-server and its transitive chrono/regex/zip). Public surface field-tested:

1. unix_ms_to_epoch_ns(time_ms: i64) -> Timestamp — the single C3 unit normalization (= ms * 1_000_000), at this one boundary and nowhere else.
2. M1Columns — the OHLCV bar transposed AoS → SoA (C7): public fields ts: Vec<Timestamp>, open/high/low/close/spread: Vec<f64>, volume: Vec<i64>; derives Clone + Debug + PartialEq.
3. transpose_m1(&[M1Parsed]) -> M1Columns — pure AoS → SoA, normalizing time at the boundary.
4. M1Columns::close_stream() -> Vec<(Timestamp, Scalar)> — the close column as the engine source-stream shape the SMA-cross sample strategy consumes.
5. load_m1_window(&Arc<DataServer>, symbol, from_ms, to_ms) -> Option<M1Columns> — drains a data-server M1 window and transposes once at the boundary.

Tested from the public interface only: cargo doc -p aura-ingest / cargo doc -p data-server rustdoc, the design ledger (C1/C3/C7/C10/C12/C16), the data-server README, and the observable behaviour of the functions — never crates/*/src. The build exercised: cargo build --workspace (green from HEAD), then each fixture via cargo run --manifest-path fieldtests/cycle-0011-ingest/Cargo.toml --bin <name> so HEAD source is always what runs. The fixture crate is a standalone non-member crate path-depending on aura-core/-engine/-std/-ingest plus the data-server git dep, exactly as a C16 research project that ingests recorded bars would.

Examples

fieldtests/cycle-0011-ingest/c0011_1_transpose_core.rs — the pure transpose/normalize core (axis a)

• Drives unix_ms_to_epoch_ns on representative values (0, 1 ms, a real 2006-era Unix-ms), then transpose_m1 on three hand-built M1Parsed bars with distinct per-field values (so a swapped column would be caught), checks each SoA column field-wise, index alignment (ts[i] ↔ close[i]), volume kept i64, purity (identical bars → equal columns via PartialEq), close_stream shape, and the empty-input edge.
• Why it fits: this is the C3/C7 boundary contract a project author transposing recorded data exercises, verified from rustdoc alone.
• Outcome: built and ran first try; every assertion held. ms*1e6 exact, field-wise transpose correct, volume i64, pure, close_stream = (normalized ts, close as Scalar::F64), empty → empty. Matched expected.

fieldtests/cycle-0011-ingest/c0011_2_close_stream_run.rs — close_stream → a real engine run (axis b)

• Transposes synthetic OHLCV bars (close rises then reverses), takes close_stream(), feeds it straight into a Harness (SMA(2)/SMA(4) → Sub → Exposure(4) → SimBroker(1.0) → two shipped aura_std::Recorder sinks), runs, drains, f64_field + summarize, asserts finite metrics, ≥1 exposure flip, that recorded equity timestamps equal the ingested normalized bar timestamps (epoch-ns end to end), and that a disjoint second run is bit-identical (C1).
• Why it fits: this is the C3 → C1 hand-off — the (Timestamp, Scalar) shape close_stream returns is exactly Harness::run's source-stream input shape; the ingestion boundary must connect to the engine with no hand-authored stream.
• Outcome: built and ran first try. equity [0,0,0,0,1,0,-0.5], exposure flips once at the reversal, total_pips -0.5, max_drawdown 1.5, recorded ts == ingested ts, two runs byte-identical. Matched expected.

fieldtests/cycle-0011-ingest/c0011_3_load_window_real.rs — load_m1_window against real data (axis c)

• Opens the local data-server archive (DEFAULT_DATA_PATH), loads the real AAPL.US 2006-08 M1 window, checks SoA column lengths aligned, every bar ts normalized to epoch-ns and inside the requested window, close_stream non-decreasing, then runs an end-to-end SMA-cross signal-quality backtest over the real close bars (deterministic across two runs, C1). Then exercises the documented None paths (far-future window, unknown symbol) and the in-coverage-but-empty sub-window grey zone. Skips cleanly where /mnt/tickdata is absent.
• Why it fits: the milestone's headline — "a backtest runs over real bars" — driven exactly as a downstream researcher would, plus the documented None path.
• Outcome: built and ran first try on the real archive. Loaded 21 bars for 2006-08; ts normalized and in-window; close_stream ascending; the end-to-end run was deterministic; far-future window → None, unknown symbol → None; an in-coverage empty sub-window → Some(empty). Surfaced two spec_gaps (below).

Findings

[working] transpose_m1 / unix_ms_to_epoch_ns / close_stream match the C3/C7 contract from rustdoc alone

• Example: c0011_1.
• What happened: unix_ms_to_epoch_ns is exactly ms * 1_000_000 (checked at 0, 1 ms, and a real 2006 ms = 1_154_390_400_000 → 1_154_390_400_000_000_000 ns); transpose_m1 produces field-wise-correct SoA columns with no column swap, volume kept i64 (C7's one i64 column), ts[i] aligned with close[i]; it is pure (identical bars → PartialEq-equal columns, C1); close_stream yields (normalized ts, Scalar::F64(close)) per bar; empty input → empty columns and empty stream.
• Why working: the entire pure boundary core is reachable, correct, and unambiguous from the public rustdoc + ledger (C3/C7/C1) without reading any implementation source. The public-fielded M1Columns + the M1Parsed public fields make a hand-built fixture trivial to author.
• Recommended action: carry-on.

[working] close_stream feeds Harness::run directly; epoch-ns flows end to end; deterministic

• Example: c0011_2.
• What happened: close_stream()'s Vec<(Timestamp, Scalar)> is exactly the shape Harness::run(vec![stream]) consumes — no adapter. Over a rise-then- reverse close series the chain warmed, the recorded equity-curve timestamps equalled the ingested (normalized epoch-ns) bar timestamps verbatim (normalization survives the whole run loop intact), one exposure sign flip was registered at the reversal, and a disjoint second run produced a bit-identical RunReport.to_json() (C1).
• Why working: the C3 (ingestion) → C1 (deterministic run) seam — the milestone's reason to exist — is reachable and correct from the public surface as a downstream consumer, mirroring the cycle-0007/0009 patterns over real ingested data rather than a hand-authored stream.
• Recommended action: carry-on.

[working] load_m1_window over real bars: in-window normalized, ascending, deterministic; far-future & unknown-symbol None paths hold

• Example: c0011_3.
• What happened: against the real archive, load_m1_window("AAPL.US", 2006-08 window) returned Some(M1Columns) with 21 bars; all SoA columns equal length, every ts normalized to epoch-ns and inside the requested (×1e6) window, close_stream non-decreasing in ts; the end-to-end backtest over the real closes ran and was bit-identical across two disjoint runs. A far-future window for the present symbol → None; an unknown symbol → None, both as the rustdoc documents.
• Why working: the real-data path — open archive, load a window, transpose once at the boundary, run a deterministic backtest — works first try from the public surface, and the two documented None conditions (no file overlaps the window; unknown symbol) behave as stated.
• Recommended action: carry-on.

[spec_gap] Some(empty) vs None: an in-coverage but bar-empty window returns Some(empty M1Columns), not None

• Example: c0011_3 (grey-zone block).
• What happened: the load_m1_window rustdoc states "None if the symbol has no data in [from_ms, to_ms]". Empirically, None is file-level: it occurs when no data-server file overlaps the window (e.g. a far-future window — those files are skipped without I/O) or the symbol is unknown. But a narrow window inside a loaded file's date coverage that happens to contain zero bars (a single 2006-08-01 day inside the loaded 2006-08 file) returns Some(M1Columns { ... empty vecs ... }), not None. So a downstream consumer pattern-matching None to mean "no bars here" is wrong for the in-coverage-empty case; it must additionally check cols.close.is_empty().
• Why spec_gap: "no data in [from_ms, to_ms]" reads bar-level on the public surface, but the realized boundary is file-level (it inherits data-server's own file-skip None). Two readings are equally plausible — "None ⇔ zero bars" (what the wording suggests) vs "None ⇔ no overlapping file, else Some (possibly empty)" (what ships). I asserted the observed (file-level) reading in the fixture; a tightening to the bar-level reading would flip that assertion.
• Recommended action: ratify-or-tighten the ledger/rustdoc. Either document precisely ("None only when no file overlaps the window or the symbol is unknown; an in-coverage window with no bars returns Some with empty columns — check .close.is_empty()"), or collapse the empty case into None so the binary Some/None cleanly means "has bars" / "has none". One rustdoc line on load_m1_window closes it.

[spec_gap] the only documented real symbol is too sparse for a meaningful signal-quality run over a realistic window

• Example: c0011_3.
• What happened: AAPL.US — the symbol named in the carrier and the gated integration test — has 21 M1 bars for all of August 2006 and ~100 for the whole year 2006 (verified via load_m1_window over a year/month/day window). The end-to-end backtest therefore ran but produced all-zero metrics (total_pips 0, max_drawdown 0, exposure_sign_flips 0): with any realistic SMA-cross (even SMA(10)/SMA(30)) the slow SMA never warms over ≤21 bars, so the signal never fires (correct C2 warm-up / no-look-ahead behaviour — not a bug). The boundary itself is faultless; the data behind the milestone's shipped example is too thin to demonstrate a non-degenerate run.
• Why spec_gap: the milestone promise is "a backtest runs over real bars" — which is technically met — but a downstream consumer following the documented surface (this symbol, this archive) cannot reach a meaningful signal-quality result, and nothing on the public surface signals that 21 bars/month is the expected density (vs a window-bounds bug, a corrupt file, or a normalization error). The consumer guesses "this is just sparse early data"; "the window is mis-bounded" is an equally plausible reading from the surface alone. I picked the sparse-data reading (the bounds were honoured, ts are in-window, the year window also returns only ~100 bars).
• Recommended action: ratify (document expected per-symbol bar density, or name a denser default symbol/window for the milestone demo) or plan a follow-up that points the walking-skeleton sample at a real window dense enough to warm a realistic signal — so "runs over real bars" demonstrably yields a non-degenerate trace, consistent with C22's "newcomer sees a populated trace". No code change to aura-ingest implied; this is about which data the milestone showcases.

Recommendation summary

Finding	Class	Action
transpose/normalize/close_stream match C3/C7 from rustdoc	working	carry-on
close_stream → deterministic engine run, epoch-ns end to end	working	carry-on
load_m1_window real bars + documented None paths	working	carry-on
Some(empty) vs None boundary not crisp	spec_gap	ratify / tighten the ledger + rustdoc
AAPL.US too sparse for a non-degenerate milestone run	spec_gap	ratify / plan a denser demo symbol+window