Aura/docs/specs/fieldtest-0011-ingest.md

# Fieldtest — cycle-0011 — 2026-06-04

**Status:** Draft — awaiting orchestrator triage
**Author:** fieldtester (dispatched by fieldtest skill)

## Scope

Cycle 0011 (Walking-skeleton milestone, #7) shipped a NEW public crate
`aura-ingest` — aura's first real data-source ingestion boundary and the
workspace's external-dependency firewall (it alone links `data-server` and its
transitive `chrono`/`regex`/`zip`). Public surface field-tested:

1. `unix_ms_to_epoch_ns(time_ms: i64) -> Timestamp` — the single C3 unit
   normalization (`= ms * 1_000_000`), at this one boundary and nowhere else.
2. `M1Columns` — the OHLCV bar transposed AoS → SoA (C7): public fields
   `ts: Vec<Timestamp>`, `open/high/low/close/spread: Vec<f64>`,
   `volume: Vec<i64>`; derives Clone + Debug + PartialEq.
3. `transpose_m1(&[M1Parsed]) -> M1Columns` — pure AoS → SoA, normalizing time
   at the boundary.
4. `M1Columns::close_stream() -> Vec<(Timestamp, Scalar)>` — the close column as
   the engine source-stream shape the SMA-cross sample strategy consumes.
5. `load_m1_window(&Arc<DataServer>, symbol, from_ms, to_ms) -> Option<M1Columns>`
   — drains a data-server M1 window and transposes once at the boundary.

Tested from the public interface only: `cargo doc -p aura-ingest` /
`cargo doc -p data-server` rustdoc, the design ledger (C1/C3/C7/C10/C12/C16),
the `data-server` README, and the observable behaviour of the functions —
never `crates/*/src`. The build exercised: `cargo build --workspace` (green from
HEAD), then each fixture via
`cargo run --manifest-path fieldtests/cycle-0011-ingest/Cargo.toml --bin <name>`
so HEAD source is always what runs. The fixture crate is a standalone non-member
crate path-depending on `aura-core/-engine/-std/-ingest` plus the `data-server`
git dep, exactly as a C16 research project that ingests recorded bars would.

## Examples

### fieldtests/cycle-0011-ingest/c0011_1_transpose_core.rs — the pure transpose/normalize core (axis a)
- Drives `unix_ms_to_epoch_ns` on representative values (0, 1 ms, a real 2006-era
  Unix-ms), then `transpose_m1` on three hand-built `M1Parsed` bars with distinct
  per-field values (so a swapped column would be caught), checks each SoA column
  field-wise, index alignment (`ts[i]` ↔ `close[i]`), `volume` kept i64, purity
  (identical bars → equal columns via PartialEq), `close_stream` shape, and the
  empty-input edge.
- Why it fits: this is the C3/C7 boundary contract a project author transposing
  recorded data exercises, verified from rustdoc alone.
- Outcome: built and ran first try; every assertion held. `ms*1e6` exact,
  field-wise transpose correct, volume i64, pure, `close_stream` =
  `(normalized ts, close as Scalar::F64)`, empty → empty. Matched expected.

### fieldtests/cycle-0011-ingest/c0011_2_close_stream_run.rs — close_stream → a real engine run (axis b)
- Transposes synthetic OHLCV bars (close rises then reverses), takes
  `close_stream()`, feeds it straight into a Harness (SMA(2)/SMA(4) → Sub →
  Exposure(4) → SimBroker(1.0) → two shipped `aura_std::Recorder` sinks), runs,
  drains, `f64_field` + `summarize`, asserts finite metrics, ≥1 exposure flip,
  that recorded equity timestamps equal the ingested normalized bar timestamps
  (epoch-ns end to end), and that a disjoint second run is bit-identical (C1).
- Why it fits: this is the C3 → C1 hand-off — the `(Timestamp, Scalar)` shape
  `close_stream` returns is exactly `Harness::run`'s source-stream input shape;
  the ingestion boundary must connect to the engine with no hand-authored stream.
- Outcome: built and ran first try. equity `[0,0,0,0,1,0,-0.5]`, exposure flips
  once at the reversal, `total_pips -0.5`, `max_drawdown 1.5`, recorded ts ==
  ingested ts, two runs byte-identical. Matched expected.

### fieldtests/cycle-0011-ingest/c0011_3_load_window_real.rs — load_m1_window against real data (axis c)
- Opens the local `data-server` archive (`DEFAULT_DATA_PATH`), loads the real
  AAPL.US 2006-08 M1 window, checks SoA column lengths aligned, every bar ts
  normalized to epoch-ns and inside the requested window, `close_stream`
  non-decreasing, then runs an end-to-end SMA-cross signal-quality backtest over
  the real close bars (deterministic across two runs, C1). Then exercises the
  documented `None` paths (far-future window, unknown symbol) and the
  in-coverage-but-empty sub-window grey zone. Skips cleanly where
  `/mnt/tickdata` is absent.
- Why it fits: the milestone's headline — "a backtest runs over real bars" —
  driven exactly as a downstream researcher would, plus the documented `None`
  path.
- Outcome: built and ran first try on the real archive. Loaded **21** bars for
  2006-08; ts normalized and in-window; `close_stream` ascending; the end-to-end
  run was deterministic; far-future window → `None`, unknown symbol → `None`;
  an in-coverage empty sub-window → `Some(empty)`. Surfaced two spec_gaps (below).

## Findings

### [working] transpose_m1 / unix_ms_to_epoch_ns / close_stream match the C3/C7 contract from rustdoc alone
- Example: c0011_1.
- What happened: `unix_ms_to_epoch_ns` is exactly `ms * 1_000_000` (checked at 0,
  1 ms, and a real 2006 ms = 1_154_390_400_000 → 1_154_390_400_000_000_000 ns);
  `transpose_m1` produces field-wise-correct SoA columns with no column swap,
  `volume` kept `i64` (C7's one i64 column), `ts[i]` aligned with `close[i]`;
  it is pure (identical bars → PartialEq-equal columns, C1); `close_stream`
  yields `(normalized ts, Scalar::F64(close))` per bar; empty input → empty
  columns and empty stream.
- Why working: the entire pure boundary core is reachable, correct, and
  unambiguous from the public rustdoc + ledger (C3/C7/C1) without reading any
  implementation source. The public-fielded `M1Columns` + the `M1Parsed` public
  fields make a hand-built fixture trivial to author.
- Recommended action: carry-on.

### [working] close_stream feeds Harness::run directly; epoch-ns flows end to end; deterministic
- Example: c0011_2.
- What happened: `close_stream()`'s `Vec<(Timestamp, Scalar)>` is exactly the
  shape `Harness::run(vec![stream])` consumes — no adapter. Over a rise-then-
  reverse close series the chain warmed, the recorded equity-curve timestamps
  equalled the ingested (normalized epoch-ns) bar timestamps verbatim
  (normalization survives the whole run loop intact), one exposure sign flip was
  registered at the reversal, and a disjoint second run produced a bit-identical
  `RunReport.to_json()` (C1).
- Why working: the C3 (ingestion) → C1 (deterministic run) seam — the milestone's
  reason to exist — is reachable and correct from the public surface as a
  downstream consumer, mirroring the cycle-0007/0009 patterns over real ingested
  data rather than a hand-authored stream.
- Recommended action: carry-on.

### [working] load_m1_window over real bars: in-window normalized, ascending, deterministic; far-future & unknown-symbol None paths hold
- Example: c0011_3.
- What happened: against the real archive, `load_m1_window("AAPL.US", 2006-08
  window)` returned `Some(M1Columns)` with 21 bars; all SoA columns equal length,
  every ts normalized to epoch-ns and inside the requested (×1e6) window,
  `close_stream` non-decreasing in ts; the end-to-end backtest over the real
  closes ran and was bit-identical across two disjoint runs. A far-future window
  for the present symbol → `None`; an unknown symbol → `None`, both as the
  rustdoc documents.
- Why working: the real-data path — open archive, load a window, transpose once
  at the boundary, run a deterministic backtest — works first try from the public
  surface, and the two documented `None` conditions (no file overlaps the window;
  unknown symbol) behave as stated.
- Recommended action: carry-on.

### [spec_gap] `Some(empty)` vs `None`: an in-coverage but bar-empty window returns `Some(empty M1Columns)`, not `None`
- Example: c0011_3 (grey-zone block).
- What happened: the `load_m1_window` rustdoc states "`None` if the symbol has no
  data in `[from_ms, to_ms]`". Empirically, `None` is **file-level**: it occurs
  when no data-server file overlaps the window (e.g. a far-future window — those
  files are skipped without I/O) or the symbol is unknown. But a **narrow window
  inside a loaded file's date coverage that happens to contain zero bars**
  (a single 2006-08-01 day inside the loaded 2006-08 file) returns
  `Some(M1Columns { ... empty vecs ... })`, **not** `None`. So a downstream
  consumer pattern-matching `None` to mean "no bars here" is wrong for the
  in-coverage-empty case; it must additionally check `cols.close.is_empty()`.
- Why spec_gap: "no data in `[from_ms, to_ms]`" reads bar-level on the public
  surface, but the realized boundary is file-level (it inherits data-server's own
  file-skip `None`). Two readings are equally plausible — "`None` ⇔ zero bars"
  (what the wording suggests) vs "`None` ⇔ no overlapping file, else `Some`
  (possibly empty)" (what ships). I asserted the **observed (file-level)** reading
  in the fixture; a tightening to the bar-level reading would flip that assertion.
- Recommended action: ratify-or-tighten the ledger/rustdoc. Either document
  precisely ("`None` only when no file overlaps the window or the symbol is
  unknown; an in-coverage window with no bars returns `Some` with empty columns —
  check `.close.is_empty()`"), or collapse the empty case into `None` so the
  binary `Some`/`None` cleanly means "has bars" / "has none". One rustdoc line on
  `load_m1_window` closes it.

### [spec_gap] the only documented real symbol is too sparse for a meaningful signal-quality run over a realistic window
- Example: c0011_3.
- What happened: AAPL.US — the symbol named in the carrier and the gated
  integration test — has **21 M1 bars for all of August 2006** and **~100 for the
  whole year 2006** (verified via `load_m1_window` over a year/month/day window).
  The end-to-end backtest therefore ran but produced **all-zero metrics**
  (`total_pips 0, max_drawdown 0, exposure_sign_flips 0`): with any realistic
  SMA-cross (even SMA(10)/SMA(30)) the slow SMA never warms over ≤21 bars, so the
  signal never fires (correct C2 warm-up / no-look-ahead behaviour — **not a
  bug**). The boundary itself is faultless; the data behind the milestone's
  shipped example is too thin to demonstrate a *non-degenerate* run.
- Why spec_gap: the milestone promise is "a backtest runs over real bars" — which
  is technically met — but a downstream consumer following the documented surface
  (this symbol, this archive) cannot reach a *meaningful* signal-quality result,
  and nothing on the public surface signals that 21 bars/month is the expected
  density (vs a window-bounds bug, a corrupt file, or a normalization error). The
  consumer guesses "this is just sparse early data"; "the window is mis-bounded"
  is an equally plausible reading from the surface alone. I picked the
  sparse-data reading (the bounds were honoured, ts are in-window, the year window
  also returns only ~100 bars).
- Recommended action: ratify (document expected per-symbol bar density, or name a
  denser default symbol/window for the milestone demo) **or** plan a follow-up
  that points the walking-skeleton sample at a real window dense enough to warm a
  realistic signal — so "runs over real bars" demonstrably yields a non-degenerate
  trace, consistent with C22's "newcomer sees a populated trace". No code change
  to `aura-ingest` implied; this is about which data the milestone showcases.

## Recommendation summary

| Finding | Class | Action |
|---|---|---|
| transpose/normalize/close_stream match C3/C7 from rustdoc | working | carry-on |
| close_stream → deterministic engine run, epoch-ns end to end | working | carry-on |
| load_m1_window real bars + documented None paths | working | carry-on |
| `Some(empty)` vs `None` boundary not crisp | spec_gap | ratify / tighten the ledger + rustdoc |
| AAPL.US too sparse for a non-degenerate milestone run | spec_gap | ratify / plan a denser demo symbol+window |