fieldtest: cycle-0011 — 3 examples, 5 findings

Per-cycle fieldtest of the aura-ingest ingestion boundary (issue #7), from the
public interface only. 3 examples (transpose/normalize core; close_stream ->
deterministic harness run with epoch-ns end-to-end; load_m1_window over real
AAPL.US 2006-08 bars + None paths), all green. 0 bugs, 0 friction.

Two spec_gaps named for follow-up (neither a code defect; both filed to the
tracker):
- load_m1_window returns Some(empty M1Columns) — not None — for an in-coverage
  but bar-empty window. The rustdoc's "None if no data in [from_ms,to_ms]" reads
  bar-level but the realized boundary is file-level (it inherits data-server's
  file-skip None). A consumer matching None == "no bars" is wrong for the
  in-coverage-empty case. -> tighten the load_m1_window rustdoc (one line).
- AAPL.US (the carrier/integration-test symbol) has only ~21 M1 bars/month, so
  the SMA-cross never warms and the end-to-end run yields all-zero (degenerate
  but valid) metrics. The boundary is correct (C2 warm-up); the milestone's
  shipped demo data is too thin to show a non-degenerate trace. -> weigh against
  C22's "newcomer sees a populated trace" before the Walking-skeleton close
  (document expected density or name a denser demo symbol/window).

docs/specs/fieldtest-0011-ingest.md

@@ -0,0 +1,187 @@
+# Fieldtest — cycle-0011 — 2026-06-04
+
+**Status:** Draft — awaiting orchestrator triage
+**Author:** fieldtester (dispatched by fieldtest skill)
+
+## Scope
+
+Cycle 0011 (Walking-skeleton milestone, #7) shipped a NEW public crate
+`aura-ingest` — aura's first real data-source ingestion boundary and the
+workspace's external-dependency firewall (it alone links `data-server` and its
+transitive `chrono`/`regex`/`zip`). Public surface field-tested:
+
+1. `unix_ms_to_epoch_ns(time_ms: i64) -> Timestamp` — the single C3 unit
+   normalization (`= ms * 1_000_000`), at this one boundary and nowhere else.
+2. `M1Columns` — the OHLCV bar transposed AoS → SoA (C7): public fields
+   `ts: Vec<Timestamp>`, `open/high/low/close/spread: Vec<f64>`,
+   `volume: Vec<i64>`; derives Clone + Debug + PartialEq.
+3. `transpose_m1(&[M1Parsed]) -> M1Columns` — pure AoS → SoA, normalizing time
+   at the boundary.
+4. `M1Columns::close_stream() -> Vec<(Timestamp, Scalar)>` — the close column as
+   the engine source-stream shape the SMA-cross sample strategy consumes.
+5. `load_m1_window(&Arc<DataServer>, symbol, from_ms, to_ms) -> Option<M1Columns>`
+   — drains a data-server M1 window and transposes once at the boundary.
+
+Tested from the public interface only: `cargo doc -p aura-ingest` /
+`cargo doc -p data-server` rustdoc, the design ledger (C1/C3/C7/C10/C12/C16),
+the `data-server` README, and the observable behaviour of the functions —
+never `crates/*/src`. The build exercised: `cargo build --workspace` (green from
+HEAD), then each fixture via
+`cargo run --manifest-path fieldtests/cycle-0011-ingest/Cargo.toml --bin <name>`
+so HEAD source is always what runs. The fixture crate is a standalone non-member
+crate path-depending on `aura-core/-engine/-std/-ingest` plus the `data-server`
+git dep, exactly as a C16 research project that ingests recorded bars would.
+
+## Examples
+
+### fieldtests/cycle-0011-ingest/c0011_1_transpose_core.rs — the pure transpose/normalize core (axis a)
+- Drives `unix_ms_to_epoch_ns` on representative values (0, 1 ms, a real 2006-era
+  Unix-ms), then `transpose_m1` on three hand-built `M1Parsed` bars with distinct
+  per-field values (so a swapped column would be caught), checks each SoA column
+  field-wise, index alignment (`ts[i]` ↔ `close[i]`), `volume` kept i64, purity
+  (identical bars → equal columns via PartialEq), `close_stream` shape, and the
+  empty-input edge.
+- Why it fits: this is the C3/C7 boundary contract a project author transposing
+  recorded data exercises, verified from rustdoc alone.
+- Outcome: built and ran first try; every assertion held. `ms*1e6` exact,
+  field-wise transpose correct, volume i64, pure, `close_stream` =
+  `(normalized ts, close as Scalar::F64)`, empty → empty. Matched expected.
+
+### fieldtests/cycle-0011-ingest/c0011_2_close_stream_run.rs — close_stream → a real engine run (axis b)
+- Transposes synthetic OHLCV bars (close rises then reverses), takes
+  `close_stream()`, feeds it straight into a Harness (SMA(2)/SMA(4) → Sub →
+  Exposure(4) → SimBroker(1.0) → two shipped `aura_std::Recorder` sinks), runs,
+  drains, `f64_field` + `summarize`, asserts finite metrics, ≥1 exposure flip,
+  that recorded equity timestamps equal the ingested normalized bar timestamps
+  (epoch-ns end to end), and that a disjoint second run is bit-identical (C1).
+- Why it fits: this is the C3 → C1 hand-off — the `(Timestamp, Scalar)` shape
+  `close_stream` returns is exactly `Harness::run`'s source-stream input shape;
+  the ingestion boundary must connect to the engine with no hand-authored stream.
+- Outcome: built and ran first try. equity `[0,0,0,0,1,0,-0.5]`, exposure flips
+  once at the reversal, `total_pips -0.5`, `max_drawdown 1.5`, recorded ts ==
+  ingested ts, two runs byte-identical. Matched expected.
+
+### fieldtests/cycle-0011-ingest/c0011_3_load_window_real.rs — load_m1_window against real data (axis c)
+- Opens the local `data-server` archive (`DEFAULT_DATA_PATH`), loads the real
+  AAPL.US 2006-08 M1 window, checks SoA column lengths aligned, every bar ts
+  normalized to epoch-ns and inside the requested window, `close_stream`
+  non-decreasing, then runs an end-to-end SMA-cross signal-quality backtest over
+  the real close bars (deterministic across two runs, C1). Then exercises the
+  documented `None` paths (far-future window, unknown symbol) and the
+  in-coverage-but-empty sub-window grey zone. Skips cleanly where
+  `/mnt/tickdata` is absent.
+- Why it fits: the milestone's headline — "a backtest runs over real bars" —
+  driven exactly as a downstream researcher would, plus the documented `None`
+  path.
+- Outcome: built and ran first try on the real archive. Loaded **21** bars for
+  2006-08; ts normalized and in-window; `close_stream` ascending; the end-to-end
+  run was deterministic; far-future window → `None`, unknown symbol → `None`;
+  an in-coverage empty sub-window → `Some(empty)`. Surfaced two spec_gaps (below).
+
+## Findings
+
+### [working] transpose_m1 / unix_ms_to_epoch_ns / close_stream match the C3/C7 contract from rustdoc alone
+- Example: c0011_1.
+- What happened: `unix_ms_to_epoch_ns` is exactly `ms * 1_000_000` (checked at 0,
+  1 ms, and a real 2006 ms = 1_154_390_400_000 → 1_154_390_400_000_000_000 ns);
+  `transpose_m1` produces field-wise-correct SoA columns with no column swap,
+  `volume` kept `i64` (C7's one i64 column), `ts[i]` aligned with `close[i]`;
+  it is pure (identical bars → PartialEq-equal columns, C1); `close_stream`
+  yields `(normalized ts, Scalar::F64(close))` per bar; empty input → empty
+  columns and empty stream.
+- Why working: the entire pure boundary core is reachable, correct, and
+  unambiguous from the public rustdoc + ledger (C3/C7/C1) without reading any
+  implementation source. The public-fielded `M1Columns` + the `M1Parsed` public
+  fields make a hand-built fixture trivial to author.
+- Recommended action: carry-on.
+
+### [working] close_stream feeds Harness::run directly; epoch-ns flows end to end; deterministic
+- Example: c0011_2.
+- What happened: `close_stream()`'s `Vec<(Timestamp, Scalar)>` is exactly the
+  shape `Harness::run(vec![stream])` consumes — no adapter. Over a rise-then-
+  reverse close series the chain warmed, the recorded equity-curve timestamps
+  equalled the ingested (normalized epoch-ns) bar timestamps verbatim
+  (normalization survives the whole run loop intact), one exposure sign flip was
+  registered at the reversal, and a disjoint second run produced a bit-identical
+  `RunReport.to_json()` (C1).
+- Why working: the C3 (ingestion) → C1 (deterministic run) seam — the milestone's
+  reason to exist — is reachable and correct from the public surface as a
+  downstream consumer, mirroring the cycle-0007/0009 patterns over real ingested
+  data rather than a hand-authored stream.
+- Recommended action: carry-on.
+
+### [working] load_m1_window over real bars: in-window normalized, ascending, deterministic; far-future & unknown-symbol None paths hold
+- Example: c0011_3.
+- What happened: against the real archive, `load_m1_window("AAPL.US", 2006-08
+  window)` returned `Some(M1Columns)` with 21 bars; all SoA columns equal length,
+  every ts normalized to epoch-ns and inside the requested (×1e6) window,
+  `close_stream` non-decreasing in ts; the end-to-end backtest over the real
+  closes ran and was bit-identical across two disjoint runs. A far-future window
+  for the present symbol → `None`; an unknown symbol → `None`, both as the
+  rustdoc documents.
+- Why working: the real-data path — open archive, load a window, transpose once
+  at the boundary, run a deterministic backtest — works first try from the public
+  surface, and the two documented `None` conditions (no file overlaps the window;
+  unknown symbol) behave as stated.
+- Recommended action: carry-on.
+
+### [spec_gap] `Some(empty)` vs `None`: an in-coverage but bar-empty window returns `Some(empty M1Columns)`, not `None`
+- Example: c0011_3 (grey-zone block).
+- What happened: the `load_m1_window` rustdoc states "`None` if the symbol has no
+  data in `[from_ms, to_ms]`". Empirically, `None` is **file-level**: it occurs
+  when no data-server file overlaps the window (e.g. a far-future window — those
+  files are skipped without I/O) or the symbol is unknown. But a **narrow window
+  inside a loaded file's date coverage that happens to contain zero bars**
+  (a single 2006-08-01 day inside the loaded 2006-08 file) returns
+  `Some(M1Columns { ... empty vecs ... })`, **not** `None`. So a downstream
+  consumer pattern-matching `None` to mean "no bars here" is wrong for the
+  in-coverage-empty case; it must additionally check `cols.close.is_empty()`.
+- Why spec_gap: "no data in `[from_ms, to_ms]`" reads bar-level on the public
+  surface, but the realized boundary is file-level (it inherits data-server's own
+  file-skip `None`). Two readings are equally plausible — "`None` ⇔ zero bars"
+  (what the wording suggests) vs "`None` ⇔ no overlapping file, else `Some`
+  (possibly empty)" (what ships). I asserted the **observed (file-level)** reading
+  in the fixture; a tightening to the bar-level reading would flip that assertion.
+- Recommended action: ratify-or-tighten the ledger/rustdoc. Either document
+  precisely ("`None` only when no file overlaps the window or the symbol is
+  unknown; an in-coverage window with no bars returns `Some` with empty columns —
+  check `.close.is_empty()`"), or collapse the empty case into `None` so the
+  binary `Some`/`None` cleanly means "has bars" / "has none". One rustdoc line on
+  `load_m1_window` closes it.
+
+### [spec_gap] the only documented real symbol is too sparse for a meaningful signal-quality run over a realistic window
+- Example: c0011_3.
+- What happened: AAPL.US — the symbol named in the carrier and the gated
+  integration test — has **21 M1 bars for all of August 2006** and **~100 for the
+  whole year 2006** (verified via `load_m1_window` over a year/month/day window).
+  The end-to-end backtest therefore ran but produced **all-zero metrics**
+  (`total_pips 0, max_drawdown 0, exposure_sign_flips 0`): with any realistic
+  SMA-cross (even SMA(10)/SMA(30)) the slow SMA never warms over ≤21 bars, so the
+  signal never fires (correct C2 warm-up / no-look-ahead behaviour — **not a
+  bug**). The boundary itself is faultless; the data behind the milestone's
+  shipped example is too thin to demonstrate a *non-degenerate* run.
+- Why spec_gap: the milestone promise is "a backtest runs over real bars" — which
+  is technically met — but a downstream consumer following the documented surface
+  (this symbol, this archive) cannot reach a *meaningful* signal-quality result,
+  and nothing on the public surface signals that 21 bars/month is the expected
+  density (vs a window-bounds bug, a corrupt file, or a normalization error). The
+  consumer guesses "this is just sparse early data"; "the window is mis-bounded"
+  is an equally plausible reading from the surface alone. I picked the
+  sparse-data reading (the bounds were honoured, ts are in-window, the year window
+  also returns only ~100 bars).
+- Recommended action: ratify (document expected per-symbol bar density, or name a
+  denser default symbol/window for the milestone demo) **or** plan a follow-up
+  that points the walking-skeleton sample at a real window dense enough to warm a
+  realistic signal — so "runs over real bars" demonstrably yields a non-degenerate
+  trace, consistent with C22's "newcomer sees a populated trace". No code change
+  to `aura-ingest` implied; this is about which data the milestone showcases.
+
+## Recommendation summary
+
+| Finding | Class | Action |
+|---|---|---|
+| transpose/normalize/close_stream match C3/C7 from rustdoc | working | carry-on |
+| close_stream → deterministic engine run, epoch-ns end to end | working | carry-on |
+| load_m1_window real bars + documented None paths | working | carry-on |
+| `Some(empty)` vs `None` boundary not crisp | spec_gap | ratify / tighten the ledger + rustdoc |
+| AAPL.US too sparse for a non-degenerate milestone run | spec_gap | ratify / plan a denser demo symbol+window |