diff --git a/docs/specs/fieldtest-0011-ingest.md b/docs/specs/fieldtest-0011-ingest.md new file mode 100644 index 0000000..766a122 --- /dev/null +++ b/docs/specs/fieldtest-0011-ingest.md @@ -0,0 +1,187 @@ +# Fieldtest — cycle-0011 — 2026-06-04 + +**Status:** Draft — awaiting orchestrator triage +**Author:** fieldtester (dispatched by fieldtest skill) + +## Scope + +Cycle 0011 (Walking-skeleton milestone, #7) shipped a NEW public crate +`aura-ingest` — aura's first real data-source ingestion boundary and the +workspace's external-dependency firewall (it alone links `data-server` and its +transitive `chrono`/`regex`/`zip`). Public surface field-tested: + +1. `unix_ms_to_epoch_ns(time_ms: i64) -> Timestamp` — the single C3 unit + normalization (`= ms * 1_000_000`), at this one boundary and nowhere else. +2. `M1Columns` — the OHLCV bar transposed AoS → SoA (C7): public fields + `ts: Vec`, `open/high/low/close/spread: Vec`, + `volume: Vec`; derives Clone + Debug + PartialEq. +3. `transpose_m1(&[M1Parsed]) -> M1Columns` — pure AoS → SoA, normalizing time + at the boundary. +4. `M1Columns::close_stream() -> Vec<(Timestamp, Scalar)>` — the close column as + the engine source-stream shape the SMA-cross sample strategy consumes. +5. `load_m1_window(&Arc, symbol, from_ms, to_ms) -> Option` + — drains a data-server M1 window and transposes once at the boundary. + +Tested from the public interface only: `cargo doc -p aura-ingest` / +`cargo doc -p data-server` rustdoc, the design ledger (C1/C3/C7/C10/C12/C16), +the `data-server` README, and the observable behaviour of the functions — +never `crates/*/src`. The build exercised: `cargo build --workspace` (green from +HEAD), then each fixture via +`cargo run --manifest-path fieldtests/cycle-0011-ingest/Cargo.toml --bin ` +so HEAD source is always what runs. The fixture crate is a standalone non-member +crate path-depending on `aura-core/-engine/-std/-ingest` plus the `data-server` +git dep, exactly as a C16 research project that ingests recorded bars would. + +## Examples + +### fieldtests/cycle-0011-ingest/c0011_1_transpose_core.rs — the pure transpose/normalize core (axis a) +- Drives `unix_ms_to_epoch_ns` on representative values (0, 1 ms, a real 2006-era + Unix-ms), then `transpose_m1` on three hand-built `M1Parsed` bars with distinct + per-field values (so a swapped column would be caught), checks each SoA column + field-wise, index alignment (`ts[i]` ↔ `close[i]`), `volume` kept i64, purity + (identical bars → equal columns via PartialEq), `close_stream` shape, and the + empty-input edge. +- Why it fits: this is the C3/C7 boundary contract a project author transposing + recorded data exercises, verified from rustdoc alone. +- Outcome: built and ran first try; every assertion held. `ms*1e6` exact, + field-wise transpose correct, volume i64, pure, `close_stream` = + `(normalized ts, close as Scalar::F64)`, empty → empty. Matched expected. + +### fieldtests/cycle-0011-ingest/c0011_2_close_stream_run.rs — close_stream → a real engine run (axis b) +- Transposes synthetic OHLCV bars (close rises then reverses), takes + `close_stream()`, feeds it straight into a Harness (SMA(2)/SMA(4) → Sub → + Exposure(4) → SimBroker(1.0) → two shipped `aura_std::Recorder` sinks), runs, + drains, `f64_field` + `summarize`, asserts finite metrics, ≥1 exposure flip, + that recorded equity timestamps equal the ingested normalized bar timestamps + (epoch-ns end to end), and that a disjoint second run is bit-identical (C1). +- Why it fits: this is the C3 → C1 hand-off — the `(Timestamp, Scalar)` shape + `close_stream` returns is exactly `Harness::run`'s source-stream input shape; + the ingestion boundary must connect to the engine with no hand-authored stream. +- Outcome: built and ran first try. equity `[0,0,0,0,1,0,-0.5]`, exposure flips + once at the reversal, `total_pips -0.5`, `max_drawdown 1.5`, recorded ts == + ingested ts, two runs byte-identical. Matched expected. + +### fieldtests/cycle-0011-ingest/c0011_3_load_window_real.rs — load_m1_window against real data (axis c) +- Opens the local `data-server` archive (`DEFAULT_DATA_PATH`), loads the real + AAPL.US 2006-08 M1 window, checks SoA column lengths aligned, every bar ts + normalized to epoch-ns and inside the requested window, `close_stream` + non-decreasing, then runs an end-to-end SMA-cross signal-quality backtest over + the real close bars (deterministic across two runs, C1). Then exercises the + documented `None` paths (far-future window, unknown symbol) and the + in-coverage-but-empty sub-window grey zone. Skips cleanly where + `/mnt/tickdata` is absent. +- Why it fits: the milestone's headline — "a backtest runs over real bars" — + driven exactly as a downstream researcher would, plus the documented `None` + path. +- Outcome: built and ran first try on the real archive. Loaded **21** bars for + 2006-08; ts normalized and in-window; `close_stream` ascending; the end-to-end + run was deterministic; far-future window → `None`, unknown symbol → `None`; + an in-coverage empty sub-window → `Some(empty)`. Surfaced two spec_gaps (below). + +## Findings + +### [working] transpose_m1 / unix_ms_to_epoch_ns / close_stream match the C3/C7 contract from rustdoc alone +- Example: c0011_1. +- What happened: `unix_ms_to_epoch_ns` is exactly `ms * 1_000_000` (checked at 0, + 1 ms, and a real 2006 ms = 1_154_390_400_000 → 1_154_390_400_000_000_000 ns); + `transpose_m1` produces field-wise-correct SoA columns with no column swap, + `volume` kept `i64` (C7's one i64 column), `ts[i]` aligned with `close[i]`; + it is pure (identical bars → PartialEq-equal columns, C1); `close_stream` + yields `(normalized ts, Scalar::F64(close))` per bar; empty input → empty + columns and empty stream. +- Why working: the entire pure boundary core is reachable, correct, and + unambiguous from the public rustdoc + ledger (C3/C7/C1) without reading any + implementation source. The public-fielded `M1Columns` + the `M1Parsed` public + fields make a hand-built fixture trivial to author. +- Recommended action: carry-on. + +### [working] close_stream feeds Harness::run directly; epoch-ns flows end to end; deterministic +- Example: c0011_2. +- What happened: `close_stream()`'s `Vec<(Timestamp, Scalar)>` is exactly the + shape `Harness::run(vec![stream])` consumes — no adapter. Over a rise-then- + reverse close series the chain warmed, the recorded equity-curve timestamps + equalled the ingested (normalized epoch-ns) bar timestamps verbatim + (normalization survives the whole run loop intact), one exposure sign flip was + registered at the reversal, and a disjoint second run produced a bit-identical + `RunReport.to_json()` (C1). +- Why working: the C3 (ingestion) → C1 (deterministic run) seam — the milestone's + reason to exist — is reachable and correct from the public surface as a + downstream consumer, mirroring the cycle-0007/0009 patterns over real ingested + data rather than a hand-authored stream. +- Recommended action: carry-on. + +### [working] load_m1_window over real bars: in-window normalized, ascending, deterministic; far-future & unknown-symbol None paths hold +- Example: c0011_3. +- What happened: against the real archive, `load_m1_window("AAPL.US", 2006-08 + window)` returned `Some(M1Columns)` with 21 bars; all SoA columns equal length, + every ts normalized to epoch-ns and inside the requested (×1e6) window, + `close_stream` non-decreasing in ts; the end-to-end backtest over the real + closes ran and was bit-identical across two disjoint runs. A far-future window + for the present symbol → `None`; an unknown symbol → `None`, both as the + rustdoc documents. +- Why working: the real-data path — open archive, load a window, transpose once + at the boundary, run a deterministic backtest — works first try from the public + surface, and the two documented `None` conditions (no file overlaps the window; + unknown symbol) behave as stated. +- Recommended action: carry-on. + +### [spec_gap] `Some(empty)` vs `None`: an in-coverage but bar-empty window returns `Some(empty M1Columns)`, not `None` +- Example: c0011_3 (grey-zone block). +- What happened: the `load_m1_window` rustdoc states "`None` if the symbol has no + data in `[from_ms, to_ms]`". Empirically, `None` is **file-level**: it occurs + when no data-server file overlaps the window (e.g. a far-future window — those + files are skipped without I/O) or the symbol is unknown. But a **narrow window + inside a loaded file's date coverage that happens to contain zero bars** + (a single 2006-08-01 day inside the loaded 2006-08 file) returns + `Some(M1Columns { ... empty vecs ... })`, **not** `None`. So a downstream + consumer pattern-matching `None` to mean "no bars here" is wrong for the + in-coverage-empty case; it must additionally check `cols.close.is_empty()`. +- Why spec_gap: "no data in `[from_ms, to_ms]`" reads bar-level on the public + surface, but the realized boundary is file-level (it inherits data-server's own + file-skip `None`). Two readings are equally plausible — "`None` ⇔ zero bars" + (what the wording suggests) vs "`None` ⇔ no overlapping file, else `Some` + (possibly empty)" (what ships). I asserted the **observed (file-level)** reading + in the fixture; a tightening to the bar-level reading would flip that assertion. +- Recommended action: ratify-or-tighten the ledger/rustdoc. Either document + precisely ("`None` only when no file overlaps the window or the symbol is + unknown; an in-coverage window with no bars returns `Some` with empty columns — + check `.close.is_empty()`"), or collapse the empty case into `None` so the + binary `Some`/`None` cleanly means "has bars" / "has none". One rustdoc line on + `load_m1_window` closes it. + +### [spec_gap] the only documented real symbol is too sparse for a meaningful signal-quality run over a realistic window +- Example: c0011_3. +- What happened: AAPL.US — the symbol named in the carrier and the gated + integration test — has **21 M1 bars for all of August 2006** and **~100 for the + whole year 2006** (verified via `load_m1_window` over a year/month/day window). + The end-to-end backtest therefore ran but produced **all-zero metrics** + (`total_pips 0, max_drawdown 0, exposure_sign_flips 0`): with any realistic + SMA-cross (even SMA(10)/SMA(30)) the slow SMA never warms over ≤21 bars, so the + signal never fires (correct C2 warm-up / no-look-ahead behaviour — **not a + bug**). The boundary itself is faultless; the data behind the milestone's + shipped example is too thin to demonstrate a *non-degenerate* run. +- Why spec_gap: the milestone promise is "a backtest runs over real bars" — which + is technically met — but a downstream consumer following the documented surface + (this symbol, this archive) cannot reach a *meaningful* signal-quality result, + and nothing on the public surface signals that 21 bars/month is the expected + density (vs a window-bounds bug, a corrupt file, or a normalization error). The + consumer guesses "this is just sparse early data"; "the window is mis-bounded" + is an equally plausible reading from the surface alone. I picked the + sparse-data reading (the bounds were honoured, ts are in-window, the year window + also returns only ~100 bars). +- Recommended action: ratify (document expected per-symbol bar density, or name a + denser default symbol/window for the milestone demo) **or** plan a follow-up + that points the walking-skeleton sample at a real window dense enough to warm a + realistic signal — so "runs over real bars" demonstrably yields a non-degenerate + trace, consistent with C22's "newcomer sees a populated trace". No code change + to `aura-ingest` implied; this is about which data the milestone showcases. + +## Recommendation summary + +| Finding | Class | Action | +|---|---|---| +| transpose/normalize/close_stream match C3/C7 from rustdoc | working | carry-on | +| close_stream → deterministic engine run, epoch-ns end to end | working | carry-on | +| load_m1_window real bars + documented None paths | working | carry-on | +| `Some(empty)` vs `None` boundary not crisp | spec_gap | ratify / tighten the ledger + rustdoc | +| AAPL.US too sparse for a non-degenerate milestone run | spec_gap | ratify / plan a denser demo symbol+window | diff --git a/fieldtests/cycle-0011-ingest/Cargo.lock b/fieldtests/cycle-0011-ingest/Cargo.lock new file mode 100644 index 0000000..fa1ec60 --- /dev/null +++ b/fieldtests/cycle-0011-ingest/Cargo.lock @@ -0,0 +1,938 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "adler2" +version = "2.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "320119579fcad9c21884f5c4861d16174d0e06250625266f50fe6898340abefa" + +[[package]] +name = "aes" +version = "0.8.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b169f7a6d4742236a0a00c541b845991d0ac43e546831af1249753ab4c3aa3a0" +dependencies = [ + "cfg-if", + "cipher", + "cpufeatures", +] + +[[package]] +name = "aho-corasick" +version = "1.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ddd31a130427c27518df266943a5308ed92d4b226cc639f5a8f1002816174301" +dependencies = [ + "memchr", +] + +[[package]] +name = "android_system_properties" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "819e7219dbd41043ac279b19830f2efc897156490d7fd6ea916720117ee66311" +dependencies = [ + "libc", +] + +[[package]] +name = "arbitrary" +version = "1.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c3d036a3c4ab069c7b410a2ce876bd74808d2d0888a82667669f8e783a898bf1" +dependencies = [ + "derive_arbitrary", +] + +[[package]] +name = "aura-core" +version = "0.1.0" + +[[package]] +name = "aura-engine" +version = "0.1.0" +dependencies = [ + "aura-core", +] + +[[package]] +name = "aura-ingest" +version = "0.1.0" +dependencies = [ + "aura-core", + "data-server", +] + +[[package]] +name = "aura-std" +version = "0.1.0" +dependencies = [ + "aura-core", +] + +[[package]] +name = "autocfg" +version = "1.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53" + +[[package]] +name = "block-buffer" +version = "0.10.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3078c7629b62d3f0439517fa394996acacc5cbc91c5a20d8c658e77abd503a71" +dependencies = [ + "generic-array", +] + +[[package]] +name = "bumpalo" +version = "3.20.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72f5acc6cb2ba439de613abc23857ec3d78374d8ed5ac84e9d11336e87da8649" + +[[package]] +name = "byteorder" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fd0f2584146f6f2ef48085050886acf353beff7305ebd1ae69500e27c67f64b" + +[[package]] +name = "bzip2" +version = "0.5.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "49ecfb22d906f800d4fe833b6282cf4dc1c298f5057ca0b5445e5c209735ca47" +dependencies = [ + "bzip2-sys", +] + +[[package]] +name = "bzip2-sys" +version = "0.1.13+1.0.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "225bff33b2141874fe80d71e07d6eec4f85c5c216453dd96388240f96e1acc14" +dependencies = [ + "cc", + "pkg-config", +] + +[[package]] +name = "c0011-fieldtest" +version = "0.0.0" +dependencies = [ + "aura-core", + "aura-engine", + "aura-ingest", + "aura-std", + "data-server", +] + +[[package]] +name = "cc" +version = "1.2.63" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "556e016178bb5662a08681bbe0f00f8e17631781a4dfc8c45e466e4b185ec27f" +dependencies = [ + "find-msvc-tools", + "jobserver", + "libc", + "shlex", +] + +[[package]] +name = "cfg-if" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" + +[[package]] +name = "chrono" +version = "0.4.45" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1aa79e62e7697b8e29b513a68abacf485adcd1fe8284a4316c5ae868e6633327" +dependencies = [ + "iana-time-zone", + "js-sys", + "num-traits", + "wasm-bindgen", + "windows-link", +] + +[[package]] +name = "cipher" +version = "0.4.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "773f3b9af64447d2ce9850330c473515014aa235e6a783b02db81ff39e4a3dad" +dependencies = [ + "crypto-common", + "inout", +] + +[[package]] +name = "constant_time_eq" +version = "0.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7c74b8349d32d297c9134b8c88677813a227df8f779daa29bfc29c183fe3dca6" + +[[package]] +name = "core-foundation-sys" +version = "0.8.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b" + +[[package]] +name = "cpufeatures" +version = "0.2.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "59ed5838eebb26a2bb2e58f6d5b5316989ae9d08bab10e0e6d103e656d1b0280" +dependencies = [ + "libc", +] + +[[package]] +name = "crc" +version = "3.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5eb8a2a1cd12ab0d987a5d5e825195d372001a4094a0376319d5a0ad71c1ba0d" +dependencies = [ + "crc-catalog", +] + +[[package]] +name = "crc-catalog" +version = "2.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "217698eaf96b4a3f0bc4f3662aaa55bdf913cd54d7204591faa790070c6d0853" + +[[package]] +name = "crc32fast" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9481c1c90cbf2ac953f07c8d4a58aa3945c425b7185c9154d67a65e4230da511" +dependencies = [ + "cfg-if", +] + +[[package]] +name = "crossbeam-utils" +version = "0.8.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d0a5c400df2834b80a4c3327b3aad3a4c4cd4de0629063962b03235697506a28" + +[[package]] +name = "crypto-common" +version = "0.1.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "78c8292055d1c1df0cce5d180393dc8cce0abec0a7102adb6c7b1eef6016d60a" +dependencies = [ + "generic-array", + "typenum", +] + +[[package]] +name = "data-server" +version = "0.1.0" +source = "git+http://192.168.178.103:3000/Brummel/data-server.git?branch=main#0f5e6655b90eeab0f43a8b0f330b8d1632e80ed4" +dependencies = [ + "chrono", + "regex", + "zip", +] + +[[package]] +name = "deflate64" +version = "0.1.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ac6b926516df9c60bfa16e107b21086399f8285a44ca9711344b9e553c5146e2" + +[[package]] +name = "deranged" +version = "0.5.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7cd812cc2bc1d69d4764bd80df88b4317eaef9e773c75226407d9bc0876b211c" +dependencies = [ + "powerfmt", +] + +[[package]] +name = "derive_arbitrary" +version = "1.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1e567bd82dcff979e4b03460c307b3cdc9e96fde3d73bed1496d2bc75d9dd62a" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "digest" +version = "0.10.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292" +dependencies = [ + "block-buffer", + "crypto-common", + "subtle", +] + +[[package]] +name = "displaydoc" +version = "0.2.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1ac70aa55017e108007fbaf5aa0f54b021c98f92ff8af59d42eda9da96e3dd4f" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "equivalent" +version = "1.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f" + +[[package]] +name = "find-msvc-tools" +version = "0.1.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5baebc0774151f905a1a2cc41989300b1e6fbb29aff0ceffa1064fdd3088d582" + +[[package]] +name = "flate2" +version = "1.1.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "843fba2746e448b37e26a819579957415c8cef339bf08564fe8b7ddbd959573c" +dependencies = [ + "crc32fast", + "miniz_oxide", +] + +[[package]] +name = "futures-core" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7e3450815272ef58cec6d564423f6e755e25379b217b0bc688e295ba24df6b1d" + +[[package]] +name = "futures-task" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "037711b3d59c33004d3856fbdc83b99d4ff37a24768fa1be9ce3538a1cde4393" + +[[package]] +name = "futures-util" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "389ca41296e6190b48053de0321d02a77f32f8a5d2461dd38762c0593805c6d6" +dependencies = [ + "futures-core", + "futures-task", + "pin-project-lite", + "slab", +] + +[[package]] +name = "generic-array" +version = "0.14.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a" +dependencies = [ + "typenum", + "version_check", +] + +[[package]] +name = "getrandom" +version = "0.3.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "899def5c37c4fd7b2664648c28120ecec138e4d395b459e5ca34f9cce2dd77fd" +dependencies = [ + "cfg-if", + "js-sys", + "libc", + "r-efi", + "wasip2", + "wasm-bindgen", +] + +[[package]] +name = "hashbrown" +version = "0.17.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ed5909b6e89a2db4456e54cd5f673791d7eca6732202bbf2a9cc504fe2f9b84a" + +[[package]] +name = "hmac" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6c49c37c09c17a53d937dfbb742eb3a961d65a994e6bcdcf37e7399d0cc8ab5e" +dependencies = [ + "digest", +] + +[[package]] +name = "iana-time-zone" +version = "0.1.65" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e31bc9ad994ba00e440a8aa5c9ef0ec67d5cb5e5cb0cc7f8b744a35b389cc470" +dependencies = [ + "android_system_properties", + "core-foundation-sys", + "iana-time-zone-haiku", + "js-sys", + "log", + "wasm-bindgen", + "windows-core", +] + +[[package]] +name = "iana-time-zone-haiku" +version = "0.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f31827a206f56af32e590ba56d5d2d085f558508192593743f16b2306495269f" +dependencies = [ + "cc", +] + +[[package]] +name = "indexmap" +version = "2.14.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d466e9454f08e4a911e14806c24e16fba1b4c121d1ea474396f396069cf949d9" +dependencies = [ + "equivalent", + "hashbrown", +] + +[[package]] +name = "inout" +version = "0.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "879f10e63c20629ecabbb64a8010319738c66a5cd0c29b02d63d272b03751d01" +dependencies = [ + "generic-array", +] + +[[package]] +name = "jobserver" +version = "0.1.34" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9afb3de4395d6b3e67a780b6de64b51c978ecf11cb9a462c66be7d4ca9039d33" +dependencies = [ + "getrandom", + "libc", +] + +[[package]] +name = "js-sys" +version = "0.3.99" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "142bc4740e452c1e57ade0cbc129f139c9093e354346f0872ef985f4f5cf5f11" +dependencies = [ + "cfg-if", + "futures-util", + "once_cell", + "wasm-bindgen", +] + +[[package]] +name = "libc" +version = "0.2.186" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "68ab91017fe16c622486840e4c83c9a37afeff978bd239b5293d61ece587de66" + +[[package]] +name = "log" +version = "0.4.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "953f07c43838f8e6f9758cab68bf5bed85465e7587ebe0b823f1bcd81978ad3a" + +[[package]] +name = "lzma-rs" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "297e814c836ae64db86b36cf2a557ba54368d03f6afcd7d947c266692f71115e" +dependencies = [ + "byteorder", + "crc", +] + +[[package]] +name = "lzma-sys" +version = "0.1.20" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5fda04ab3764e6cde78b9974eec4f779acaba7c4e84b36eca3cf77c581b85d27" +dependencies = [ + "cc", + "libc", + "pkg-config", +] + +[[package]] +name = "memchr" +version = "2.8.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6b947ae49db0d222b1dbc6b113ce7248a3fc3a6ca21b696717bfc000ba4484d8" + +[[package]] +name = "miniz_oxide" +version = "0.8.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fa76a2c86f704bdb222d66965fb3d63269ce38518b83cb0575fca855ebb6316" +dependencies = [ + "adler2", + "simd-adler32", +] + +[[package]] +name = "num-conv" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "521739c6d2bac4aa25192232afe6841231376b2b26d4d9fae5ecf8ca5772e441" + +[[package]] +name = "num-traits" +version = "0.2.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "071dfc062690e90b734c0b2273ce72ad0ffa95f0c74596bc250dcfd960262841" +dependencies = [ + "autocfg", +] + +[[package]] +name = "once_cell" +version = "1.21.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" + +[[package]] +name = "pbkdf2" +version = "0.12.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f8ed6a7761f76e3b9f92dfb0a60a6a6477c61024b775147ff0973a02653abaf2" +dependencies = [ + "digest", + "hmac", +] + +[[package]] +name = "pin-project-lite" +version = "0.2.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd" + +[[package]] +name = "pkg-config" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "19f132c84eca552bf34cab8ec81f1c1dcc229b811638f9d283dceabe58c5569e" + +[[package]] +name = "powerfmt" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "439ee305def115ba05938db6eb1644ff94165c5ab5e9420d1c1bcedbba909391" + +[[package]] +name = "proc-macro2" +version = "1.0.106" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8fd00f0bb2e90d81d1044c2b32617f68fcb9fa3bb7640c23e9c748e53fb30934" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "quote" +version = "1.0.45" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "41f2619966050689382d2b44f664f4bc593e129785a36d6ee376ddf37259b924" +dependencies = [ + "proc-macro2", +] + +[[package]] +name = "r-efi" +version = "5.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "69cdb34c158ceb288df11e18b4bd39de994f6657d83847bdffdbd7f346754b0f" + +[[package]] +name = "regex" +version = "1.12.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e10754a14b9137dd7b1e3e5b0493cc9171fdd105e0ab477f51b72e7f3ac0e276" +dependencies = [ + "aho-corasick", + "memchr", + "regex-automata", + "regex-syntax", +] + +[[package]] +name = "regex-automata" +version = "0.4.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6e1dd4122fc1595e8162618945476892eefca7b88c52820e74af6262213cae8f" +dependencies = [ + "aho-corasick", + "memchr", + "regex-syntax", +] + +[[package]] +name = "regex-syntax" +version = "0.8.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dc897dd8d9e8bd1ed8cdad82b5966c3e0ecae09fb1907d58efaa013543185d0a" + +[[package]] +name = "rustversion" +version = "1.0.22" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b39cdef0fa800fc44525c84ccb54a029961a8215f9619753635a9c0d2538d46d" + +[[package]] +name = "serde_core" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "41d385c7d4ca58e59fc732af25c3983b67ac852c1a25000afe1175de458b67ad" +dependencies = [ + "serde_derive", +] + +[[package]] +name = "serde_derive" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d540f220d3187173da220f885ab66608367b6574e925011a9353e4badda91d79" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "sha1" +version = "0.10.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e3bf829a2d51ab4a5ddf1352d8470c140cadc8301b2ae1789db023f01cedd6ba" +dependencies = [ + "cfg-if", + "cpufeatures", + "digest", +] + +[[package]] +name = "shlex" +version = "2.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba" + +[[package]] +name = "simd-adler32" +version = "0.3.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "703d5c7ef118737c72f1af64ad2f6f8c5e1921f818cdcb97b8fe6fc69bf66214" + +[[package]] +name = "slab" +version = "0.4.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5" + +[[package]] +name = "subtle" +version = "2.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "13c2bddecc57b384dee18652358fb23172facb8a2c51ccc10d74c157bdea3292" + +[[package]] +name = "syn" +version = "2.0.117" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e665b8803e7b1d2a727f4023456bbbbe74da67099c585258af0ad9c5013b9b99" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "thiserror" +version = "2.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4288b5bcbc7920c07a1149a35cf9590a2aa808e0bc1eafaade0b80947865fbc4" +dependencies = [ + "thiserror-impl", +] + +[[package]] +name = "thiserror-impl" +version = "2.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ebc4ee7f67670e9b64d05fa4253e753e016c6c95ff35b89b7941d6b856dec1d5" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "time" +version = "0.3.47" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "743bd48c283afc0388f9b8827b976905fb217ad9e647fae3a379a9283c4def2c" +dependencies = [ + "deranged", + "num-conv", + "powerfmt", + "serde_core", + "time-core", +] + +[[package]] +name = "time-core" +version = "0.1.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7694e1cfe791f8d31026952abf09c69ca6f6fa4e1a1229e18988f06a04a12dca" + +[[package]] +name = "typenum" +version = "1.20.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6f5e870be6c3b371b77fe0ee0bafb859fa4964b4404c27de1d380043c4dda20" + +[[package]] +name = "unicode-ident" +version = "1.0.24" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" + +[[package]] +name = "version_check" +version = "0.9.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a" + +[[package]] +name = "wasip2" +version = "1.0.3+wasi-0.2.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "20064672db26d7cdc89c7798c48a0fdfac8213434a1186e5ef29fd560ae223d6" +dependencies = [ + "wit-bindgen", +] + +[[package]] +name = "wasm-bindgen" +version = "0.2.122" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3ed04576f974d2b2fba0f38c51dbc5518011e38c36bf1143164be765528fd409" +dependencies = [ + "cfg-if", + "once_cell", + "rustversion", + "wasm-bindgen-macro", + "wasm-bindgen-shared", +] + +[[package]] +name = "wasm-bindgen-macro" +version = "0.2.122" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "916151b09da36bd82f6615cbf3a419e2f0ba23a03c6160e8e92eb6bd4aa1dec6" +dependencies = [ + "quote", + "wasm-bindgen-macro-support", +] + +[[package]] +name = "wasm-bindgen-macro-support" +version = "0.2.122" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "299047362ccbfce148b67ab7e73349f77748e00c8296f9542adfad2ad82c5c5e" +dependencies = [ + "bumpalo", + "proc-macro2", + "quote", + "syn", + "wasm-bindgen-shared", +] + +[[package]] +name = "wasm-bindgen-shared" +version = "0.2.122" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9a929b2c61f11ba3e9bc35b50c1f25cb38e0e892c0c231ae2b8cf78d5dad4437" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "windows-core" +version = "0.62.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b8e83a14d34d0623b51dce9581199302a221863196a1dde71a7663a4c2be9deb" +dependencies = [ + "windows-implement", + "windows-interface", + "windows-link", + "windows-result", + "windows-strings", +] + +[[package]] +name = "windows-implement" +version = "0.60.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "053e2e040ab57b9dc951b72c264860db7eb3b0200ba345b4e4c3b14f67855ddf" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "windows-interface" +version = "0.59.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3f316c4a2570ba26bbec722032c4099d8c8bc095efccdc15688708623367e358" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "windows-link" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" + +[[package]] +name = "windows-result" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7781fa89eaf60850ac3d2da7af8e5242a5ea78d1a11c49bf2910bb5a73853eb5" +dependencies = [ + "windows-link", +] + +[[package]] +name = "windows-strings" +version = "0.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7837d08f69c77cf6b07689544538e017c1bfcf57e34b4c0ff58e6c2cd3b37091" +dependencies = [ + "windows-link", +] + +[[package]] +name = "wit-bindgen" +version = "0.57.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1ebf944e87a7c253233ad6766e082e3cd714b5d03812acc24c318f549614536e" + +[[package]] +name = "xz2" +version = "0.1.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "388c44dc09d76f1536602ead6d325eb532f5c122f17782bd57fb47baeeb767e2" +dependencies = [ + "lzma-sys", +] + +[[package]] +name = "zeroize" +version = "1.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b97154e67e32c85465826e8bcc1c59429aaaf107c1e4a9e53c8d8ccd5eff88d0" +dependencies = [ + "zeroize_derive", +] + +[[package]] +name = "zeroize_derive" +version = "1.4.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "85a5b4158499876c763cb03bc4e49185d3cccbabb15b33c627f7884f43db852e" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "zip" +version = "2.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fabe6324e908f85a1c52063ce7aa26b68dcb7eb6dbc83a2d148403c9bc3eba50" +dependencies = [ + "aes", + "arbitrary", + "bzip2", + "constant_time_eq", + "crc32fast", + "crossbeam-utils", + "deflate64", + "displaydoc", + "flate2", + "getrandom", + "hmac", + "indexmap", + "lzma-rs", + "memchr", + "pbkdf2", + "sha1", + "thiserror", + "time", + "xz2", + "zeroize", + "zopfli", + "zstd", +] + +[[package]] +name = "zopfli" +version = "0.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f05cd8797d63865425ff89b5c4a48804f35ba0ce8d125800027ad6017d2b5249" +dependencies = [ + "bumpalo", + "crc32fast", + "log", + "simd-adler32", +] + +[[package]] +name = "zstd" +version = "0.13.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e91ee311a569c327171651566e07972200e76fcfe2242a4fa446149a3881c08a" +dependencies = [ + "zstd-safe", +] + +[[package]] +name = "zstd-safe" +version = "7.2.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f49c4d5f0abb602a93fb8736af2a4f4dd9512e36f7f570d66e65ff867ed3b9d" +dependencies = [ + "zstd-sys", +] + +[[package]] +name = "zstd-sys" +version = "2.0.16+zstd.1.5.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "91e19ebc2adc8f83e43039e79776e3fda8ca919132d68a1fed6a5faca2683748" +dependencies = [ + "cc", + "pkg-config", +] diff --git a/fieldtests/cycle-0011-ingest/Cargo.toml b/fieldtests/cycle-0011-ingest/Cargo.toml new file mode 100644 index 0000000..4c7c621 --- /dev/null +++ b/fieldtests/cycle-0011-ingest/Cargo.toml @@ -0,0 +1,37 @@ +# Standalone downstream-consumer crate for the cycle-0011 fieldtest +# (the aura-ingest data-server M1 ingestion boundary). +# +# Like the cycle-0007..0010 fixtures, this is NOT a member of the aura +# workspace — it path-depends on the engine crates exactly as a real research +# project (C16) would. It additionally path-deps aura-ingest (this cycle's new +# crate) and the external data-server git dep, as a project that ingests real +# recorded bars would. Built/run via +# cargo run --manifest-path fieldtests/cycle-0011-ingest/Cargo.toml --bin +# so HEAD source is always what runs. +# Empty [workspace] table: marks this fixture crate as its OWN workspace root. +[workspace] + +[package] +name = "c0011-fieldtest" +version = "0.0.0" +edition = "2024" +publish = false + +[dependencies] +aura-core = { path = "../../crates/aura-core" } +aura-engine = { path = "../../crates/aura-engine" } +aura-std = { path = "../../crates/aura-std" } +aura-ingest = { path = "../../crates/aura-ingest" } +data-server = { git = "http://192.168.178.103:3000/Brummel/data-server.git", branch = "main" } + +[[bin]] +name = "c0011_1_transpose_core" +path = "c0011_1_transpose_core.rs" + +[[bin]] +name = "c0011_2_close_stream_run" +path = "c0011_2_close_stream_run.rs" + +[[bin]] +name = "c0011_3_load_window_real" +path = "c0011_3_load_window_real.rs" \ No newline at end of file diff --git a/fieldtests/cycle-0011-ingest/c0011_1_transpose_core.rs b/fieldtests/cycle-0011-ingest/c0011_1_transpose_core.rs new file mode 100644 index 0000000..93eccc6 --- /dev/null +++ b/fieldtests/cycle-0011-ingest/c0011_1_transpose_core.rs @@ -0,0 +1,93 @@ +//! Fieldtest c0011 #1 — the pure transpose/normalize core (axis a). +//! +//! Question under test: a project author who has recorded a handful of M1 bars +//! (as data-server `M1Parsed` AoS records) calls `transpose_m1`, inspects the +//! resulting SoA `M1Columns`, and the C3/C7 boundary contract — field-wise +//! transpose, Unix-ms -> epoch-ns normalization, volume kept as i64 — holds +//! exactly as the public rustdoc promises, WITHOUT reading crates/*/src. +//! +//! Public-surface facts used (aura-ingest rustdoc + design ledger C3/C7): +//! - `unix_ms_to_epoch_ns(time_ms) -> Timestamp` = "Normalize Unix-millisecond +//! to canonical epoch-ns" (the single C3 unit normalization). The feat-commit +//! surface line states `= ms * 1_000_000`. +//! - `transpose_m1(&[M1Parsed]) -> M1Columns` — "Pure: identical bars yield +//! identical columns (C1)". AoS -> SoA (C7). +//! - `M1Columns { ts: Vec, open/high/low/close/spread: Vec, +//! volume: Vec }` — public fields; "All columns share an index: ts[i] +//! is the timestamp of close[i]"; derives Clone + Debug + PartialEq. +//! - `M1Columns::close_stream() -> Vec<(Timestamp, Scalar)>` — close column +//! zipped with normalized ts as `Scalar::F64`. +//! - data-server rustdoc: `M1Parsed { time_ms: i64, open/high/low/close/spread: +//! f64, volume: i64 }`, all public fields, Copy. + +use aura_core::{Scalar, Timestamp}; +use aura_ingest::{M1Columns, transpose_m1, unix_ms_to_epoch_ns}; +use data_server::records::M1Parsed; + +fn main() { + // ---- Part 1: the unit-normalization primitive, standalone (C3). -------- + // The surface claims ns = ms * 1_000_000. Check a few representative values + // a project author would care about: zero, a small ms, and a realistic + // 2006-era Pepperstone Unix-ms timestamp. + assert_eq!(unix_ms_to_epoch_ns(0), Timestamp(0), "zero maps to zero"); + assert_eq!(unix_ms_to_epoch_ns(1), Timestamp(1_000_000), "1 ms = 1e6 ns"); + // 2006-08-01T00:00:00Z is 1_154_390_400_000 ms; ns must be *1e6 of that. + let ms_2006 = 1_154_390_400_000_i64; + assert_eq!( + unix_ms_to_epoch_ns(ms_2006), + Timestamp(ms_2006 * 1_000_000), + "realistic 2006 ms scales by exactly 1e6" + ); + println!("unix_ms_to_epoch_ns: 0->0, 1->1e6, 2006 ms -> {:?}", unix_ms_to_epoch_ns(ms_2006)); + + // ---- Part 2: field-wise AoS -> SoA transpose (C7). --------------------- + // Three hand-built bars, distinct values per field so a transposition that + // swapped two columns (e.g. high<->low) would be caught. + let bars = [ + M1Parsed { time_ms: 1_000, open: 10.0, high: 11.0, low: 9.5, close: 10.5, spread: 0.2, volume: 100 }, + M1Parsed { time_ms: 2_000, open: 10.5, high: 12.0, low: 10.1, close: 11.8, spread: 0.3, volume: 250 }, + M1Parsed { time_ms: 3_000, open: 11.8, high: 12.4, low: 11.0, close: 11.2, spread: 0.25, volume: 75 }, + ]; + + let cols: M1Columns = transpose_m1(&bars); + println!("transposed M1Columns = {cols:?}"); + + // Each column must be exactly the corresponding field of each bar, in order. + assert_eq!(cols.ts, vec![Timestamp(1_000_000_000), Timestamp(2_000_000_000), Timestamp(3_000_000_000)], + "ts column = each bar's time_ms normalized to epoch-ns"); + assert_eq!(cols.open, vec![10.0, 10.5, 11.8], "open column"); + assert_eq!(cols.high, vec![11.0, 12.0, 12.4], "high column (not swapped with low)"); + assert_eq!(cols.low, vec![9.5, 10.1, 11.0], "low column"); + assert_eq!(cols.close, vec![10.5, 11.8, 11.2], "close column"); + assert_eq!(cols.spread, vec![0.2, 0.3, 0.25], "spread column"); + // volume stays i64 (C7: the one i64 column), never coerced to f64. + assert_eq!(cols.volume, vec![100_i64, 250, 75], "volume column kept as i64"); + + // Index alignment: ts[i] is the timestamp of close[i] (rustdoc claim). + for i in 0..bars.len() { + assert_eq!(cols.ts[i], unix_ms_to_epoch_ns(bars[i].time_ms), "ts[{i}] normalized"); + assert_eq!(cols.close[i], bars[i].close, "close[{i}] aligned"); + } + + // ---- Part 3: purity (C1) — identical bars -> identical columns. -------- + let cols_again = transpose_m1(&bars); + assert_eq!(cols, cols_again, "transpose is pure: same input -> same output (PartialEq)"); + + // ---- Part 4: close_stream — the engine source-stream shape (C7). ------- + // The close column zipped with normalized ts as Scalar::F64. + let stream = cols.close_stream(); + println!("close_stream = {stream:?}"); + let expected: Vec<(Timestamp, Scalar)> = vec![ + (Timestamp(1_000_000_000), Scalar::F64(10.5)), + (Timestamp(2_000_000_000), Scalar::F64(11.8)), + (Timestamp(3_000_000_000), Scalar::F64(11.2)), + ]; + assert_eq!(stream, expected, "close_stream = (normalized ts, close as Scalar::F64) per bar"); + + // ---- Part 5: empty edge — a window with no bars. ----------------------- + let empty = transpose_m1(&[]); + assert!(empty.ts.is_empty() && empty.close.is_empty(), "empty input -> empty columns"); + assert!(empty.close_stream().is_empty(), "empty columns -> empty close_stream"); + + println!("c0011_1 OK: transpose_m1 / unix_ms_to_epoch_ns / close_stream match the rustdoc C3/C7 contract"); +} diff --git a/fieldtests/cycle-0011-ingest/c0011_2_close_stream_run.rs b/fieldtests/cycle-0011-ingest/c0011_2_close_stream_run.rs new file mode 100644 index 0000000..0821102 --- /dev/null +++ b/fieldtests/cycle-0011-ingest/c0011_2_close_stream_run.rs @@ -0,0 +1,178 @@ +//! Fieldtest c0011 #2 — close_stream -> a real engine run (axis b). +//! +//! Question under test: a project author transposes recorded M1 bars to +//! `M1Columns`, takes `close_stream()`, and feeds it straight into a Harness +//! (SMA-cross -> Exposure -> SimBroker -> Recorder, the cycle-0007/0009 pattern +//! over the shipped aura_std::Recorder), runs it, drains the recorded equity +//! curve, and the recorded stream is what a downstream consumer expects — the +//! ingestion boundary connects to the engine without a hand-authored stream. +//! +//! This is the C3 -> C1 hand-off: the (Timestamp, Scalar) shape `close_stream` +//! returns is exactly `Harness::run`'s source-stream input shape. +//! +//! Public-surface facts used (rustdoc + design ledger): +//! - aura_ingest::{M1Columns via transpose_m1, close_stream}. +//! - aura_std::{Sma, Sub, Exposure, SimBroker, Recorder}. +//! - aura_engine::{Harness, Edge, SourceSpec, Target, RunManifest, RunReport, +//! f64_field, summarize}. +//! - SimBroker slots: 0 = exposure, 1 = price (ledger C10 Realization 0007). + +use std::sync::mpsc; + +use aura_core::{Firing, Scalar, ScalarKind, Timestamp}; +use aura_engine::{Edge, Harness, RunManifest, RunReport, SourceSpec, Target, f64_field, summarize}; +use aura_ingest::transpose_m1; +use aura_std::{Exposure, Recorder, SimBroker, Sma, Sub}; +use data_server::records::M1Parsed; + +/// Build a deterministic synthetic set of M1 bars (close rises then reverses), +/// the way a project author would after recording a real window. We only use +/// the close column downstream, but transpose_m1 carries the full OHLCV bundle. +fn synthetic_bars() -> Vec { + // close: 100,102,104,106,108,106,104 — a rise then a reversal, so the + // SMA-cross flips sign once (exercises exposure_sign_flips like the CLI demo). + let closes = [100.0, 102.0, 104.0, 106.0, 108.0, 106.0, 104.0]; + closes + .iter() + .enumerate() + .map(|(i, &c)| M1Parsed { + time_ms: (i as i64 + 1) * 60_000, // one M1 bar per minute, Unix-ms + open: c, + high: c + 0.5, + low: c - 0.5, + close: c, + spread: 0.1, + volume: 1_000, + }) + .collect() +} + +fn main() { + // --- Ingestion boundary: AoS bars -> SoA columns -> close source stream --- + let bars = synthetic_bars(); + let cols = transpose_m1(&bars); + let price_stream: Vec<(Timestamp, Scalar)> = cols.close_stream(); + println!("close source stream from ingestion = {price_stream:?}"); + + // The C3 precondition close_stream documents: ascending in timestamp. + for w in price_stream.windows(2) { + assert!(w[0].0 < w[1].0, "close_stream ascending in timestamp (C3 precondition)"); + } + + // --- Wire the signal-quality harness over the ingested close stream. --- + let (eq_tx, eq_rx) = mpsc::channel::<(Timestamp, Vec)>(); + let (exp_tx, exp_rx) = mpsc::channel::<(Timestamp, Vec)>(); + + // nodes: 0=SMA(2) fast, 1=SMA(4) slow, 2=Sub, 3=Exposure(4), 4=SimBroker(1.0), + // 5=equity sink (taps broker), 6=exposure sink (taps Exposure). + let mut h = Harness::bootstrap( + vec![ + Box::new(Sma::new(2)), + Box::new(Sma::new(4)), + Box::new(Sub::new()), + Box::new(Exposure::new(4.0)), + Box::new(SimBroker::new(1.0)), + Box::new(Recorder::new(&[ScalarKind::F64], Firing::Any, eq_tx)), + Box::new(Recorder::new(&[ScalarKind::F64], Firing::Any, exp_tx)), + ], + vec![SourceSpec { + kind: ScalarKind::F64, + targets: vec![ + Target { node: 0, slot: 0 }, // -> SMA fast + Target { node: 1, slot: 0 }, // -> SMA slow + Target { node: 4, slot: 1 }, // -> SimBroker price (slot 1) + ], + }], + vec![ + Edge { from: 0, to: 2, slot: 0, from_field: 0 }, // SMA2 -> Sub.in0 + Edge { from: 1, to: 2, slot: 1, from_field: 0 }, // SMA4 -> Sub.in1 + Edge { from: 2, to: 3, slot: 0, from_field: 0 }, // Sub -> Exposure + Edge { from: 3, to: 4, slot: 0, from_field: 0 }, // Exposure -> SimBroker.in0 + Edge { from: 3, to: 6, slot: 0, from_field: 0 }, // Exposure -> exposure sink + Edge { from: 4, to: 5, slot: 0, from_field: 0 }, // SimBroker -> equity sink + ], + ) + .expect("valid signal-quality DAG over the ingested close stream"); + + // Run #1 over the ingested stream. + h.run(vec![price_stream.clone()]); + let equity_rows: Vec<(Timestamp, Vec)> = eq_rx.try_iter().collect(); + let exposure_rows: Vec<(Timestamp, Vec)> = exp_rx.try_iter().collect(); + println!("equity rows = {equity_rows:?}"); + println!("exposure rows = {exposure_rows:?}"); + + let equity = f64_field(&equity_rows, 0); + let exposure = f64_field(&exposure_rows, 0); + let metrics = summarize(&equity, &exposure); + println!("metrics over ingested bars = {metrics:?}"); + + // The recorded equity curve must be non-empty (the source stream warmed the + // chain) and the metrics finite — the downstream "did the ingested data + // actually drive a sim?" check. + assert!(!equity_rows.is_empty(), "ingested close stream produced a recorded equity curve"); + assert!(metrics.total_pips.is_finite(), "total_pips finite"); + assert!(metrics.max_drawdown.is_finite() && metrics.max_drawdown >= 0.0, "drawdown finite, >= 0"); + // The reversal in the close series must register at least one exposure flip. + assert!(metrics.exposure_sign_flips >= 1, "the rise-then-reverse close flips exposure at least once"); + + // The recorded equity timestamps are exactly the ingested (normalized) bar + // timestamps — confirming epoch-ns flowed end to end, unchanged, through the + // run loop (the broker fires on every price-fresh cycle, C10 Realization). + let recorded_ts: Vec = equity_rows.iter().map(|(t, _)| *t).collect(); + let ingested_ts: Vec = price_stream.iter().map(|(t, _)| *t).collect(); + assert_eq!(recorded_ts, ingested_ts, "recorded equity ts == ingested normalized bar ts (epoch-ns end to end)"); + + // --- Determinism (C1): a second disjoint run over the same ingested stream + // yields a bit-identical report. --- + let (eq_tx2, eq_rx2) = mpsc::channel::<(Timestamp, Vec)>(); + let (exp_tx2, exp_rx2) = mpsc::channel::<(Timestamp, Vec)>(); + let mut h2 = Harness::bootstrap( + vec![ + Box::new(Sma::new(2)), + Box::new(Sma::new(4)), + Box::new(Sub::new()), + Box::new(Exposure::new(4.0)), + Box::new(SimBroker::new(1.0)), + Box::new(Recorder::new(&[ScalarKind::F64], Firing::Any, eq_tx2)), + Box::new(Recorder::new(&[ScalarKind::F64], Firing::Any, exp_tx2)), + ], + vec![SourceSpec { + kind: ScalarKind::F64, + targets: vec![ + Target { node: 0, slot: 0 }, + Target { node: 1, slot: 0 }, + Target { node: 4, slot: 1 }, + ], + }], + vec![ + Edge { from: 0, to: 2, slot: 0, from_field: 0 }, + Edge { from: 1, to: 2, slot: 1, from_field: 0 }, + Edge { from: 2, to: 3, slot: 0, from_field: 0 }, + Edge { from: 3, to: 4, slot: 0, from_field: 0 }, + Edge { from: 3, to: 6, slot: 0, from_field: 0 }, + Edge { from: 4, to: 5, slot: 0, from_field: 0 }, + ], + ) + .expect("valid second DAG"); + h2.run(vec![price_stream]); + let equity2 = f64_field(&eq_rx2.try_iter().collect::>(), 0); + let exposure2 = f64_field(&exp_rx2.try_iter().collect::>(), 0); + let metrics2 = summarize(&equity2, &exposure2); + + let report1 = RunReport { + manifest: RunManifest { + commit: "c0011-ingested-synthetic".to_string(), + params: vec![("sma_fast".to_string(), 2.0), ("sma_slow".to_string(), 4.0)], + window: (ingested_ts[0], *ingested_ts.last().unwrap()), + seed: 0, + broker: "sim-optimal(pip_size=1)".to_string(), + }, + metrics, + }; + let report2 = RunReport { + manifest: report1.manifest.clone(), + metrics: metrics2, + }; + assert_eq!(report1.to_json(), report2.to_json(), "two runs over the same ingested stream are bit-identical (C1)"); + println!("c0011_2 OK: ingested close_stream drove a deterministic end-to-end signal-quality run"); +} diff --git a/fieldtests/cycle-0011-ingest/c0011_3_load_window_real.rs b/fieldtests/cycle-0011-ingest/c0011_3_load_window_real.rs new file mode 100644 index 0000000..189f41a --- /dev/null +++ b/fieldtests/cycle-0011-ingest/c0011_3_load_window_real.rs @@ -0,0 +1,181 @@ +//! Fieldtest c0011 #3 — load_m1_window against real data (axis c). +//! +//! Question under test: a downstream researcher opens the local data-server +//! archive, asks `load_m1_window` for a real symbol/window, gets back SoA +//! `M1Columns`, takes `close_stream()`, and runs an end-to-end signal-quality +//! backtest over REAL recorded AAPL.US 2006 M1 close bars — deterministically. +//! And: the documented `None` path (symbol/window miss) behaves as the rustdoc +//! says. +//! +//! Public-surface facts used (rustdoc + data-server README): +//! - aura_ingest::load_m1_window(&Arc, symbol, from_ms, to_ms) +//! -> Option; "None if the symbol has no data in [from_ms, +//! to_ms]"; inclusive Unix-ms bounds (data-server's contract). +//! - data_server::{DataServer, DEFAULT_DATA_PATH}; DataServer::new(base_path); +//! has_symbol(symbol) -> bool; stream is over Arc. +//! - close_stream() ascending in timestamp -> Harness::run source shape. +//! +//! Skips cleanly (exit 0, prints SKIP) where the local archive is absent, so it +//! is hermetic on machines without /mnt/tickdata — mirroring the gated +//! integration test's documented behaviour. + +use std::path::Path; +use std::sync::{Arc, mpsc}; + +use aura_core::{Firing, Scalar, ScalarKind, Timestamp}; +use aura_engine::{Edge, Harness, SourceSpec, Target, f64_field, summarize}; +use aura_ingest::load_m1_window; +use aura_std::{Exposure, Recorder, SimBroker, Sma, Sub}; +use data_server::{DEFAULT_DATA_PATH, DataServer}; + +// AAPL.US 2006-08 is present per the carrier. Window the whole month, inclusive. +const SYMBOL: &str = "AAPL.US"; +// 2006-08-01T00:00:00Z .. 2006-09-01T00:00:00Z in Unix-ms. +const FROM_MS: i64 = 1_154_390_400_000; +const TO_MS: i64 = 1_157_068_800_000; + +fn main() { + if !Path::new(DEFAULT_DATA_PATH).exists() { + println!("c0011_3 SKIP: {DEFAULT_DATA_PATH} absent (no local archive)"); + return; + } + + let server = Arc::new(DataServer::new(DEFAULT_DATA_PATH)); + if !server.has_symbol(SYMBOL) { + println!("c0011_3 SKIP: symbol {SYMBOL} not in local archive"); + return; + } + + // ---- The happy path: load a real M1 window. ---------------------------- + let cols = match load_m1_window(&server, SYMBOL, FROM_MS, TO_MS) { + Some(c) => c, + None => { + println!("c0011_3 SKIP: {SYMBOL} present but no bars in the 2006-08 window"); + return; + } + }; + let n = cols.close.len(); + println!("loaded {n} real M1 bars for {SYMBOL} in 2006-08"); + assert!(n > 0, "a present symbol+window yields at least one bar"); + + // SoA column lengths are all aligned (C7: one index across columns). + assert_eq!(cols.ts.len(), n, "ts column length matches close"); + assert_eq!(cols.open.len(), n, "open length"); + assert_eq!(cols.high.len(), n, "high length"); + assert_eq!(cols.low.len(), n, "low length"); + assert_eq!(cols.spread.len(), n, "spread length"); + assert_eq!(cols.volume.len(), n, "volume (i64) length"); + + // Timestamps normalized to epoch-ns and inside the requested window + // (the boundary normalizes Unix-ms -> ns; the window bounds are Unix-ms). + let from_ns = Timestamp(FROM_MS * 1_000_000); + let to_ns = Timestamp(TO_MS * 1_000_000); + for (i, t) in cols.ts.iter().enumerate() { + assert!(*t >= from_ns && *t <= to_ns, "bar {i} ts {t:?} inside the normalized window"); + } + + // ---- close_stream is ascending (the C3 precondition Harness::run needs). - + let price_stream: Vec<(Timestamp, Scalar)> = cols.close_stream(); + assert_eq!(price_stream.len(), n, "one close-stream entry per bar"); + for w in price_stream.windows(2) { + assert!(w[0].0 <= w[1].0, "real close_stream non-decreasing in timestamp (C3)"); + } + println!( + "first bar = (ts {:?}, close {:?}), last = (ts {:?}, close {:?})", + price_stream[0].0, price_stream[0].1, + price_stream[n - 1].0, price_stream[n - 1].1, + ); + + // ---- End-to-end signal-quality run over REAL close bars. --------------- + let report1 = run_signal_quality(&price_stream); + let report2 = run_signal_quality(&price_stream); // disjoint second run + println!("metrics over real AAPL.US 2006-08 close bars = {report1:?}"); + + assert!(report1.0.is_finite(), "total_pips finite over real bars"); + assert!(report1.1.is_finite() && report1.1 >= 0.0, "max_drawdown finite, >= 0 over real bars"); + // Determinism (C1): same recorded input -> bit-identical metrics. + assert_eq!(report1, report2, "two runs over the same real window are bit-identical (C1)"); + + // ---- The documented None path: a symbol with no data in the window. ---- + // A far-future window for a present symbol -> data-server's own None. + let future_from = 30_000_000_000_000_i64; // ~year 2920 in Unix-ms + let future_to = 30_000_086_400_000_i64; + let miss = load_m1_window(&server, SYMBOL, future_from, future_to); + assert!(miss.is_none(), "present symbol, empty window -> None (rustdoc)"); + println!("None-path: {SYMBOL} with a far-future window -> {miss:?}"); + + // An unknown symbol -> None as well. + let miss_sym = load_m1_window(&server, "NO.SUCH.SYMBOL_XYZ", FROM_MS, TO_MS); + assert!(miss_sym.is_none(), "unknown symbol -> None (rustdoc / data-server contract)"); + println!("None-path: unknown symbol -> {miss_sym:?}"); + + // ---- The grey zone: a window INSIDE the file's date coverage but with no + // bars in it. The rustdoc says "None if the symbol has no data in + // [from_ms, to_ms]". A narrow single-day window inside 2006-08 (whose + // file IS loaded) but containing zero bars returns Some(EMPTY columns), + // NOT None — data-server's `None` is FILE-level (no file overlaps the + // window, skipped without I/O), while a loaded file that yields no bar + // in the exact sub-window transposes to Some(empty). Recorded as a + // spec_gap: the Some(empty) vs None boundary is not crisp on the public + // surface. We assert the OBSERVED reading. + let one_day_from = FROM_MS; // 2006-08-01T00:00Z + let one_day_to = FROM_MS + 86_400_000; // +1 day + let narrow = load_m1_window(&server, SYMBOL, one_day_from, one_day_to); + match &narrow { + Some(c) if c.close.is_empty() => { + println!("grey-zone: in-coverage empty sub-window -> Some(EMPTY), not None"); + } + Some(c) => println!("grey-zone: in-coverage sub-window -> Some({} bars)", c.close.len()), + None => println!("grey-zone: in-coverage empty sub-window -> None"), + } + // Observed on this archive: Some(empty). Asserting the observed reading so + // the fixture is a faithful record (a future tightening would flip this). + assert!( + matches!(&narrow, Some(c) if c.close.is_empty()), + "in-coverage empty sub-window returns Some(empty), not None (observed reading)" + ); + + println!("c0011_3 OK: load_m1_window drove a deterministic end-to-end run over real bars; None paths hold"); +} + +/// SMA-cross -> Exposure -> SimBroker -> Recorder; returns (total_pips, +/// max_drawdown, exposure_sign_flips) as the summarized metrics tuple. +fn run_signal_quality(price_stream: &[(Timestamp, Scalar)]) -> (f64, f64, u64) { + let (eq_tx, eq_rx) = mpsc::channel::<(Timestamp, Vec)>(); + let (exp_tx, exp_rx) = mpsc::channel::<(Timestamp, Vec)>(); + + let mut h = Harness::bootstrap( + vec![ + Box::new(Sma::new(10)), + Box::new(Sma::new(30)), + Box::new(Sub::new()), + Box::new(Exposure::new(1.0)), + Box::new(SimBroker::new(0.0001)), // AAPL.US pip_size; reference metadata + Box::new(Recorder::new(&[ScalarKind::F64], Firing::Any, eq_tx)), + Box::new(Recorder::new(&[ScalarKind::F64], Firing::Any, exp_tx)), + ], + vec![SourceSpec { + kind: ScalarKind::F64, + targets: vec![ + Target { node: 0, slot: 0 }, + Target { node: 1, slot: 0 }, + Target { node: 4, slot: 1 }, + ], + }], + vec![ + Edge { from: 0, to: 2, slot: 0, from_field: 0 }, + Edge { from: 1, to: 2, slot: 1, from_field: 0 }, + Edge { from: 2, to: 3, slot: 0, from_field: 0 }, + Edge { from: 3, to: 4, slot: 0, from_field: 0 }, + Edge { from: 3, to: 6, slot: 0, from_field: 0 }, + Edge { from: 4, to: 5, slot: 0, from_field: 0 }, + ], + ) + .expect("valid real-bars signal-quality DAG"); + + h.run(vec![price_stream.to_vec()]); + let equity = f64_field(&eq_rx.try_iter().collect::>(), 0); + let exposure = f64_field(&exp_rx.try_iter().collect::>(), 0); + let m = summarize(&equity, &exposure); + (m.total_pips, m.max_drawdown, m.exposure_sign_flips) +}