diff --git a/docs/specs/fieldtest-research-breakout-deepdive.md b/docs/specs/fieldtest-research-breakout-deepdive.md new file mode 100644 index 0000000..05f8413 --- /dev/null +++ b/docs/specs/fieldtest-research-breakout-deepdive.md @@ -0,0 +1,246 @@ +# Fieldtest — research deep-dive on the GER40 session-breakout demo — 2026-06-17 + +**Status:** Draft — awaiting orchestrator triage +**Author:** fieldtester (downstream-quant-researcher dispatch) + +## Scope +The just-shipped runnable demo `crates/aura-ingest/examples/ger40_breakout_real.rs` +(+ `examples/shared/breakout_real.rs`, commit `5b5f034`) backtests the 15m +Frankfurt session-breakout over real GER40 M1 bars for one month. This dispatch +pushes that strategy through the escalating sequence a serious researcher attempts +next: (1) scale to a full year and a 7-year span; (2) walk-forward across real +history; (3) sweep the structural params (entry/exit bar, bar period); (4) compare +signal quality across instruments (GER40 vs FRA40). Worked ONLY from the public +surface (design ledger, glossary, the example corpus, `cargo doc`); no engine +`crates/*/src` was read. All runs are against the real `/mnt/tickdata/Pepperstone` +archive. Binaries path-dep the engine crates (HEAD source), built `--release`. + +The central, recurring tension: **the demo ships the breakout as a hand-wired raw +`FlatGraph`, but every World orchestration family (`sweep` / `walk_forward` / +`optimize`) consumes a `Composite` blueprint with `param_space()` / +`bootstrap_with_cells`.** A `FlatGraph` has no param space, so to take *this exact +strategy* into the World a researcher must first re-author it as a `Composite` — +re-typing the 13-node wiring by hand. The demo provides no such blueprint. + +## Examples +All fixtures live under `fieldtests/research-breakout-deepdive/` (a standalone +C16-style consumer crate). Output transcripts are the sibling `out_*.txt`. + +### t1_scale.rs — scale the breakout over long horizons +- Reuses the shipped `shared/breakout_real.rs` wiring verbatim (via `#[path]`), + changing only the window: month → year → 7 years; adds an RSS probe + timer. +- Fits scope: the first thing a researcher does with a one-month demo is run it + over real history; the docs make a load-bearing O(one-chunk) residency claim. +- Outcome: built, ran. 2024-09 −45.0 pips / 5 sessions (matches the shipped + demo). 2024 full year +203.4 / 62 sessions. 2018..2024 (7y) +520.2 / 401 + sessions in 1.12s. **Determinism held** (rerun report JSON bit-identical). + **But RSS grew 8→127 MiB on the 7y run.** + +### t1b_residency.rs / t1c_residency_scaling.rs — residency attribution +- Isolates where the memory goes: drives 4 raw `M1FieldSource`s with no harness/ + sinks, then the full harness; and sweeps one source over 1mo→10yr windows. +- Fits scope: pin down whether the O(one-chunk) claim holds at the process level. +- Outcome: built, ran. `resident_records ≤ 1024` (the docs' asserted predicate) + holds at every window length, **yet VmHWM scales linearly with window**: 6 → 15 + → 24 → 78 → 173 MiB across 1mo → 6mo → 1y → 4y → 10y of one symbol's Close. + +### t2_walkforward.rs — walk-forward the shipped breakout (2018..2024) +- Rolls the breakout with `walk_forward` (12mo IS / 3mo OOS / 3mo step, Rolling), + stitching the OOS pip curve. +- Fits scope: the headline World promise on the headline strategy. +- Outcome: built, ran. 24 windows; stitched OOS curve 131508 points ending + +618.8 pips. **But the walk-forward is degenerate** — see finding below: the + breakout being a raw `FlatGraph` means `space = vec![]`, `chosen_params = + vec![]`, and there is NO in-sample optimization (the whole point of WFO). + +### t3_sweep.rs / t3b_period_desync.rs — sweep the structural params +- Re-authors the breakout as a `Composite`, prints `param_space()`, runs a 3×3 + entry_bar×exit_bar grid, and demonstrates the period-coupling failure. +- Fits scope: the mission's explicit coupling question (Resample period == + Session period; entry/exit as `EqConst` targets). +- Outcome: built, ran. Grid works; `entry_bar=3,exit_bar=5` reproduces the + shipped −45.0 exactly (re-authoring fidelity). The coupling is **not** + expressible, and violating it **fails silently** (period-desync run → 0.0 pips, + no error). + +### t4_compare.rs — GER40 vs FRA40 signal quality +- Reuses the `Composite`, loops over `(symbol, pip_size)`, compares 2024. +- Fits scope: the structural "which instrument" axis + the per-asset pip question. +- Outcome: built, ran. GER40 +203.4 / 62 sessions vs FRA40 +83.7 / 63 sessions + (both pip=1.0 index points). Comparison is honest *only because the researcher + hand-supplied matching pip sizes* — there is no instrument registry. + +## Findings + +### [spec_gap] The shipped breakout is a raw FlatGraph, unreachable by the World API +- Example(s): t2_walkforward, t3_sweep, t4_compare (all three had to re-author it). +- What happened: `sweep` / `walk_forward` / `optimize` operate on a `Composite` + (they need `param_space()` + `bootstrap_with_cells`). The demo's breakout is a + hand-wired `FlatGraph` built by `Harness::bootstrap`, which has no param space. + To do *any* sweep/optimize/WFO of the shipped strategy I re-typed all 13 nodes + + 14 edges + 4 roles as a `Composite` (t3_sweep.rs `breakout_composite`). +- Why spec_gap: C20/C21 promise the strategy blueprint is "reused across backtest, + sweep, visual workspaces"; the milestone fixtures demonstrate this for SMA-cross. + But the *just-shipped, real-data flagship* strategy ships in a form the World + cannot consume, and the design ledger does not say which form a shippable + strategy should take. A researcher cannot tell from the public surface whether + the FlatGraph form is intentional or a gap. The re-reading I picked (re-author as + Composite) is plausible; "the demo should have shipped a Composite" is equally + plausible — the ledger is silent. +- Recommended action: ship the breakout as a reusable `Composite` blueprint + (alongside or instead of the FlatGraph), so the demo's strategy is World-ready; + or document in the example that the FlatGraph is a substrate-level fixture and a + blueprint is the consumer form. + +### [bug] Process residency scales with window length despite the O(one-chunk) claim +- Example(s): t1_scale, t1b_residency, t1c_residency_scaling. +- What happened: the docs (C12 cycle-0041 realization; `tests/streaming_seam.rs` + `residency_is_bounded_by_one_chunk_independent_of_window_length`) claim "a + 20-year window streams in the same memory as a one-day one". Empirically the + aura-level predicate `resident_records ≤ CHUNK_SIZE (1024)` holds at every + window length, but VmHWM (OS peak RSS) grows ~linearly: 6→15→24→78→173 MiB for + 1mo→6mo→1y→4y→10y of ONE symbol's Close field; the 7-year 4-field breakout hits + 127 MiB streaming, 210 MiB after `drain_trace`. The test's predicate measures + only the aura `M1FieldSource` ring, not the underlying `data-server` chunk/file + retention, so it passes while the process-level promise is false. +- Why bug: a load-bearing, documented, test-asserted guarantee ("streaming + residency is O(one chunk), independent of window length") is contradicted by the + process's own VmHWM. A researcher who budgets RAM on the doc's promise (one chunk + ≈ tens of KiB) will be off by 100-1000×; a real multi-symbol sweep over decades + would OOM where the docs say it streams flat. +- Repro: `cargo run --release --manifest-path + fieldtests/research-breakout-deepdive/Cargo.toml --bin t1c_residency_scaling` + → VmHWM column grows monotonically with the record count while the + `aura-resident<=1024?` column stays `true`. +- Recommended action: `debug` — either the `data-server` chunks/decoded buffers + are retained across pulls (release them), or the residency claim must be narrowed + in the docs/test to "aura-level ring residency" (which is all the predicate + proves) and the true per-window process cost documented. + +### [spec_gap] walk_forward demands a param-space even with nothing to optimize +- Example(s): t2_walkforward. +- What happened: `walk_forward(roller, space: Vec, F)` with `WindowRun { + chosen_params: Vec, .. }`. For a fixed-param OOS roll of a param-free + strategy I passed `space = vec![]` and `chosen_params = vec![]`. It ran, but the + result is a walk-forward with no in-sample optimization — the OOS verdict of a + strategy that was never tuned per window. The signature gives no hint whether the + empty-space idiom is the intended way to express "OOS-only, no optimize", or + whether WFO without per-window optimize is even a supported use. +- Why spec_gap: I guessed the empty-space reading; "WFO requires a non-empty space + and per-window optimize" is an equally plausible reading the ledger does not + settle. Combined with the FlatGraph finding, the headline WFO demo on the + headline strategy is unreachable in its *optimizing* form without re-authoring. +- Recommended action: `ratify` the param-free OOS-roll idiom (document that empty + space + empty chosen_params is a valid fixed-strategy walk-forward), or tighten + `walk_forward` to make the optimize-per-window contract explicit. + +### [bug] Structural deformers (Delay.lag, Resample.period_minutes) leak into param_space +- Example(s): t3_sweep (the grid panicked on the un-enumerated `delay.lag` slot). +- What happened: the breakout `Composite`'s `param_space()` is `{ + resample.period_minutes, delay.lag, entry_bar.target, exit_bar.target }`. + `delay.lag` is the breakout's "previous bar's high" — a STRUCTURAL constant + (varying it deforms the strategy, the exact C34 deform-vs-tune discriminator), + yet `Delay::builder()` ships `lag` as a tunable param. So `GridSpace::new` + REQUIRES the researcher to enumerate `delay.lag` (or it errors), and a naive + random sweep would silently deform the strategy. C34's `bind()` can remove it, + but the node pollutes the space by default. +- Why bug: this is the precise failure C34 (cycle 0034) exists to prevent — a + structural deformer presented as a tuning knob. The strategy author did not opt + into it; the node's default `builder()` forces the pollution onto every consumer. +- Repro: `cargo run --release --manifest-path + fieldtests/research-breakout-deepdive/Cargo.toml --bin t3_sweep` → `param_space()` + lists `delay.lag`; removing its grid axis makes `GridSpace::new` reject the grid. +- Recommended action: `debug` / `planner` — decide whether `Delay.lag` (and + `Resample.period_minutes`) are tuning params or structural constants; if + structural, they should not appear in a default `param_space()` (or the demo + should `bind()` them out and the example should model that for consumers). + +### [bug] The Resample/Session period coupling is unenforceable and fails silently +- Example(s): t3_sweep, t3b_period_desync. +- What happened: the breakout's bar period lives in TWO places — + `Resample::builder()` exposes `period_minutes` as a tunable param, but + `Session::builder(open, tz, period)` BAKES its period (its rustdoc: "NOT scalar + params, the declared param list is empty"). So `param_space()` has + `resample.period_minutes` but NO session period. Sweeping `resample.period_minutes + = 30` while Session stays baked at 15 bootstraps and runs with NO error, but the + two clocks desync: `entry_bar.target == 3` no longer means "3rd 30m bar". The + desynced run produced **0.0 pips / no error**; a coherent 30m strategy (+63.0 + pips) was reachable ONLY by rebuilding the `Composite` with `session_period = 30` + (a different blueprint — topology, not a sweep). +- Why bug: the public sweep API silently admits a parameter combination that is + structurally meaningless, and the failure mode is a quiet 0.0-pip result rather + than a diagnostic. The two periods are one logical knob split across a tunable + param and a baked factory arg with no coupling mechanism on the public surface. +- Repro: `cargo run --release --manifest-path + fieldtests/research-breakout-deepdive/Cargo.toml --bin t3b_period_desync` → the + resample=30/session=15 row prints `total_pips=0.0` with no error. +- Recommended action: `debug` / `planner` — either make Session's period a tunable + param so a single swept value feeds both (the C8/C12 way), or provide a coupling/ + derived-param mechanism, or have the demo `bind` the period as a structural + constant so it cannot be swept into a desync. A silent 0.0 is the worst outcome. + +### [spec_gap] No per-instrument pip/metadata registry; cross-asset honesty is manual +- Example(s): t4_compare. +- What happened: `SimBroker::new(pip_size)` takes a bare `f64`. There is no + instrument/pip-metadata registry on the public surface — `aura-ingest` exposes + only M1 streaming, and `Aura.toml` (which the glossary says carries + "instrument/pip metadata") does not exist (open architectural thread). So the + researcher hard-codes pip_size per symbol from outside knowledge (GER40=1.0, + FRA40=1.0, EURUSD=0.0001). The GER40-vs-FRA40 comparison is honest only because + both happen to be index points and the researcher knew to pass 1.0 for each; + nothing flags or prevents comparing GER40 index-pips to EURUSD fx-pips. +- Why spec_gap: C10/C20 lean on "per-instrument pip metadata (reference data + beside the hot path)" for the sim-optimal broker's currency-free yardstick, but + the public surface offers no place to register or look up that metadata, so the + yardstick's cross-asset honesty is entirely on the researcher. The ledger names + the metadata; the surface does not realize it. (Acknowledged as an open thread, + so this is a confirmation that the gap bites at the first real comparison.) +- Recommended action: `ratify` / `planner` — sequence the `Aura.toml` + instrument/pip schema (or a minimal pip-metadata source) so cross-asset + comparison can be honest by construction, not by researcher discipline. + +### [working] Determinism, the streaming source, and the sweep/WFO cores are solid +- Example(s): t1_scale, t1c, t2, t3, t4. +- What happened: every long-horizon run was bit-identical on rerun (C1 held over + 7 years / 148k bars). The lazy `M1FieldSource` streamed a 7-year, 4-field, 8.6M- + record window in 1.12s with no errors. The `Composite` re-authoring reproduced + the shipped FlatGraph's numbers EXACTLY (`entry=3,exit=5` → −45.0; 2024 full + year → +203.4 / 62 sessions, identical in t1 and t4) — strong evidence the + bootstrap/compile and `bootstrap_with_cells` paths are behaviour-preserving + (C23). `sweep` parallelized the 9-point grid and `walk_forward` produced 24 + windows + a correctly stitched, timestamp-monotone OOS curve. The named-param + surface (`param_space`, `named_params`, `GridSpace::new` validating axes against + the space) is clean and self-describing where the strategy IS a Composite. +- Why working: these are the load-bearing guarantees the whole World rests on, and + they held under real multi-year load on the first or second try. Worth recording + so they are protected from drift. +- Recommended action: `carry-on`. + +### [friction] ns↔ms conversion is one-directional; residency probe not on the trait +- Example(s): t2_walkforward, t1b_residency. +- What happened: (a) `WindowBounds`/`RunManifest.window` are epoch-ns but + `M1FieldSource::open` wants Unix-ms; `aura-ingest` exports `unix_ms_to_epoch_ns` + but NOT its inverse, so every real-data walk-forward hand-divides `ts.0 / + 1_000_000` (the milestone `mw_3` fixture hit the identical seam). (b) + `resident_records()` is an inherent method on the concrete `M1FieldSource`, not + on the public `Source` trait, so a consumer driving `Box` (as `run` + does) cannot read residency — I had to drop to the concrete type to probe it. +- Why friction: both tasks completed, but the surface forced a verbose dance the + ledger's "producer-supplied window" / "residency is O(one chunk)" framing implied + would be cleaner. +- Recommended action: `plan` — add `epoch_ns_to_unix_ms` to `aura-ingest` (mirror + of the existing forward fn), and consider surfacing residency on the `Source` + trait (or document that it is concrete-only). + +## Recommendation summary + +| Finding | Action | +|---|---| +| spec_gap — breakout ships as FlatGraph, not a World-consumable Composite | `planner` (ship a Composite blueprint) | +| bug — process residency scales with window despite O(one-chunk) claim | `debug` | +| spec_gap — walk_forward empty-space idiom undefined for param-free strategy | `ratify` | +| bug — Delay.lag / Resample.period_minutes leak into default param_space | `debug` / `planner` | +| bug — Resample/Session period coupling unenforceable, desyncs silently to 0 pips | `debug` / `planner` | +| spec_gap — no per-instrument pip/metadata registry; cross-asset honesty manual | `ratify` / `planner` | +| working — C1 determinism, streaming source, sweep/WFO cores, re-author fidelity | `carry-on` | +| friction — one-directional ns/ms conversion; residency not on Source trait | `plan` | diff --git a/fieldtests/research-breakout-deepdive/Cargo.lock b/fieldtests/research-breakout-deepdive/Cargo.lock new file mode 100644 index 0000000..38d8f15 --- /dev/null +++ b/fieldtests/research-breakout-deepdive/Cargo.lock @@ -0,0 +1,1023 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "adler2" +version = "2.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "320119579fcad9c21884f5c4861d16174d0e06250625266f50fe6898340abefa" + +[[package]] +name = "aes" +version = "0.8.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b169f7a6d4742236a0a00c541b845991d0ac43e546831af1249753ab4c3aa3a0" +dependencies = [ + "cfg-if", + "cipher", + "cpufeatures", +] + +[[package]] +name = "aho-corasick" +version = "1.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ddd31a130427c27518df266943a5308ed92d4b226cc639f5a8f1002816174301" +dependencies = [ + "memchr", +] + +[[package]] +name = "android_system_properties" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "819e7219dbd41043ac279b19830f2efc897156490d7fd6ea916720117ee66311" +dependencies = [ + "libc", +] + +[[package]] +name = "arbitrary" +version = "1.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c3d036a3c4ab069c7b410a2ce876bd74808d2d0888a82667669f8e783a898bf1" +dependencies = [ + "derive_arbitrary", +] + +[[package]] +name = "aura-core" +version = "0.1.0" +dependencies = [ + "serde", +] + +[[package]] +name = "aura-engine" +version = "0.1.0" +dependencies = [ + "aura-core", + "serde", + "serde_json", +] + +[[package]] +name = "aura-ingest" +version = "0.1.0" +dependencies = [ + "aura-core", + "aura-engine", + "data-server", +] + +[[package]] +name = "aura-registry" +version = "0.1.0" +dependencies = [ + "aura-engine", + "serde", + "serde_json", +] + +[[package]] +name = "aura-std" +version = "0.1.0" +dependencies = [ + "aura-core", + "chrono", + "chrono-tz", +] + +[[package]] +name = "autocfg" +version = "1.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53" + +[[package]] +name = "block-buffer" +version = "0.10.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3078c7629b62d3f0439517fa394996acacc5cbc91c5a20d8c658e77abd503a71" +dependencies = [ + "generic-array", +] + +[[package]] +name = "bumpalo" +version = "3.20.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72f5acc6cb2ba439de613abc23857ec3d78374d8ed5ac84e9d11336e87da8649" + +[[package]] +name = "byteorder" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fd0f2584146f6f2ef48085050886acf353beff7305ebd1ae69500e27c67f64b" + +[[package]] +name = "bzip2" +version = "0.5.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "49ecfb22d906f800d4fe833b6282cf4dc1c298f5057ca0b5445e5c209735ca47" +dependencies = [ + "bzip2-sys", +] + +[[package]] +name = "bzip2-sys" +version = "0.1.13+1.0.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "225bff33b2141874fe80d71e07d6eec4f85c5c216453dd96388240f96e1acc14" +dependencies = [ + "cc", + "pkg-config", +] + +[[package]] +name = "cc" +version = "1.2.64" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dad887fd958be91b5098c0248def011f4523ab786cd411be668777e55063501f" +dependencies = [ + "find-msvc-tools", + "jobserver", + "libc", + "shlex", +] + +[[package]] +name = "cfg-if" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" + +[[package]] +name = "chrono" +version = "0.4.45" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1aa79e62e7697b8e29b513a68abacf485adcd1fe8284a4316c5ae868e6633327" +dependencies = [ + "iana-time-zone", + "js-sys", + "num-traits", + "wasm-bindgen", + "windows-link", +] + +[[package]] +name = "chrono-tz" +version = "0.10.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a6139a8597ed92cf816dfb33f5dd6cf0bb93a6adc938f11039f371bc5bcd26c3" +dependencies = [ + "chrono", + "phf", +] + +[[package]] +name = "cipher" +version = "0.4.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "773f3b9af64447d2ce9850330c473515014aa235e6a783b02db81ff39e4a3dad" +dependencies = [ + "crypto-common", + "inout", +] + +[[package]] +name = "constant_time_eq" +version = "0.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7c74b8349d32d297c9134b8c88677813a227df8f779daa29bfc29c183fe3dca6" + +[[package]] +name = "core-foundation-sys" +version = "0.8.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b" + +[[package]] +name = "cpufeatures" +version = "0.2.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "59ed5838eebb26a2bb2e58f6d5b5316989ae9d08bab10e0e6d103e656d1b0280" +dependencies = [ + "libc", +] + +[[package]] +name = "crc" +version = "3.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5eb8a2a1cd12ab0d987a5d5e825195d372001a4094a0376319d5a0ad71c1ba0d" +dependencies = [ + "crc-catalog", +] + +[[package]] +name = "crc-catalog" +version = "2.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "217698eaf96b4a3f0bc4f3662aaa55bdf913cd54d7204591faa790070c6d0853" + +[[package]] +name = "crc32fast" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9481c1c90cbf2ac953f07c8d4a58aa3945c425b7185c9154d67a65e4230da511" +dependencies = [ + "cfg-if", +] + +[[package]] +name = "crossbeam-utils" +version = "0.8.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d0a5c400df2834b80a4c3327b3aad3a4c4cd4de0629063962b03235697506a28" + +[[package]] +name = "crypto-common" +version = "0.1.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "78c8292055d1c1df0cce5d180393dc8cce0abec0a7102adb6c7b1eef6016d60a" +dependencies = [ + "generic-array", + "typenum", +] + +[[package]] +name = "data-server" +version = "0.1.0" +source = "git+http://192.168.178.103:3000/Brummel/data-server.git?branch=main#0f5e6655b90eeab0f43a8b0f330b8d1632e80ed4" +dependencies = [ + "chrono", + "regex", + "zip", +] + +[[package]] +name = "deflate64" +version = "0.1.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ac6b926516df9c60bfa16e107b21086399f8285a44ca9711344b9e553c5146e2" + +[[package]] +name = "deranged" +version = "0.5.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7cd812cc2bc1d69d4764bd80df88b4317eaef9e773c75226407d9bc0876b211c" + +[[package]] +name = "derive_arbitrary" +version = "1.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1e567bd82dcff979e4b03460c307b3cdc9e96fde3d73bed1496d2bc75d9dd62a" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "digest" +version = "0.10.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292" +dependencies = [ + "block-buffer", + "crypto-common", + "subtle", +] + +[[package]] +name = "displaydoc" +version = "0.2.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1ac70aa55017e108007fbaf5aa0f54b021c98f92ff8af59d42eda9da96e3dd4f" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "equivalent" +version = "1.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f" + +[[package]] +name = "find-msvc-tools" +version = "0.1.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5baebc0774151f905a1a2cc41989300b1e6fbb29aff0ceffa1064fdd3088d582" + +[[package]] +name = "flate2" +version = "1.1.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "843fba2746e448b37e26a819579957415c8cef339bf08564fe8b7ddbd959573c" +dependencies = [ + "crc32fast", + "miniz_oxide", +] + +[[package]] +name = "futures-core" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7e3450815272ef58cec6d564423f6e755e25379b217b0bc688e295ba24df6b1d" + +[[package]] +name = "futures-task" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "037711b3d59c33004d3856fbdc83b99d4ff37a24768fa1be9ce3538a1cde4393" + +[[package]] +name = "futures-util" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "389ca41296e6190b48053de0321d02a77f32f8a5d2461dd38762c0593805c6d6" +dependencies = [ + "futures-core", + "futures-task", + "pin-project-lite", + "slab", +] + +[[package]] +name = "generic-array" +version = "0.14.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a" +dependencies = [ + "typenum", + "version_check", +] + +[[package]] +name = "getrandom" +version = "0.3.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "899def5c37c4fd7b2664648c28120ecec138e4d395b459e5ca34f9cce2dd77fd" +dependencies = [ + "cfg-if", + "js-sys", + "libc", + "r-efi", + "wasip2", + "wasm-bindgen", +] + +[[package]] +name = "hashbrown" +version = "0.17.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ed5909b6e89a2db4456e54cd5f673791d7eca6732202bbf2a9cc504fe2f9b84a" + +[[package]] +name = "hmac" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6c49c37c09c17a53d937dfbb742eb3a961d65a994e6bcdcf37e7399d0cc8ab5e" +dependencies = [ + "digest", +] + +[[package]] +name = "iana-time-zone" +version = "0.1.65" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e31bc9ad994ba00e440a8aa5c9ef0ec67d5cb5e5cb0cc7f8b744a35b389cc470" +dependencies = [ + "android_system_properties", + "core-foundation-sys", + "iana-time-zone-haiku", + "js-sys", + "log", + "wasm-bindgen", + "windows-core", +] + +[[package]] +name = "iana-time-zone-haiku" +version = "0.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f31827a206f56af32e590ba56d5d2d085f558508192593743f16b2306495269f" +dependencies = [ + "cc", +] + +[[package]] +name = "indexmap" +version = "2.14.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d466e9454f08e4a911e14806c24e16fba1b4c121d1ea474396f396069cf949d9" +dependencies = [ + "equivalent", + "hashbrown", +] + +[[package]] +name = "inout" +version = "0.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "879f10e63c20629ecabbb64a8010319738c66a5cd0c29b02d63d272b03751d01" +dependencies = [ + "generic-array", +] + +[[package]] +name = "itoa" +version = "1.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" + +[[package]] +name = "jobserver" +version = "0.1.34" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9afb3de4395d6b3e67a780b6de64b51c978ecf11cb9a462c66be7d4ca9039d33" +dependencies = [ + "getrandom", + "libc", +] + +[[package]] +name = "js-sys" +version = "0.3.102" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "03d04c30968dffe80775bd4d7fb676131cd04a1fb46d2686dbffbaec2d9dfd31" +dependencies = [ + "cfg-if", + "futures-util", + "wasm-bindgen", +] + +[[package]] +name = "libc" +version = "0.2.186" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "68ab91017fe16c622486840e4c83c9a37afeff978bd239b5293d61ece587de66" + +[[package]] +name = "log" +version = "0.4.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "953f07c43838f8e6f9758cab68bf5bed85465e7587ebe0b823f1bcd81978ad3a" + +[[package]] +name = "lzma-rs" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "297e814c836ae64db86b36cf2a557ba54368d03f6afcd7d947c266692f71115e" +dependencies = [ + "byteorder", + "crc", +] + +[[package]] +name = "lzma-sys" +version = "0.1.20" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5fda04ab3764e6cde78b9974eec4f779acaba7c4e84b36eca3cf77c581b85d27" +dependencies = [ + "cc", + "libc", + "pkg-config", +] + +[[package]] +name = "memchr" +version = "2.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "88904434abc2901f197fe8cc55f0445e7ded921dba5911dad2e2b39b48e663c4" + +[[package]] +name = "miniz_oxide" +version = "0.8.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fa76a2c86f704bdb222d66965fb3d63269ce38518b83cb0575fca855ebb6316" +dependencies = [ + "adler2", + "simd-adler32", +] + +[[package]] +name = "num-conv" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "521739c6d2bac4aa25192232afe6841231376b2b26d4d9fae5ecf8ca5772e441" + +[[package]] +name = "num-traits" +version = "0.2.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "071dfc062690e90b734c0b2273ce72ad0ffa95f0c74596bc250dcfd960262841" +dependencies = [ + "autocfg", +] + +[[package]] +name = "once_cell" +version = "1.21.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" + +[[package]] +name = "pbkdf2" +version = "0.12.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f8ed6a7761f76e3b9f92dfb0a60a6a6477c61024b775147ff0973a02653abaf2" +dependencies = [ + "digest", + "hmac", +] + +[[package]] +name = "phf" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "913273894cec178f401a31ec4b656318d95473527be05c0752cc41cdc32be8b7" +dependencies = [ + "phf_shared", +] + +[[package]] +name = "phf_shared" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "06005508882fb681fd97892ecff4b7fd0fee13ef1aa569f8695dae7ab9099981" +dependencies = [ + "siphasher", +] + +[[package]] +name = "pin-project-lite" +version = "0.2.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd" + +[[package]] +name = "pkg-config" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "19f132c84eca552bf34cab8ec81f1c1dcc229b811638f9d283dceabe58c5569e" + +[[package]] +name = "powerfmt" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "439ee305def115ba05938db6eb1644ff94165c5ab5e9420d1c1bcedbba909391" + +[[package]] +name = "proc-macro2" +version = "1.0.106" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8fd00f0bb2e90d81d1044c2b32617f68fcb9fa3bb7640c23e9c748e53fb30934" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "quote" +version = "1.0.45" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "41f2619966050689382d2b44f664f4bc593e129785a36d6ee376ddf37259b924" +dependencies = [ + "proc-macro2", +] + +[[package]] +name = "r-efi" +version = "5.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "69cdb34c158ceb288df11e18b4bd39de994f6657d83847bdffdbd7f346754b0f" + +[[package]] +name = "regex" +version = "1.12.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f1292b7759ae1cb9ec195452d1390a074f0cd8541ab7a5a8c31cd6db45d4a6ba" +dependencies = [ + "aho-corasick", + "memchr", + "regex-automata", + "regex-syntax", +] + +[[package]] +name = "regex-automata" +version = "0.4.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6e1dd4122fc1595e8162618945476892eefca7b88c52820e74af6262213cae8f" +dependencies = [ + "aho-corasick", + "memchr", + "regex-syntax", +] + +[[package]] +name = "regex-syntax" +version = "0.8.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d6f6ff9a378485b298a5286656da665ba74413d36db0979633275d2e708145d4" + +[[package]] +name = "research-breakout-deepdive" +version = "0.0.0" +dependencies = [ + "aura-core", + "aura-engine", + "aura-ingest", + "aura-registry", + "aura-std", + "chrono", + "chrono-tz", + "data-server", +] + +[[package]] +name = "rustversion" +version = "1.0.22" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b39cdef0fa800fc44525c84ccb54a029961a8215f9619753635a9c0d2538d46d" + +[[package]] +name = "serde" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9a8e94ea7f378bd32cbbd37198a4a91436180c5bb472411e48b5ec2e2124ae9e" +dependencies = [ + "serde_core", + "serde_derive", +] + +[[package]] +name = "serde_core" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "41d385c7d4ca58e59fc732af25c3983b67ac852c1a25000afe1175de458b67ad" +dependencies = [ + "serde_derive", +] + +[[package]] +name = "serde_derive" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d540f220d3187173da220f885ab66608367b6574e925011a9353e4badda91d79" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "serde_json" +version = "1.0.150" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e8014e44b4736ed0538adeecded0fce2a272f22dc9578a7eb6b2d9993c74cfb9" +dependencies = [ + "itoa", + "memchr", + "serde", + "serde_core", + "zmij", +] + +[[package]] +name = "sha1" +version = "0.10.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e3bf829a2d51ab4a5ddf1352d8470c140cadc8301b2ae1789db023f01cedd6ba" +dependencies = [ + "cfg-if", + "cpufeatures", + "digest", +] + +[[package]] +name = "shlex" +version = "2.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba" + +[[package]] +name = "simd-adler32" +version = "0.3.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "703d5c7ef118737c72f1af64ad2f6f8c5e1921f818cdcb97b8fe6fc69bf66214" + +[[package]] +name = "siphasher" +version = "1.0.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8ee5873ec9cce0195efcb7a4e9507a04cd49aec9c83d0389df45b1ef7ba2e649" + +[[package]] +name = "slab" +version = "0.4.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5" + +[[package]] +name = "subtle" +version = "2.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "13c2bddecc57b384dee18652358fb23172facb8a2c51ccc10d74c157bdea3292" + +[[package]] +name = "syn" +version = "2.0.118" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1b9ae57f904213ebb649ce6895b8a66c66f0203b9319718f69a5612a065b1422" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "thiserror" +version = "2.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4288b5bcbc7920c07a1149a35cf9590a2aa808e0bc1eafaade0b80947865fbc4" +dependencies = [ + "thiserror-impl", +] + +[[package]] +name = "thiserror-impl" +version = "2.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ebc4ee7f67670e9b64d05fa4253e753e016c6c95ff35b89b7941d6b856dec1d5" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "time" +version = "0.3.49" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "711a53c2d47bbd818258c498c8dbfe186a2526c631495cfe7e078567f86b8469" +dependencies = [ + "deranged", + "num-conv", + "powerfmt", + "serde_core", + "time-core", +] + +[[package]] +name = "time-core" +version = "0.1.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9e1c906769ad99c88eaa54e728060edef082f8e358ff32030cb7c7d315e81109" + +[[package]] +name = "typenum" +version = "1.20.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6f5e870be6c3b371b77fe0ee0bafb859fa4964b4404c27de1d380043c4dda20" + +[[package]] +name = "unicode-ident" +version = "1.0.24" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" + +[[package]] +name = "version_check" +version = "0.9.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a" + +[[package]] +name = "wasip2" +version = "1.0.4+wasi-0.2.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b67efb37e106e55ce722a510d6b5f9c17f083e5fc79afc2badeb12cc313d9487" +dependencies = [ + "wit-bindgen", +] + +[[package]] +name = "wasm-bindgen" +version = "0.2.125" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8ddb3f79143bced6de84270411622a2699cee572fc0875aeaf1e7867cf9fca1a" +dependencies = [ + "cfg-if", + "once_cell", + "rustversion", + "wasm-bindgen-macro", + "wasm-bindgen-shared", +] + +[[package]] +name = "wasm-bindgen-macro" +version = "0.2.125" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4e21a184b13fb19e157296e2c46056aec9092264fab83e4ba59e68c61b323c3d" +dependencies = [ + "quote", + "wasm-bindgen-macro-support", +] + +[[package]] +name = "wasm-bindgen-macro-support" +version = "0.2.125" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fecefd9c35bd935a20fc3fc344b5f29138961e4f47fb03297d88f2587afb5ebd" +dependencies = [ + "bumpalo", + "proc-macro2", + "quote", + "syn", + "wasm-bindgen-shared", +] + +[[package]] +name = "wasm-bindgen-shared" +version = "0.2.125" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "23939e44bb9a5d7576fa2b563dc2e136628f1224e88a8deed09e04858b77871f" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "windows-core" +version = "0.62.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b8e83a14d34d0623b51dce9581199302a221863196a1dde71a7663a4c2be9deb" +dependencies = [ + "windows-implement", + "windows-interface", + "windows-link", + "windows-result", + "windows-strings", +] + +[[package]] +name = "windows-implement" +version = "0.60.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "053e2e040ab57b9dc951b72c264860db7eb3b0200ba345b4e4c3b14f67855ddf" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "windows-interface" +version = "0.59.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3f316c4a2570ba26bbec722032c4099d8c8bc095efccdc15688708623367e358" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "windows-link" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" + +[[package]] +name = "windows-result" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7781fa89eaf60850ac3d2da7af8e5242a5ea78d1a11c49bf2910bb5a73853eb5" +dependencies = [ + "windows-link", +] + +[[package]] +name = "windows-strings" +version = "0.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7837d08f69c77cf6b07689544538e017c1bfcf57e34b4c0ff58e6c2cd3b37091" +dependencies = [ + "windows-link", +] + +[[package]] +name = "wit-bindgen" +version = "0.57.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1ebf944e87a7c253233ad6766e082e3cd714b5d03812acc24c318f549614536e" + +[[package]] +name = "xz2" +version = "0.1.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "388c44dc09d76f1536602ead6d325eb532f5c122f17782bd57fb47baeeb767e2" +dependencies = [ + "lzma-sys", +] + +[[package]] +name = "zeroize" +version = "1.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e13c156562582aa81c60cb29407084cdb54c4164760106ab78e6c5b0858cf64e" +dependencies = [ + "zeroize_derive", +] + +[[package]] +name = "zeroize_derive" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3c50655cbb0fe3fc43170059e702f1ce5e19b84cec58dc87b037a09935c2f328" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "zip" +version = "2.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fabe6324e908f85a1c52063ce7aa26b68dcb7eb6dbc83a2d148403c9bc3eba50" +dependencies = [ + "aes", + "arbitrary", + "bzip2", + "constant_time_eq", + "crc32fast", + "crossbeam-utils", + "deflate64", + "displaydoc", + "flate2", + "getrandom", + "hmac", + "indexmap", + "lzma-rs", + "memchr", + "pbkdf2", + "sha1", + "thiserror", + "time", + "xz2", + "zeroize", + "zopfli", + "zstd", +] + +[[package]] +name = "zmij" +version = "1.0.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b8848ee67ecc8aedbaf3e4122217aff892639231befc6a1b58d29fff4c2cabaa" + +[[package]] +name = "zopfli" +version = "0.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f05cd8797d63865425ff89b5c4a48804f35ba0ce8d125800027ad6017d2b5249" +dependencies = [ + "bumpalo", + "crc32fast", + "log", + "simd-adler32", +] + +[[package]] +name = "zstd" +version = "0.13.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e91ee311a569c327171651566e07972200e76fcfe2242a4fa446149a3881c08a" +dependencies = [ + "zstd-safe", +] + +[[package]] +name = "zstd-safe" +version = "7.2.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f49c4d5f0abb602a93fb8736af2a4f4dd9512e36f7f570d66e65ff867ed3b9d" +dependencies = [ + "zstd-sys", +] + +[[package]] +name = "zstd-sys" +version = "2.0.16+zstd.1.5.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "91e19ebc2adc8f83e43039e79776e3fda8ca919132d68a1fed6a5faca2683748" +dependencies = [ + "cc", + "pkg-config", +] diff --git a/fieldtests/research-breakout-deepdive/Cargo.toml b/fieldtests/research-breakout-deepdive/Cargo.toml new file mode 100644 index 0000000..438db62 --- /dev/null +++ b/fieldtests/research-breakout-deepdive/Cargo.toml @@ -0,0 +1,61 @@ +# Downstream-consumer fieldtest crate: a quant researcher taking the SHIPPED +# GER40 session-breakout (crates/aura-ingest/examples/ger40_breakout_real.rs) +# through an escalating sequence of real research tasks — scale / long-horizon, +# walk-forward, parameter sweep, multi-instrument comparison. +# +# NOT a member of the aura workspace: it path-deps the engine crates exactly as +# a real C16 research project would, and drives the PUBLIC surface only (design +# ledger + glossary + the example corpus + cargo doc). No crates/*/src was read +# to author these. +# +# Built/run via, e.g.: +# cargo run --release --manifest-path fieldtests/research-breakout-deepdive/Cargo.toml --bin t1_scale +[workspace] + +[package] +name = "research-breakout-deepdive" +version = "0.0.0" +edition = "2024" +publish = false + +[dependencies] +aura-core = { path = "../../crates/aura-core" } +aura-engine = { path = "../../crates/aura-engine" } +aura-registry = { path = "../../crates/aura-registry" } +aura-ingest = { path = "../../crates/aura-ingest" } +aura-std = { path = "../../crates/aura-std" } +# M1FieldSource::open wants &Arc, and aura-ingest re-exports neither +# DataServer nor DEFAULT_DATA_PATH — a real consumer must pull the same external +# git dep aura-ingest uses. +data-server = { git = "http://192.168.178.103:3000/Brummel/data-server.git", branch = "main" } +# Session timezone + UTC window math — same dev-deps the shipped example uses. +chrono = { version = "0.4", default-features = false, features = ["clock"] } +chrono-tz = { version = "0.10", default-features = false } + +[[bin]] +name = "t1_scale" +path = "t1_scale.rs" + +[[bin]] +name = "t1b_residency" +path = "t1b_residency.rs" + +[[bin]] +name = "t2_walkforward" +path = "t2_walkforward.rs" + +[[bin]] +name = "t3_sweep" +path = "t3_sweep.rs" + +[[bin]] +name = "t4_compare" +path = "t4_compare.rs" + +[[bin]] +name = "t1c_residency_scaling" +path = "t1c_residency_scaling.rs" + +[[bin]] +name = "t3b_period_desync" +path = "t3b_period_desync.rs" diff --git a/fieldtests/research-breakout-deepdive/out_t1_scale.txt b/fieldtests/research-breakout-deepdive/out_t1_scale.txt new file mode 100644 index 0000000..f768550 --- /dev/null +++ b/fieldtests/research-breakout-deepdive/out_t1_scale.txt @@ -0,0 +1,10 @@ +=== Task 1: scale the GER40 breakout over long horizons === +startup RSS = 6 MiB + +2024-09 (baseline) bars=1679 sessions=5 total_pips= -45.0 max_dd= 77.0 flips=10 | 0.01s RSS 7->8 MiB +2024 full year bars=21377 sessions=62 total_pips= 203.4 max_dd= 168.5 flips=124 | 0.17s RSS 9->18 MiB +2018..2024 (7y) bars=148688 sessions=401 total_pips= 520.2 max_dd= 799.1 flips=802 | 1.12s RSS 8->127 MiB + +--- determinism (C1): rerun 2024 full year, compare report JSON --- +rerun report JSON bit-identical: true + total_pips r1=203.4 r2=203.4 diff --git a/fieldtests/research-breakout-deepdive/out_t1b_residency.txt b/fieldtests/research-breakout-deepdive/out_t1b_residency.txt new file mode 100644 index 0000000..a29ef77 --- /dev/null +++ b/fieldtests/research-breakout-deepdive/out_t1b_residency.txt @@ -0,0 +1,12 @@ +=== Task 1 deep dive: residency attribution (7-year GER40 window) === +VmHWM at start = 6 MiB + +(B) 4 raw sources, NO harness/sinks: + streamed 8634128 records, peak resident_records/source = 1024 + VmHWM after = 125 MiB + +(A) full breakout harness + 4 recorder taps + drain_trace: + trace rows = 148688 + VmHWM after = 210 MiB + +VmHWM final = 210 MiB diff --git a/fieldtests/research-breakout-deepdive/out_t1c_residency_scaling.txt b/fieldtests/research-breakout-deepdive/out_t1c_residency_scaling.txt new file mode 100644 index 0000000..104b0a8 --- /dev/null +++ b/fieldtests/research-breakout-deepdive/out_t1c_residency_scaling.txt @@ -0,0 +1,10 @@ +=== Task 1 deep dive: does VmHWM scale with window length? === +(the docs promise it does NOT — O(one chunk) residency) + +baseline VmHWM = 4 MiB + +1 month records=24466 aura-resident<=1024? true | VmHWM=6 MiB +6 months records=159156 aura-resident<=1024? true | VmHWM=15 MiB +1 year records=305878 aura-resident<=1024? true | VmHWM=24 MiB +4 years records=1295897 aura-resident<=1024? true | VmHWM=78 MiB +~10 years records=3064025 aura-resident<=1024? true | VmHWM=173 MiB diff --git a/fieldtests/research-breakout-deepdive/out_t2_walkforward.txt b/fieldtests/research-breakout-deepdive/out_t2_walkforward.txt new file mode 100644 index 0000000..3276ab9 --- /dev/null +++ b/fieldtests/research-breakout-deepdive/out_t2_walkforward.txt @@ -0,0 +1,32 @@ +=== Task 2: walk-forward the shipped GER40 breakout (2018..2024) === + +walk-forward produced 24 windows + + window 0: is=[1514764800000..1545868799999] oos=[1545868800000..1553644799999] oos_pips= 182.4 oos_dd= 56.0 + window 1: is=[1522540800000..1553644799999] oos=[1553644800000..1561420799999] oos_pips= 66.1 oos_dd= 36.8 + window 2: is=[1530316800000..1561420799999] oos=[1561420800000..1569196799999] oos_pips= 94.2 oos_dd= 31.2 + window 3: is=[1538092800000..1569196799999] oos=[1569196800000..1576972799999] oos_pips= 33.4 oos_dd= 72.2 + window 4: is=[1545868800000..1576972799999] oos=[1576972800000..1584748799999] oos_pips= -37.5 oos_dd= 164.7 + window 5: is=[1553644800000..1584748799999] oos=[1584748800000..1592524799999] oos_pips= -35.4 oos_dd= 84.8 + window 6: is=[1561420800000..1592524799999] oos=[1592524800000..1600300799999] oos_pips= -237.1 oos_dd= 424.8 + window 7: is=[1569196800000..1600300799999] oos=[1600300800000..1608076799999] oos_pips= 137.2 oos_dd= 61.0 + window 8: is=[1576972800000..1608076799999] oos=[1608076800000..1615852799999] oos_pips= -203.0 oos_dd= 222.4 + window 9: is=[1584748800000..1615852799999] oos=[1615852800000..1623628799999] oos_pips= -217.8 oos_dd= 228.3 + window 10: is=[1592524800000..1623628799999] oos=[1623628800000..1631404799999] oos_pips= -17.2 oos_dd= 103.0 + window 11: is=[1600300800000..1631404799999] oos=[1631404800000..1639180799999] oos_pips= 30.8 oos_dd= 65.2 + window 12: is=[1608076800000..1639180799999] oos=[1639180800000..1646956799999] oos_pips= 39.5 oos_dd= 93.7 + window 13: is=[1615852800000..1646956799999] oos=[1646956800000..1654732799999] oos_pips= 53.9 oos_dd= 64.9 + window 14: is=[1623628800000..1654732799999] oos=[1654732800000..1662508799999] oos_pips= 156.2 oos_dd= 32.1 + window 15: is=[1631404800000..1662508799999] oos=[1662508800000..1670284799999] oos_pips= 87.1 oos_dd= 155.7 + window 16: is=[1639180800000..1670284799999] oos=[1670284800000..1678060799999] oos_pips= -19.8 oos_dd= 113.1 + window 17: is=[1646956800000..1678060799999] oos=[1678060800000..1685836799999] oos_pips= 127.8 oos_dd= 57.8 + window 18: is=[1654732800000..1685836799999] oos=[1685836800000..1693612799999] oos_pips= 156.8 oos_dd= 71.3 + window 19: is=[1662508800000..1693612799999] oos=[1693612800000..1701388799999] oos_pips= -15.6 oos_dd= 119.1 + window 20: is=[1670284800000..1701388799999] oos=[1701388800000..1709164799999] oos_pips= -64.5 oos_dd= 116.7 + window 21: is=[1678060800000..1709164799999] oos=[1709164800000..1716940799999] oos_pips= 66.2 oos_dd= 34.3 + window 22: is=[1685836800000..1716940799999] oos=[1716940800000..1724716799999] oos_pips= 240.1 oos_dd= 54.5 + window 23: is=[1693612800000..1724716799999] oos=[1724716800000..1732492799999] oos_pips= -5.0 oos_dd= 153.5 + +stitched OOS curve: 131508 points, final cumulative = 618.8 pips + +OK: fixed-param OOS walk-forward over the shipped breakout FlatGraph. diff --git a/fieldtests/research-breakout-deepdive/out_t3_sweep.txt b/fieldtests/research-breakout-deepdive/out_t3_sweep.txt new file mode 100644 index 0000000..04faf1d --- /dev/null +++ b/fieldtests/research-breakout-deepdive/out_t3_sweep.txt @@ -0,0 +1,25 @@ +=== Task 3: sweep the breakout's structural params === + +param_space() of the breakout Composite: + resample.period_minutes : I64 + delay.lag : I64 + entry_bar.target : I64 + exit_bar.target : I64 + +coupling check: + resample.period_minutes in space? true + any session.* period in space? false + => the bar-period coupling is NOT expressible as one swept param. + +sweeping entry_bar x exit_bar (period pinned 15m) over GER40 2024-09: + { delay.lag=1, entry_bar=2, exit_bar=5 } total_pips= -80.0 max_dd= 115.0 + { delay.lag=1, entry_bar=2, exit_bar=6 } total_pips= -112.5 max_dd= 122.0 + { delay.lag=1, entry_bar=2, exit_bar=8 } total_pips= -102.5 max_dd= 110.5 + { delay.lag=1, entry_bar=3, exit_bar=5 } total_pips= -45.0 max_dd= 77.0 + { delay.lag=1, entry_bar=3, exit_bar=6 } total_pips= -43.0 max_dd= 93.0 + { delay.lag=1, entry_bar=3, exit_bar=8 } total_pips= -118.5 max_dd= 161.0 + { delay.lag=1, entry_bar=4, exit_bar=5 } total_pips= -3.0 max_dd= 33.0 + { delay.lag=1, entry_bar=4, exit_bar=6 } total_pips= 43.0 max_dd= 63.5 + { delay.lag=1, entry_bar=4, exit_bar=8 } total_pips= -75.0 max_dd= 165.5 + +OK: entry/exit-bar grid sweep works; bar-period coupling does not. diff --git a/fieldtests/research-breakout-deepdive/out_t3b_period_desync.txt b/fieldtests/research-breakout-deepdive/out_t3b_period_desync.txt new file mode 100644 index 0000000..2506b1c --- /dev/null +++ b/fieldtests/research-breakout-deepdive/out_t3b_period_desync.txt @@ -0,0 +1,10 @@ +=== Task 3 deep dive: the silent period desync === +entry_bar=3, exit_bar=5 fixed; Session period BAKED at 15m. + +resample=15m, session=15m (COUPLED, correct): bars=1679 total_pips=-45.0 +resample=30m, session=15m (DESYNCED, silent): bars=839 total_pips=0.0 +resample=30m, session=30m (REBUILT blueprint): bars=839 total_pips=63.0 + +Observation: all three bootstrap+run with NO error. The desynced +middle run is silently wrong — the param system cannot couple the two +periods, and only a blueprint rebuild (not a sweep) gives a coherent 30m. diff --git a/fieldtests/research-breakout-deepdive/out_t4_compare.txt b/fieldtests/research-breakout-deepdive/out_t4_compare.txt new file mode 100644 index 0000000..3b9d4b1 --- /dev/null +++ b/fieldtests/research-breakout-deepdive/out_t4_compare.txt @@ -0,0 +1,13 @@ +=== Task 4: multi-instrument comparison (same breakout, GER40 vs FRA40) === + +symbol pip-meaning sessions total_pips max_dd flips +-------------------------------------------------------------- +GER40 index point 62 203.4 168.5 124 +FRA40 index point 63 83.7 62.7 126 + +Honesty of the cross-asset comparison: + Both GER40 and FRA40 are index points (pip=1.0), so their pip totals + ARE directly comparable here — BUT only because the researcher knew + to pass pip=1.0 for both. The engine has NO instrument registry: pip + is a bare f64 to SimBroker::new, and nothing would stop (or flag) a + comparison of GER40 index-pips to EURUSD fx-pips with mismatched scale. diff --git a/fieldtests/research-breakout-deepdive/t1_scale.rs b/fieldtests/research-breakout-deepdive/t1_scale.rs new file mode 100644 index 0000000..a9c041d --- /dev/null +++ b/fieldtests/research-breakout-deepdive/t1_scale.rs @@ -0,0 +1,128 @@ +// Task 1 — Scale / long horizon. +// +// The downstream researcher's task: "the shipped GER40 breakout demo runs ONE +// month. I want to know if the strategy holds up over a full year (2024) and a +// multi-year span (2018-01 .. 2024-12). Does it stay deterministic? Does memory +// stay bounded (the docs claim streaming residency is O(one chunk) — does opening +// 4 independent OHLC sources break that)? How slow is it?" +// +// PUBLIC INTERFACE ONLY. I start from the shipped example's shared wiring module +// (crates/aura-ingest/examples/shared/breakout_real.rs), included by #[path] — +// exactly as the shipped example and its test do — so I re-author NOTHING about +// the proven 13-node FlatGraph; I only change the WINDOW (a month -> a year -> +// 7 years) and add a self-RSS probe + a wall-clock timer. No crates/*/src read. + +use std::sync::Arc; +use std::time::Instant; + +use chrono::TimeZone; +use data_server::{DataServer, DEFAULT_DATA_PATH}; + +// Reuse the SHIPPED breakout wiring verbatim, the same way the shipped example +// does. The researcher does not re-type 13 nodes — the example exports them. +#[path = "../../crates/aura-ingest/examples/shared/breakout_real.rs"] +mod breakout_real; +use breakout_real::*; + +/// Read this process's resident set size (KiB) from /proc/self/statm * page size. +/// A coarse but honest peak-RSS probe — no allocator hooks, just the OS view. +fn rss_kib() -> u64 { + let statm = std::fs::read_to_string("/proc/self/statm").unwrap_or_default(); + let resident_pages: u64 = statm + .split_whitespace() + .nth(1) + .and_then(|s| s.parse().ok()) + .unwrap_or(0); + resident_pages * 4 // page size 4 KiB on this box +} + +/// Build the inclusive Unix-ms window spanning whole calendar months +/// [from (y0,m0) .. through end of (y1,m1)] in UTC. +fn utc_span_ms(y0: i32, m0: u32, y1: i32, m1: u32) -> (i64, i64) { + let from = chrono::Utc.with_ymd_and_hms(y0, m0, 1, 0, 0, 0).unwrap().timestamp_millis(); + let (ny, nm) = if m1 == 12 { (y1 + 1, 1) } else { (y1, m1 + 1) }; + let next = chrono::Utc.with_ymd_and_hms(ny, nm, 1, 0, 0, 0).unwrap().timestamp_millis(); + (from, next - 1) +} + +fn run_span(server: &Arc, label: &str, from_ms: i64, to_ms: i64) { + let Some(sources) = open_ohlc_sources(server, from_ms, to_ms) else { + println!("{label}: no overlapping {SYMBOL} files — skip"); + return; + }; + + let peak_before = rss_kib(); + let t0 = Instant::now(); + + let (mut h, taps) = build_harness(); + h.run(sources); + let trace = drain_trace(&taps); + let report = report_from_trace(&trace, from_ms, to_ms); + + let elapsed = t0.elapsed(); + let peak_after = rss_kib(); + + // count sessions (0->1 transitions) + let mut entries = 0u32; + let mut prev = 0.0; + for b in &trace { + if b.held == 1.0 && prev == 0.0 { + entries += 1; + } + prev = b.held; + } + + let m = &report.metrics; + println!( + "{label:<22} bars={:<7} sessions={:<4} total_pips={:>9.1} max_dd={:>8.1} flips={:<4} \ + | {:>6.2}s RSS {}->{} MiB", + trace.len(), + entries, + m.total_pips, + m.max_drawdown, + m.exposure_sign_flips, + elapsed.as_secs_f64(), + peak_before / 1024, + peak_after / 1024, + ); +} + +fn main() { + let server = Arc::new(DataServer::new(DEFAULT_DATA_PATH)); + if !server.has_symbol(SYMBOL) { + println!("skip: no local {SYMBOL} data at {DEFAULT_DATA_PATH}"); + return; + } + + println!("=== Task 1: scale the GER40 breakout over long horizons ==="); + println!("startup RSS = {} MiB\n", rss_kib() / 1024); + + // 1. The shipped one-month baseline, for reference. + let (f, t) = utc_span_ms(2024, 9, 2024, 9); + run_span(&server, "2024-09 (baseline)", f, t); + + // 2. A full year of GER40 (2024). + let (f, t) = utc_span_ms(2024, 1, 2024, 12); + run_span(&server, "2024 full year", f, t); + + // 3. A multi-year span: 2018-01 .. 2024-12 (7 years of M1 across 4 sources). + let (f, t) = utc_span_ms(2018, 1, 2024, 12); + run_span(&server, "2018..2024 (7y)", f, t); + + // 4. Determinism: rerun the full-year span and assert bit-identical report. + println!("\n--- determinism (C1): rerun 2024 full year, compare report JSON ---"); + let (f, t) = utc_span_ms(2024, 1, 2024, 12); + let rep = |srv: &Arc| { + let sources = open_ohlc_sources(srv, f, t).expect("year sources"); + let (mut h, taps) = build_harness(); + h.run(sources); + let trace = drain_trace(&taps); + report_from_trace(&trace, f, t) + }; + let r1 = rep(&server); + let r2 = rep(&server); + let j1 = r1.to_json(); + let j2 = r2.to_json(); + println!("rerun report JSON bit-identical: {}", j1 == j2); + println!(" total_pips r1={:.1} r2={:.1}", r1.metrics.total_pips, r2.metrics.total_pips); +} diff --git a/fieldtests/research-breakout-deepdive/t1b_residency.rs b/fieldtests/research-breakout-deepdive/t1b_residency.rs new file mode 100644 index 0000000..b34b877 --- /dev/null +++ b/fieldtests/research-breakout-deepdive/t1b_residency.rs @@ -0,0 +1,95 @@ +// Task 1 (deep dive) — WHERE does the memory go? The docs (C12 realization +// cycle 0041; the streaming_seam.rs residency test) claim a M1FieldSource keeps +// O(one chunk) resident, "independent of window length". Yet the 7-year breakout +// run grew RSS to ~127 MiB. Is that the SOURCES (which the docs promise is +// bounded), or the recorder SINKS accumulating every bar into unbounded mpsc +// channels + the drain_trace Vec? +// +// This probe isolates the two by driving the SAME sources two ways over the SAME +// 7-year window: +// (A) full breakout harness with all 4 recorder taps + drain_trace (what the +// shipped example does), then +// (B) the four M1FieldSource streams driven to exhaustion with NO harness, NO +// sinks — peek/next only, the exact loop streaming_seam.rs calls bounded. +// VmHWM (peak RSS, /proc/self/status) is read after each. PUBLIC SURFACE ONLY: +// Source::{peek,next,resident_records} are the public trait methods the docs use. + +use std::sync::Arc; + +use chrono::TimeZone; +use data_server::{DataServer, DEFAULT_DATA_PATH}; + +use aura_engine::Source; +use aura_ingest::{M1Field, M1FieldSource}; + +#[path = "../../crates/aura-ingest/examples/shared/breakout_real.rs"] +mod breakout_real; +use breakout_real::*; + +fn vmhwm_mib() -> u64 { + let status = std::fs::read_to_string("/proc/self/status").unwrap_or_default(); + for line in status.lines() { + if let Some(rest) = line.strip_prefix("VmHWM:") { + let kib: u64 = rest.split_whitespace().next().and_then(|s| s.parse().ok()).unwrap_or(0); + return kib / 1024; + } + } + 0 +} + +fn utc_span_ms(y0: i32, m0: u32, y1: i32, m1: u32) -> (i64, i64) { + let from = chrono::Utc.with_ymd_and_hms(y0, m0, 1, 0, 0, 0).unwrap().timestamp_millis(); + let (ny, nm) = if m1 == 12 { (y1 + 1, 1) } else { (y1, m1 + 1) }; + let next = chrono::Utc.with_ymd_and_hms(ny, nm, 1, 0, 0, 0).unwrap().timestamp_millis(); + (from, next - 1) +} + +fn main() { + let server = Arc::new(DataServer::new(DEFAULT_DATA_PATH)); + if !server.has_symbol(SYMBOL) { + println!("skip: no local {SYMBOL} data"); + return; + } + let (from_ms, to_ms) = utc_span_ms(2018, 1, 2024, 12); + + println!("=== Task 1 deep dive: residency attribution (7-year GER40 window) ==="); + println!("VmHWM at start = {} MiB\n", vmhwm_mib()); + + // (B) FIRST (so its peak is measured before the harness allocates): drive the + // four field sources to exhaustion with no harness and no sinks. Per the docs + // each keeps O(one chunk) resident — peak resident_records should stay tiny. + { + // NOTE: resident_records() is an INHERENT method on the concrete + // M1FieldSource, NOT on the public Source trait (the streaming_seam test + // calls it on a concrete source). A consumer driving boxed `dyn Source`s + // therefore cannot read residency — recorded as a finding. We use the + // concrete type here, per field, exactly as the docs' residency test does. + let fields = [M1Field::Open, M1Field::High, M1Field::Low, M1Field::Close]; + let mut peak_resident = 0usize; + let mut total = 0u64; + for f in fields { + let mut s = M1FieldSource::open(&server, SYMBOL, Some(from_ms), Some(to_ms), f).unwrap(); + while s.peek().is_some() { + peak_resident = peak_resident.max(s.resident_records()); + Source::next(&mut s); + total += 1; + } + } + println!("(B) 4 raw sources, NO harness/sinks:"); + println!(" streamed {total} records, peak resident_records/source = {peak_resident}"); + println!(" VmHWM after = {} MiB", vmhwm_mib()); + } + + // (A) the full shipped breakout harness with 4 recorder taps + drain_trace. + { + let sources = open_ohlc_sources(&server, from_ms, to_ms).unwrap(); + let (mut h, taps) = build_harness(); + h.run(sources); + let trace = drain_trace(&taps); + println!("\n(A) full breakout harness + 4 recorder taps + drain_trace:"); + println!(" trace rows = {}", trace.len()); + println!(" VmHWM after = {} MiB", vmhwm_mib()); + } + + println!("\nVmHWM final = {} MiB", vmhwm_mib()); +} diff --git a/fieldtests/research-breakout-deepdive/t1c_residency_scaling.rs b/fieldtests/research-breakout-deepdive/t1c_residency_scaling.rs new file mode 100644 index 0000000..828178b --- /dev/null +++ b/fieldtests/research-breakout-deepdive/t1c_residency_scaling.rs @@ -0,0 +1,78 @@ +// Task 1 (deep dive 2) — the smoking gun: does PROCESS residency scale with +// window length, contradicting the docs' "a 20-year window streams in the same +// memory as a one-day one" (C12 cycle-0041 realization)? +// +// Drives ONE M1FieldSource (Close) to exhaustion over increasingly long windows, +// reading VmHWM after each. The aura-level resident_records stays <= CHUNK_SIZE +// (the docs' asserted predicate) the whole time; the question is whether VmHWM +// (the OS peak RSS) is flat or grows with the window. Fresh DataServer per window +// so a cache inside it cannot mask the effect. PUBLIC SURFACE ONLY. + +use std::sync::Arc; + +use chrono::TimeZone; +use data_server::{DataServer, DEFAULT_DATA_PATH}; + +use aura_engine::Source; +use aura_ingest::{M1Field, M1FieldSource}; + +const SYMBOL: &str = "GER40"; + +fn vmhwm_mib() -> u64 { + let status = std::fs::read_to_string("/proc/self/status").unwrap_or_default(); + for line in status.lines() { + if let Some(rest) = line.strip_prefix("VmHWM:") { + let kib: u64 = rest.split_whitespace().next().and_then(|s| s.parse().ok()).unwrap_or(0); + return kib / 1024; + } + } + 0 +} + +fn ms(y0: i32, m0: u32, y1: i32, m1: u32) -> (i64, i64) { + let from = chrono::Utc.with_ymd_and_hms(y0, m0, 1, 0, 0, 0).unwrap().timestamp_millis(); + let (ny, nm) = if m1 == 12 { (y1 + 1, 1) } else { (y1, m1 + 1) }; + let next = chrono::Utc.with_ymd_and_hms(ny, nm, 1, 0, 0, 0).unwrap().timestamp_millis(); + (from, next - 1) +} + +fn drive(label: &str, from_ms: i64, to_ms: i64) { + // Fresh server each time: no shared chunk cache carrying over. + let server = Arc::new(DataServer::new(DEFAULT_DATA_PATH)); + let mut s = M1FieldSource::open(&server, SYMBOL, Some(from_ms), Some(to_ms), M1Field::Close) + .expect("window overlaps data"); + let mut total = 0u64; + let mut peak_resident = 0usize; + while s.peek().is_some() { + peak_resident = peak_resident.max(s.resident_records()); + Source::next(&mut s); + total += 1; + } + println!( + "{label:<16} records={:<9} aura-resident<=1024? {:<5} | VmHWM={} MiB", + total, + peak_resident <= 1024, + vmhwm_mib(), + ); +} + +fn main() { + if !Arc::new(DataServer::new(DEFAULT_DATA_PATH)).has_symbol(SYMBOL) { + println!("skip: no {SYMBOL} data"); + return; + } + println!("=== Task 1 deep dive: does VmHWM scale with window length? ==="); + println!("(the docs promise it does NOT — O(one chunk) residency)\n"); + println!("baseline VmHWM = {} MiB\n", vmhwm_mib()); + + let (f, t) = ms(2024, 9, 2024, 9); + drive("1 month", f, t); + let (f, t) = ms(2024, 1, 2024, 6); + drive("6 months", f, t); + let (f, t) = ms(2024, 1, 2024, 12); + drive("1 year", f, t); + let (f, t) = ms(2021, 1, 2024, 12); + drive("4 years", f, t); + let (f, t) = ms(2014, 8, 2024, 12); + drive("~10 years", f, t); +} diff --git a/fieldtests/research-breakout-deepdive/t2_walkforward.rs b/fieldtests/research-breakout-deepdive/t2_walkforward.rs new file mode 100644 index 0000000..89399a4 --- /dev/null +++ b/fieldtests/research-breakout-deepdive/t2_walkforward.rs @@ -0,0 +1,133 @@ +// Task 2 — Walk-forward the SHIPPED breakout across real GER40 history. +// +// The researcher's task: "the engine advertises a World walk-forward family. +// Roll my GER40 breakout across 2018..2024 with in-sample/out-of-sample windows +// and stitch the OOS pip curve." The breakout I want to roll is the SHIPPED one +// (crates/aura-ingest/examples/shared/breakout_real.rs) — a hand-wired FlatGraph. +// +// This file records the discovery journey verbatim. The public walk_forward is: +// walk_forward(roller: WindowRoller, space: Vec, +// run_window: Fn(WindowBounds) -> WindowRun) -> WalkForwardResult +// with WindowRun { chosen_params: Vec, oos_equity, oos_report }. +// +// SEAM 1 (recorded, not worked around): the breakout has NO swept params in this +// scenario (a fixed-param OOS roll), yet walk_forward DEMANDS a space: +// Vec and each WindowRun a chosen_params: Vec. With a raw +// FlatGraph there is no param_space() to hand it — the FlatGraph is built by +// Harness::bootstrap, which takes no params. So I pass an EMPTY space (vec![]) and +// EMPTY chosen_params (vec![]). Whether that is the intended idiom for a +// param-free strategy is a spec question — recorded. +// +// SEAM 2 (recorded): the breakout source is FOUR M1FieldSources (OHLC) opened by +// Unix-ms, but WindowBounds.{is,oos} are epoch-ns. aura-ingest exports +// unix_ms_to_epoch_ns but NOT its inverse, so I hand-divide ns->ms (the milestone +// fixture's mw_3 hit the same seam). +// +// PUBLIC SURFACE ONLY: walk_forward signature from cargo doc; the breakout wiring +// reused verbatim from the shipped example's shared module. + +use std::sync::Arc; + +use chrono::TimeZone; +use data_server::{DataServer, DEFAULT_DATA_PATH}; + +use aura_core::Timestamp; +use aura_engine::{ + walk_forward, RollMode, RunReport, WindowBounds, WindowRoller, WindowRun, +}; + +#[path = "../../crates/aura-ingest/examples/shared/breakout_real.rs"] +mod breakout_real; +use breakout_real::*; + +// SEAM 2: ns -> ms (aura-ingest exports only the forward direction). +fn ns_to_ms(ts: Timestamp) -> i64 { + ts.0 / 1_000_000 +} + +// Run the SHIPPED breakout over one [from, to] epoch-ns window of real GER40 M1 +// OHLC, returning the OOS pip-equity stream + the folded RunReport. +fn run_breakout_window( + server: &Arc, + from: Timestamp, + to: Timestamp, +) -> (Vec<(Timestamp, f64)>, RunReport) { + let from_ms = ns_to_ms(from); + let to_ms = ns_to_ms(to); + // Open the four OHLC field sources directly (the shipped open_ohlc_sources + // helper hard-codes Some(from)/Some(to) and returns Option — reuse it). + let sources = open_ohlc_sources(server, from_ms, to_ms).expect("window overlaps GER40 data"); + let (mut h, taps) = build_harness(); + h.run(sources); + let trace = drain_trace(&taps); + let report = report_from_trace(&trace, from_ms, to_ms); + let oos_equity: Vec<(Timestamp, f64)> = trace.iter().map(|b| (b.ts, b.equity)).collect(); + (oos_equity, report) +} + +fn main() { + let server = Arc::new(DataServer::new(DEFAULT_DATA_PATH)); + if !server.has_symbol(SYMBOL) { + println!("skip: no local {SYMBOL} data"); + return; + } + + println!("=== Task 2: walk-forward the shipped GER40 breakout (2018..2024) ===\n"); + + // The roll span: 2018-01 .. 2024-12 in epoch-ns. is=12mo, oos=3mo, step=3mo. + let day_ns: i64 = 86_400_000 * 1_000_000; + let month_ns: i64 = 30 * day_ns; + let origin = aura_ingest::unix_ms_to_epoch_ns( + chrono::Utc.with_ymd_and_hms(2018, 1, 1, 0, 0, 0).unwrap().timestamp_millis(), + ); + let end = aura_ingest::unix_ms_to_epoch_ns( + chrono::Utc.with_ymd_and_hms(2024, 12, 31, 23, 59, 59).unwrap().timestamp_millis(), + ); + let roller = WindowRoller::new( + (origin, end), + 12 * month_ns, // in-sample length + 3 * month_ns, // out-of-sample length + 3 * month_ns, // step + RollMode::Rolling, + ) + .expect("span fits >= 1 window"); + + let server_for_closure = Arc::clone(&server); + + // SEAM 1: the breakout is param-free here, so space = vec![] and each + // WindowRun carries chosen_params = vec![]. There is no in-sample optimize to + // do (no params to tune with a raw FlatGraph), so each window just runs the + // fixed strategy on its OOS slice. (Tuning the breakout's structural params is + // Task 3 — and the seam there is that they are NOT a FlatGraph param_space.) + let result = walk_forward(roller, vec![], move |w: WindowBounds| -> WindowRun { + let (oos_equity, oos_report) = + run_breakout_window(&server_for_closure, w.oos.0, w.oos.1); + WindowRun { chosen_params: vec![], oos_equity, oos_report } + }); + + println!("walk-forward produced {} windows\n", result.windows.len()); + for (k, w) in result.windows.iter().enumerate() { + let b = w.bounds; + println!( + " window {k}: is=[{}..{}] oos=[{}..{}] oos_pips={:>8.1} oos_dd={:>7.1}", + ns_to_ms(b.is.0), + ns_to_ms(b.is.1), + ns_to_ms(b.oos.0), + ns_to_ms(b.oos.1), + w.run.oos_report.metrics.total_pips, + w.run.oos_report.metrics.max_drawdown, + ); + } + + let stitched = &result.stitched_oos_equity; + println!( + "\nstitched OOS curve: {} points, final cumulative = {:.1} pips", + stitched.len(), + stitched.last().map(|(_, v)| *v).unwrap_or(0.0), + ); + + // The stitched OOS curve is the breakout's out-of-sample pip total over the + // concatenated OOS quarters (no in-sample optimization happened — same fixed + // FlatGraph strategy everywhere, since a raw FlatGraph has no param_space()). + println!("\nOK: fixed-param OOS walk-forward over the shipped breakout FlatGraph."); +} diff --git a/fieldtests/research-breakout-deepdive/t3_sweep.rs b/fieldtests/research-breakout-deepdive/t3_sweep.rs new file mode 100644 index 0000000..cb695a7 --- /dev/null +++ b/fieldtests/research-breakout-deepdive/t3_sweep.rs @@ -0,0 +1,229 @@ +// Task 3 — Parameter sweep of the breakout's STRUCTURAL params. +// +// The researcher's task: "sweep the breakout's entry bar index, exit bar index, +// and bar period. NOTE the coupling: the Resample period and the Session period +// must stay EQUAL, and the entry/exit bar indices are the EqConst targets. Can +// the public named-param sweep express this without a topology change?" +// +// To sweep ANYTHING I must first RE-AUTHOR the shipped breakout (a hand-wired +// FlatGraph with no param_space()) as a Composite blueprint — only a Composite +// exposes param_space() / the named-axis sweep / bootstrap_with_cells that the +// World sweep API consumes. The shipped example does NOT ship the breakout as a +// Composite, so this re-authoring is forced on the researcher (recorded as the +// headline Task-3 friction). The 13-node wiring below is the SAME topology as the +// shipped shared/breakout_real.rs, re-expressed as a Composite. +// +// PUBLIC SURFACE ONLY: Composite/axis/param_space/bootstrap_with_cells from cargo +// doc; EqConst/Resample/Session builder param declarations from their rustdoc +// (EqConst declares `target`; Resample declares `period_minutes`; Session BAKES +// open/tz/period — "NOT scalar params, the declared param list is empty"). + +use std::sync::mpsc; +use std::sync::Arc; + +use aura_core::{Cell, Firing, Scalar, ScalarKind, Timestamp}; +use aura_engine::{ + sweep, summarize, BlueprintNode, Composite, Edge, GridSpace, RunManifest, RunReport, + Role, Target, +}; +use aura_ingest::{M1Field, M1FieldSource}; +use aura_std::{And, Delay, EqConst, Gt, Latch, Recorder, Resample, Session, SimBroker}; +use chrono::TimeZone; +use chrono_tz::Europe::Berlin; +use data_server::{DataServer, DEFAULT_DATA_PATH}; + +const SYMBOL: &str = "GER40"; +const PIP_SIZE: f64 = 1.0; +const BAR_MINUTES: i64 = 15; + +// The breakout re-authored as a Composite. Same 9-node strategy topology + +// SimBroker + 2 recorder sinks (equity, held). Sources bind to the 4 OHLC roles. +// Returns the blueprint plus the equity/held receivers a run will drain. +fn breakout_composite() -> ( + Composite, + mpsc::Receiver<(Timestamp, Vec)>, + mpsc::Receiver<(Timestamp, Vec)>, +) { + let (tx_eq, rx_eq) = mpsc::channel(); + let (tx_held, rx_held) = mpsc::channel(); + + // Node indices, parallel to the shipped FlatGraph (sans the two extra taps). + // 0 Resample, 1 Delay, 2 Gt, 3 Session, 4 EqConst(entry), 5 EqConst(exit), + // 6 And, 7 Latch, 8 SimBroker, 9 Rec(equity), 10 Rec(held). + let nodes: Vec = vec![ + Resample::builder().into(), // 0 + Delay::builder().into(), // 1 + Gt::builder().into(), // 2 + Session::builder(9, 0, Berlin, BAR_MINUTES).into(), // 3 (period BAKED) + EqConst::builder().named("entry_bar").into(), // 4 (target tunable) + EqConst::builder().named("exit_bar").into(), // 5 (target tunable) + And::builder().into(), // 6 + Latch::builder().into(), // 7 + SimBroker::builder(PIP_SIZE).into(), // 8 + Recorder::builder(vec![ScalarKind::F64], Firing::Any, tx_eq).into(), // 9 + Recorder::builder(vec![ScalarKind::F64], Firing::Any, tx_held).into(), // 10 + ]; + + const F_HIGH15: usize = 1; + const F_CLOSE15: usize = 3; + let edges = vec![ + Edge { from: 0, to: 2, slot: 0, from_field: F_CLOSE15 }, // close15 -> Gt.a + Edge { from: 0, to: 1, slot: 0, from_field: F_HIGH15 }, // high15 -> Delay + Edge { from: 1, to: 2, slot: 1, from_field: 0 }, // prevHigh15 -> Gt.b + Edge { from: 0, to: 3, slot: 0, from_field: F_CLOSE15 }, // close15 -> Session.trigger + Edge { from: 3, to: 4, slot: 0, from_field: 0 }, // bars -> EqConst(entry) + Edge { from: 3, to: 5, slot: 0, from_field: 0 }, // bars -> EqConst(exit) + Edge { from: 2, to: 6, slot: 0, from_field: 0 }, // breakout -> And.a + Edge { from: 4, to: 6, slot: 1, from_field: 0 }, // isEntryBar -> And.b + Edge { from: 6, to: 7, slot: 0, from_field: 0 }, // entry -> Latch.set + Edge { from: 5, to: 7, slot: 1, from_field: 0 }, // isExitBar -> Latch.reset + Edge { from: 7, to: 8, slot: 0, from_field: 0 }, // held -> SimBroker.exposure + Edge { from: 0, to: 8, slot: 1, from_field: F_CLOSE15 }, // close15 -> SimBroker.price + Edge { from: 8, to: 9, slot: 0, from_field: 0 }, // equity -> sink + Edge { from: 7, to: 10, slot: 0, from_field: 0 }, // held -> sink + ]; + + // Four bound OHLC source roles -> the Resample's four Barrier(0) slots, in the + // fixed C4 merge order open(0) high(1) low(2) close(3). + let roles = (0..4) + .map(|slot| Role { + name: ["open", "high", "low", "close"][slot as usize].into(), + targets: vec![Target { node: 0, slot }], + source: Some(ScalarKind::F64), + }) + .collect(); + + let bp = Composite::new("ger40_breakout", nodes, edges, roles, vec![]); + (bp, rx_eq, rx_held) +} + +fn utc_month_window_ms(year: i32, month: u32) -> (i64, i64) { + let from = chrono::Utc.with_ymd_and_hms(year, month, 1, 0, 0, 0).unwrap().timestamp_millis(); + let (ny, nm) = if month == 12 { (year + 1, 1) } else { (year, month + 1) }; + let next = chrono::Utc.with_ymd_and_hms(ny, nm, 1, 0, 0, 0).unwrap().timestamp_millis(); + (from, next - 1) +} + +fn run_point(server: &Arc, point: &[Cell], from_ms: i64, to_ms: i64) -> RunReport { + let (bp, rx_eq, rx_held) = breakout_composite(); + let mut h = bp + .bootstrap_with_cells(point) + .expect("sweep point kind-checked against param_space"); + let fields = [M1Field::Open, M1Field::High, M1Field::Low, M1Field::Close]; + let srcs: Vec> = fields + .iter() + .map(|&f| { + Box::new( + M1FieldSource::open(server, SYMBOL, Some(from_ms), Some(to_ms), f) + .expect("OHLC field overlaps window"), + ) as Box + }) + .collect(); + h.run(srcs); + drop(h); + let eq: Vec<(Timestamp, f64)> = rx_eq.try_iter().map(|(t, r)| (t, r[0].as_f64())).collect(); + let ex: Vec<(Timestamp, f64)> = rx_held.try_iter().map(|(t, r)| (t, r[0].as_f64())).collect(); + RunReport { + manifest: RunManifest { + commit: "breakout-sweep".into(), + params: vec![], + window: ( + aura_ingest::unix_ms_to_epoch_ns(from_ms), + aura_ingest::unix_ms_to_epoch_ns(to_ms), + ), + seed: 0, + broker: "sim-optimal(pip_size=1)".into(), + }, + metrics: summarize(&eq, &ex), + } +} + +fn main() { + let server = Arc::new(DataServer::new(DEFAULT_DATA_PATH)); + if !server.has_symbol(SYMBOL) { + println!("skip: no local {SYMBOL} data"); + return; + } + + println!("=== Task 3: sweep the breakout's structural params ===\n"); + + // 1. What does param_space() actually expose for the re-authored breakout? + let space = breakout_composite().0.param_space(); + println!("param_space() of the breakout Composite:"); + for ps in &space { + println!(" {:<28} : {:?}", ps.name, ps.kind); + } + println!(); + + // The CRUX (recorded): note what is and is NOT in the space. + // - entry_bar.target / exit_bar.target ARE here (EqConst declares `target`). + // - resample.period_minutes IS here (Resample declares it tunable). + // - There is NO session.* period knob — Session BAKES its period (a structural + // factory arg). So "Resample period == Session period" CANNOT be a single + // swept named param: the Session leg of the coupling is invisible to the + // sweep. Sweeping resample.period_minutes alone would silently DESYNC the + // two periods (a correctness trap), and changing the Session period at all + // requires REBUILDING the Composite (a different blueprint), exactly the + // "topology change, not a param" line C11 draws. + let has_resample_period = space.iter().any(|p| p.name.contains("period_minutes")); + let has_session_period = space.iter().any(|p| p.name.to_lowercase().contains("session")); + println!("coupling check:"); + println!(" resample.period_minutes in space? {has_resample_period}"); + println!(" any session.* period in space? {has_session_period}"); + println!(" => the bar-period coupling is NOT expressible as one swept param.\n"); + + // 2. The sweep that DOES work: entry_bar x exit_bar (both EqConst targets), + // at the FIXED 15m period (no period coupling to violate). A 3x3 grid over + // one month of GER40. + let (from_ms, to_ms) = utc_month_window_ms(2024, 9); + let grid = GridSpace::new( + &space, + space + .iter() + .map(|ps| match ps.name.as_str() { + "entry_bar.target" => vec![Scalar::I64(2), Scalar::I64(3), Scalar::I64(4)], + "exit_bar.target" => vec![Scalar::I64(5), Scalar::I64(6), Scalar::I64(8)], + // resample.period_minutes pinned to 15 (== the baked Session 15m). + "resample.period_minutes" => vec![Scalar::I64(15)], + // SURPRISE (recorded): delay.lag is a STRUCTURAL deformer (the + // "previous bar's high" of the breakout) yet ships in param_space + // as a tunable knob — so the grid is FORCED to enumerate it (here + // pinned to its structural value 1) or panic. C34's bind() would + // remove it from the space, but the node exposes it by default. + "delay.lag" => vec![Scalar::I64(1)], + other => panic!("unexpected param slot {other}"), + }) + .collect(), + ) + .expect("grid well-formed for the space"); + + println!("sweeping entry_bar x exit_bar (period pinned 15m) over GER40 2024-09:"); + let server_for_closure = Arc::clone(&server); + let family = sweep(&grid, |pt: &[Cell]| run_point(&server_for_closure, pt, from_ms, to_ms)); + + // Print each point with its named params (by index — SweepFamily::named_params(i)). + for (i, p) in family.points.iter().enumerate() { + let named = family.named_params(i); + let label: Vec = named + .iter() + .filter(|(n, _)| n != "resample.period_minutes") + .map(|(n, v)| format!("{}={}", n.replace(".target", ""), scalar_str(v))) + .collect(); + println!( + " {{ {} }} total_pips={:>8.1} max_dd={:>7.1}", + label.join(", "), + p.report.metrics.total_pips, + p.report.metrics.max_drawdown, + ); + } + println!("\nOK: entry/exit-bar grid sweep works; bar-period coupling does not."); +} + +fn scalar_str(s: &Scalar) -> String { + match s { + Scalar::I64(v) => v.to_string(), + Scalar::F64(v) => format!("{v:.2}"), + Scalar::Bool(v) => v.to_string(), + Scalar::Timestamp(t) => t.0.to_string(), + } +} diff --git a/fieldtests/research-breakout-deepdive/t3b_period_desync.rs b/fieldtests/research-breakout-deepdive/t3b_period_desync.rs new file mode 100644 index 0000000..9dfeea0 --- /dev/null +++ b/fieldtests/research-breakout-deepdive/t3b_period_desync.rs @@ -0,0 +1,143 @@ +// Task 3 (deep dive) — the silent period DESYNC. +// +// The coupling claim from t3_sweep: Resample's period IS a swept param, but +// Session's period is BAKED, so they cannot move together. This probe shows the +// SHARP edge of that: a researcher who sweeps `resample.period_minutes` to 30 +// (expecting "30-minute bars") gets a graph where the BARS are 30m but the +// SESSION still indexes 15m slots. `entry_bar.target == 3` then no longer means +// "the 3rd 30m bar"; the two clocks silently disagree and the strategy computes a +// DIFFERENT, unintended thing — with NO error, NO warning. That is the danger of a +// coupling the param system cannot enforce. +// +// We run the SAME entry/exit-bar config at period_minutes = 15 (coupled, correct) +// vs period_minutes = 30 (desynced) and show both bootstrap+run happily, but the +// 30m run is semantically broken (Session still ticks 15m). +// +// PUBLIC SURFACE ONLY. The Composite + builders are the public re-authoring; the +// desync is a property of Session BAKING its period (its rustdoc) while Resample +// exposes its own. + +use std::sync::mpsc; +use std::sync::Arc; + +use aura_core::{Cell, Firing, Scalar, ScalarKind, Timestamp}; +use aura_engine::{summarize, BlueprintNode, Composite, Edge, Role, RunManifest, RunReport, Target}; +use aura_ingest::{M1Field, M1FieldSource}; +use aura_std::{And, Delay, EqConst, Gt, Latch, Recorder, Resample, Session, SimBroker}; +use chrono::TimeZone; +use chrono_tz::Europe::Berlin; +use data_server::{DataServer, DEFAULT_DATA_PATH}; + +const SYMBOL: &str = "GER40"; +// The SESSION period is baked here — it never moves no matter what the sweep does +// to resample.period_minutes. THIS is the coupling the param system cannot keep. +const SESSION_PERIOD_BAKED: i64 = 15; + +fn breakout( + session_period: i64, +) -> (Composite, mpsc::Receiver<(Timestamp, Vec)>, mpsc::Receiver<(Timestamp, Vec)>) { + let (tx_eq, rx_eq) = mpsc::channel(); + let (tx_held, rx_held) = mpsc::channel(); + let nodes: Vec = vec![ + Resample::builder().into(), + Delay::builder().into(), + Gt::builder().into(), + Session::builder(9, 0, Berlin, session_period).into(), + EqConst::builder().named("entry_bar").into(), + EqConst::builder().named("exit_bar").into(), + And::builder().into(), + Latch::builder().into(), + SimBroker::builder(1.0).into(), + Recorder::builder(vec![ScalarKind::F64], Firing::Any, tx_eq).into(), + Recorder::builder(vec![ScalarKind::F64], Firing::Any, tx_held).into(), + ]; + let edges = vec![ + Edge { from: 0, to: 2, slot: 0, from_field: 3 }, + Edge { from: 0, to: 1, slot: 0, from_field: 1 }, + Edge { from: 1, to: 2, slot: 1, from_field: 0 }, + Edge { from: 0, to: 3, slot: 0, from_field: 3 }, + Edge { from: 3, to: 4, slot: 0, from_field: 0 }, + Edge { from: 3, to: 5, slot: 0, from_field: 0 }, + Edge { from: 2, to: 6, slot: 0, from_field: 0 }, + Edge { from: 4, to: 6, slot: 1, from_field: 0 }, + Edge { from: 6, to: 7, slot: 0, from_field: 0 }, + Edge { from: 5, to: 7, slot: 1, from_field: 0 }, + Edge { from: 7, to: 8, slot: 0, from_field: 0 }, + Edge { from: 0, to: 8, slot: 1, from_field: 3 }, + Edge { from: 8, to: 9, slot: 0, from_field: 0 }, + Edge { from: 7, to: 10, slot: 0, from_field: 0 }, + ]; + let roles = (0..4) + .map(|slot| Role { + name: ["open", "high", "low", "close"][slot as usize].into(), + targets: vec![Target { node: 0, slot }], + source: Some(ScalarKind::F64), + }) + .collect(); + (Composite::new("breakout", nodes, edges, roles, vec![]), rx_eq, rx_held) +} + +fn run(server: &Arc, resample_period: i64, session_period: i64, from_ms: i64, to_ms: i64) -> (RunReport, usize) { + let (bp, rx_eq, rx_held) = breakout(session_period); + let space = bp.param_space(); + // Build the param vector in param_space() order. + let point: Vec = space + .iter() + .map(|ps| match ps.name.as_str() { + "resample.period_minutes" => Scalar::I64(resample_period).cell(), + "delay.lag" => Scalar::I64(1).cell(), + "entry_bar.target" => Scalar::I64(3).cell(), + "exit_bar.target" => Scalar::I64(5).cell(), + other => panic!("unexpected slot {other}"), + }) + .collect(); + let mut h = bp.bootstrap_with_cells(&point).expect("kind-checked"); + let fields = [M1Field::Open, M1Field::High, M1Field::Low, M1Field::Close]; + let srcs: Vec> = fields + .iter() + .map(|&f| Box::new(M1FieldSource::open(server, SYMBOL, Some(from_ms), Some(to_ms), f).unwrap()) as Box) + .collect(); + h.run(srcs); + drop(h); + let eq: Vec<(Timestamp, f64)> = rx_eq.try_iter().map(|(t, r)| (t, r[0].as_f64())).collect(); + let n_bars = eq.len(); + let ex: Vec<(Timestamp, f64)> = rx_held.try_iter().map(|(t, r)| (t, r[0].as_f64())).collect(); + let rep = RunReport { + manifest: RunManifest { commit: "desync".into(), params: vec![], window: (Timestamp(0), Timestamp(0)), seed: 0, broker: "sim".into() }, + metrics: summarize(&eq, &ex), + }; + (rep, n_bars) +} + +fn main() { + let server = Arc::new(DataServer::new(DEFAULT_DATA_PATH)); + if !server.has_symbol(SYMBOL) { + println!("skip: no local {SYMBOL} data"); + return; + } + let from = chrono::Utc.with_ymd_and_hms(2024, 9, 1, 0, 0, 0).unwrap().timestamp_millis(); + let to = chrono::Utc.with_ymd_and_hms(2024, 10, 1, 0, 0, 0).unwrap().timestamp_millis() - 1; + + println!("=== Task 3 deep dive: the silent period desync ==="); + println!("entry_bar=3, exit_bar=5 fixed; Session period BAKED at {SESSION_PERIOD_BAKED}m.\n"); + + // Coupled (correct): resample 15m == session 15m. This is the shipped baseline. + let (coupled, n15) = run(&server, 15, SESSION_PERIOD_BAKED, from, to); + println!("resample=15m, session=15m (COUPLED, correct): bars={n15:<6} total_pips={:.1}", coupled.metrics.total_pips); + + // Desynced: a researcher sweeps resample to 30m, EXPECTING 30m bars. Session + // still ticks 15m (baked). entry_bar==3 now indexes a 15m slot on a 30m bar + // stream — a different, unintended signal. NO error is raised. + let (desync, n30) = run(&server, 30, SESSION_PERIOD_BAKED, from, to); + println!("resample=30m, session=15m (DESYNCED, silent): bars={n30:<6} total_pips={:.1}", desync.metrics.total_pips); + + // Contrast: to ACTUALLY get a coherent 30m strategy the researcher must REBUILD + // the Composite with session_period=30 (a different blueprint — topology, not a + // swept param). That run is coherent but is NOT reachable from the sweep. + let (rebuilt, n30b) = run(&server, 30, 30, from, to); + println!("resample=30m, session=30m (REBUILT blueprint): bars={n30b:<6} total_pips={:.1}", rebuilt.metrics.total_pips); + + println!("\nObservation: all three bootstrap+run with NO error. The desynced"); + println!("middle run is silently wrong — the param system cannot couple the two"); + println!("periods, and only a blueprint rebuild (not a sweep) gives a coherent 30m."); +} diff --git a/fieldtests/research-breakout-deepdive/t4_compare.rs b/fieldtests/research-breakout-deepdive/t4_compare.rs new file mode 100644 index 0000000..4ae2dd4 --- /dev/null +++ b/fieldtests/research-breakout-deepdive/t4_compare.rs @@ -0,0 +1,160 @@ +// Task 4 — Multi-instrument comparison: the SAME breakout on GER40 vs FRA40. +// +// The researcher's task: "run my breakout on another index (FRA40 — Paris, also +// CET 09:00) and compare signal quality against GER40. Is the comparison honest +// across instruments? What does the engine do about per-instrument pip metadata?" +// +// The experiment matrix here is the structural axis "which instrument" (C20): one +// strategy x N symbols, a plain Rust loop. The breakout Composite is REUSED +// verbatim across symbols; only the symbol the sources read and the pip_size the +// SimBroker bakes change. +// +// THE PIP QUESTION (the headline Task-4 finding): SimBroker::new(pip_size) takes a +// BARE f64. There is NO instrument/pip-metadata registry on the public surface +// (aura-ingest exposes only M1 streaming; Aura.toml — which the glossary says +// carries "instrument/pip metadata" — does not exist yet, it is an open thread). +// So the researcher must HARD-CODE pip_size per symbol from outside knowledge: +// GER40=1.0 (index point), FRA40=1.0 (index point), EURUSD=0.0001 (FX). Nothing in +// the engine knows or checks this. A GER40-vs-FRA40 comparison is honest ONLY +// because the researcher happens to know both are index points; the engine offers +// no help and no guard against comparing index pips to FX pips. +// +// PUBLIC SURFACE ONLY. + +use std::sync::mpsc; +use std::sync::Arc; + +use aura_core::{Cell, Firing, Scalar, ScalarKind, Timestamp}; +use aura_engine::{summarize, BlueprintNode, Composite, Edge, Role, RunManifest, RunReport, Target}; +use aura_ingest::{M1Field, M1FieldSource}; +use aura_std::{And, Delay, EqConst, Gt, Latch, Recorder, Resample, Session, SimBroker}; +use chrono::TimeZone; +use chrono_tz::Europe::Berlin; +use data_server::{DataServer, DEFAULT_DATA_PATH}; + +// What the researcher must supply BY HAND per instrument — there is no registry. +// (symbol, pip_size, human note on what a "pip" is for this asset) +const INSTRUMENTS: &[(&str, f64, &str)] = &[ + ("GER40", 1.0, "index point"), + ("FRA40", 1.0, "index point"), +]; + +fn breakout( + pip_size: f64, +) -> (Composite, mpsc::Receiver<(Timestamp, Vec)>, mpsc::Receiver<(Timestamp, Vec)>) { + let (tx_eq, rx_eq) = mpsc::channel(); + let (tx_held, rx_held) = mpsc::channel(); + let nodes: Vec = vec![ + Resample::builder().into(), + Delay::builder().into(), + Gt::builder().into(), + Session::builder(9, 0, Berlin, 15).into(), + EqConst::builder().named("entry_bar").into(), + EqConst::builder().named("exit_bar").into(), + And::builder().into(), + Latch::builder().into(), + SimBroker::builder(pip_size).into(), + Recorder::builder(vec![ScalarKind::F64], Firing::Any, tx_eq).into(), + Recorder::builder(vec![ScalarKind::F64], Firing::Any, tx_held).into(), + ]; + let edges = vec![ + Edge { from: 0, to: 2, slot: 0, from_field: 3 }, + Edge { from: 0, to: 1, slot: 0, from_field: 1 }, + Edge { from: 1, to: 2, slot: 1, from_field: 0 }, + Edge { from: 0, to: 3, slot: 0, from_field: 3 }, + Edge { from: 3, to: 4, slot: 0, from_field: 0 }, + Edge { from: 3, to: 5, slot: 0, from_field: 0 }, + Edge { from: 2, to: 6, slot: 0, from_field: 0 }, + Edge { from: 4, to: 6, slot: 1, from_field: 0 }, + Edge { from: 6, to: 7, slot: 0, from_field: 0 }, + Edge { from: 5, to: 7, slot: 1, from_field: 0 }, + Edge { from: 7, to: 8, slot: 0, from_field: 0 }, + Edge { from: 0, to: 8, slot: 1, from_field: 3 }, + Edge { from: 8, to: 9, slot: 0, from_field: 0 }, + Edge { from: 7, to: 10, slot: 0, from_field: 0 }, + ]; + let roles = (0..4) + .map(|slot| Role { + name: ["open", "high", "low", "close"][slot as usize].into(), + targets: vec![Target { node: 0, slot }], + source: Some(ScalarKind::F64), + }) + .collect(); + (Composite::new("breakout", nodes, edges, roles, vec![]), rx_eq, rx_held) +} + +fn run_symbol(server: &Arc, symbol: &str, pip_size: f64, from_ms: i64, to_ms: i64) -> Option<(RunReport, u32)> { + let (bp, rx_eq, rx_held) = breakout(pip_size); + let space = bp.param_space(); + // entry=3, exit=5, period=15, lag=1 — the shipped baseline config. + let point: Vec = space + .iter() + .map(|ps| match ps.name.as_str() { + "resample.period_minutes" => Scalar::I64(15).cell(), + "delay.lag" => Scalar::I64(1).cell(), + "entry_bar.target" => Scalar::I64(3).cell(), + "exit_bar.target" => Scalar::I64(5).cell(), + other => panic!("unexpected slot {other}"), + }) + .collect(); + let mut h = bp.bootstrap_with_cells(&point).expect("kind-checked"); + let fields = [M1Field::Open, M1Field::High, M1Field::Low, M1Field::Close]; + let mut srcs: Vec> = Vec::new(); + for &f in &fields { + let s = M1FieldSource::open(server, symbol, Some(from_ms), Some(to_ms), f)?; + srcs.push(Box::new(s)); + } + h.run(srcs); + drop(h); + let eq: Vec<(Timestamp, f64)> = rx_eq.try_iter().map(|(t, r)| (t, r[0].as_f64())).collect(); + let held: Vec<(Timestamp, f64)> = rx_held.try_iter().map(|(t, r)| (t, r[0].as_f64())).collect(); + // count entries (0->1 transitions) + let mut entries = 0u32; + let mut prev = 0.0; + for (_, v) in &held { + if *v == 1.0 && prev == 0.0 { + entries += 1; + } + prev = *v; + } + let rep = RunReport { + manifest: RunManifest { commit: "compare".into(), params: vec![], window: (Timestamp(0), Timestamp(0)), seed: 0, broker: format!("sim(pip={pip_size})") }, + metrics: summarize(&eq, &held), + }; + Some((rep, entries)) +} + +fn main() { + let server = Arc::new(DataServer::new(DEFAULT_DATA_PATH)); + println!("=== Task 4: multi-instrument comparison (same breakout, GER40 vs FRA40) ===\n"); + + // Full year 2024 so the comparison has statistical body. + let from = chrono::Utc.with_ymd_and_hms(2024, 1, 1, 0, 0, 0).unwrap().timestamp_millis(); + let to = chrono::Utc.with_ymd_and_hms(2024, 12, 31, 23, 59, 59).unwrap().timestamp_millis(); + + println!("{:<8} {:<14} {:>8} {:>10} {:>10} {:>7}", "symbol", "pip-meaning", "sessions", "total_pips", "max_dd", "flips"); + println!("{}", "-".repeat(62)); + for &(symbol, pip_size, note) in INSTRUMENTS { + if !server.has_symbol(symbol) { + println!("{symbol:<8} (no local data — skip)"); + continue; + } + match run_symbol(&server, symbol, pip_size, from, to) { + Some((rep, entries)) => { + let m = &rep.metrics; + println!( + "{symbol:<8} {note:<14} {entries:>8} {:>10.1} {:>10.1} {:>7}", + m.total_pips, m.max_drawdown, m.exposure_sign_flips + ); + } + None => println!("{symbol:<8} (no overlapping OHLC files — skip)"), + } + } + + println!("\nHonesty of the cross-asset comparison:"); + println!(" Both GER40 and FRA40 are index points (pip=1.0), so their pip totals"); + println!(" ARE directly comparable here — BUT only because the researcher knew"); + println!(" to pass pip=1.0 for both. The engine has NO instrument registry: pip"); + println!(" is a bare f64 to SimBroker::new, and nothing would stop (or flag) a"); + println!(" comparison of GER40 index-pips to EURUSD fx-pips with mismatched scale."); +}