diff --git a/docs/plans/0074-pip-from-sidecar.md b/docs/plans/0074-pip-from-sidecar.md new file mode 100644 index 0000000..623b209 --- /dev/null +++ b/docs/plans/0074-pip-from-sidecar.md @@ -0,0 +1,385 @@ +# Provider-sourced pip; drop the authored instrument floor — Implementation Plan + +> **Parent spec:** `docs/specs/0074-pip-from-sidecar.md` +> +> **For agentic workers:** REQUIRED SUB-SKILL: use the `implement` skill to run +> this plan. Steps use `- [ ]` checkboxes for tracking. + +**Goal:** Resolve the real-path pip from the data-server geometry sidecar and +remove the now-redundant authored instrument floor (`InstrumentSpec`, +`instrument_spec`, `VETTED_SYMBOLS`) and its table-only tests, keeping only the +one datum production consumes — `pip_size`. + +**Architecture:** The resolution collapses to `recorded sidecar geometry → +refuse-don't-guess`. `aura_ingest::instrument_geometry` (the #143 accessor) is the +sole pip source. The authored floor and its 9 fields (incl. the unconsumed +`instrument_id`) are deleted; speculative fields revive from `InstrumentGeometry` +when the Stage-2 broker consumes them. Tasks are dependency-ordered so every +intermediate `cargo test` gate is satisfiable: consumers are repointed (Tasks +1–2) *before* the definitions are removed (Task 3). Each task is self-contained +green (RED→GREEN within the task per TDD). + +**Tech Stack:** aura-cli (real-data pip resolution), aura-ingest (lib surface + +example fixtures), the design ledger. + +--- + +**Files this plan creates or modifies:** + +- Modify: `crates/aura-cli/src/main.rs:726–806` — `instrument_spec_or_refuse`→`pip_or_refuse` (sidecar-sourced `f64`); reorder both call sites (server before pip) +- Modify: `crates/aura-cli/tests/cli_run.rs` — new RED test (non-vetted symbol resolves from sidecar) + refusal-message-pin updates + AAPL.US→NONEXISTENT premise fix +- Modify: `crates/aura-ingest/examples/shared/breakout_real.rs:71–80` — `pip_size_of` repointed to the sidecar +- Modify: `crates/aura-ingest/tests/ger40_breakout_blueprint.rs:40,67,68` — literal pip in the two pure-structural tests +- Modify: `crates/aura-ingest/src/lib.rs` — remove `InstrumentSpec` (361–390), `VETTED_SYMBOLS` (392–397), `instrument_spec` (399–436), 3 unit tests (453–557); fix the `:353` intra-doc link +- Delete: `crates/aura-ingest/tests/instrument_geometry_crosscheck.rs` (floor cross-check, moot) +- Delete: `crates/aura-ingest/tests/instrument_spec_deploy_metadata.rs` (asserts removed fields) +- Create: `crates/aura-ingest/tests/instrument_geometry.rs` — relocated seam test guarding the surviving `InstrumentGeometry` re-export +- Modify: `docs/design/INDEX.md` — C15 realization note (~825–834), #124 hierarchy note, #22 realization (~604–609), #106 amendment (~1236) +- Modify: `crates/aura-engine/src/report.rs:226–227` — comment naming the removed `InstrumentSpec` +- Modify: `crates/aura-ingest/examples/ger40_breakout_compare.rs:8,43,50` — doc-prose naming `instrument_spec` + +--- + +### Task 1: aura-cli — resolve pip from the sidecar (RED → GREEN) + +**Files:** +- Modify: `crates/aura-cli/tests/cli_run.rs` (new test after `run_real_vetted_index_pip_reaches_the_emitted_manifest` ~line 249; message pins ~145, ~188, ~1189–1195) +- Modify: `crates/aura-cli/src/main.rs:726–806` + +- [ ] **Step 1: Write the failing test (RED)** + +Add this test — it captures the directive's whole point: the authored table no +longer gates; any symbol with recorded data resolves. USDJPY was never in the +vetted 5-symbol table but has a sidecar (`pip 0.01`) and bars on this host. + +```rust +/// Property: the authored instrument table no longer gates real runs — a symbol +/// absent from the (removed) vetted floor but carrying a recorded geometry sidecar +/// resolves its pip from the sidecar and runs. USDJPY (never vetted; JPY pip 0.01) +/// renders the JPY pip in the manifest, proving the pip is sourced from the provider +/// geometry, not a hand-authored table entry. Gated: skips cleanly when USDJPY has +/// no local geometry/data (no sidecar → "no recorded geometry"; sidecar but no bars +/// → "no local data"). +#[test] +fn run_real_nonvetted_symbol_resolves_pip_from_sidecar() { + // FX trades continuously; the GER40 Sept-2024 window also has USDJPY bars. + const FROM_MS: &str = "1725148800000"; + const TO_MS: &str = "1727740799999"; + let out = std::process::Command::new(env!("CARGO_BIN_EXE_aura")) + .args(["run", "--real", "USDJPY", "--from", FROM_MS, "--to", TO_MS]) + .output() + .expect("spawn aura"); + + // Skip on a host without USDJPY geometry/data — either refusal is a clean skip, + // never the old table-miss "no vetted pip". + if out.status.code() == Some(2) { + let stderr = String::from_utf8_lossy(&out.stderr); + if stderr.contains("no recorded geometry") || stderr.contains("no local data") { + eprintln!("skip: no local USDJPY geometry/data"); + return; + } + } + + assert_eq!(out.status.code(), Some(0), "USDJPY should resolve and run; exit: {:?}, stderr: {}", + out.status, String::from_utf8_lossy(&out.stderr)); + let stdout = String::from_utf8(out.stdout).expect("utf-8 stdout"); + assert!( + stdout.contains("pip_size=0.01"), + "USDJPY must render the JPY sidecar pip (0.01) in the manifest, got: {}", + stdout.trim_end() + ); +} +``` + +- [ ] **Step 2: Run the test to verify it fails RED** + +Run: `cargo test -p aura-cli --test cli_run run_real_nonvetted_symbol_resolves_pip_from_sidecar` +Expected: FAIL — current code refuses USDJPY (`instrument_spec("USDJPY")` is `None` +→ exit 2 with "no vetted pip/instrument spec"), which matches neither skip +substring, so `assert_eq!(.., Some(0))` fails. (Archive present on this host, so it +does not skip.) + +- [ ] **Step 3: Replace the helper with a sidecar-sourced resolver** + +Replace `instrument_spec_or_refuse` (main.rs:726–738): + +```rust +/// Resolve the per-instrument pip from the recorded geometry sidecar, or refuse +/// (stderr + exit 2) when the symbol has no recorded geometry — the single home of +/// the guessed-pip refusal, shared by `open_real_source` and `from_choice`. Reads +/// the symbol's geometry metadata (not bar data) before the run source opens, so an +/// instrument with no recorded geometry refuses without a guessed pip; the pip is +/// the provider's recorded value, honest by construction. +fn pip_or_refuse(server: &std::sync::Arc, symbol: &str) -> f64 { + match aura_ingest::instrument_geometry(server, symbol) { + Some(geo) => geo.pip_size, + None => { + eprintln!( + "aura: no recorded geometry for symbol '{symbol}' at {} — \ + refusing to run a real instrument with a guessed pip", + data_server::DEFAULT_DATA_PATH + ); + std::process::exit(2); + } + } +} +``` + +- [ ] **Step 4: Reorder `open_real_source` (server before pip)** + +In `open_real_source` (main.rs:767–787): build the server first, resolve the pip, +return `pip` in the tuple (was `spec.pip_size`). + +```rust + // Per-instrument pip from the recorded sidecar; resolved BEFORE bar-data access + // so an instrument with no geometry refuses without touching the archive. + let server = std::sync::Arc::new(data_server::DataServer::new(data_server::DEFAULT_DATA_PATH)); + let pip = pip_or_refuse(&server, symbol); + if !server.has_symbol(symbol) { + no_real_data(symbol); + } + // ... probe_window + M1FieldSource::open unchanged ... + (source, window, pip) +``` + +- [ ] **Step 5: Reorder `DataSource::from_choice`** + +In `from_choice` (main.rs:794–805): build the server first, resolve the pip, set +`pip` in `DataSource::Real`. Update the doc-comment at :792 +(`instrument_spec_or_refuse` → `pip_or_refuse`). + +```rust + DataChoice::Real { symbol, from_ms, to_ms } => { + let server = std::sync::Arc::new(data_server::DataServer::new(data_server::DEFAULT_DATA_PATH)); + let pip = pip_or_refuse(&server, &symbol); + if !server.has_symbol(&symbol) { + no_real_data(&symbol); + } + DataSource::Real { server, symbol, from_ms, to_ms, pip } + } +``` + +- [ ] **Step 6: Update the refusal-message pins + the AAPL.US premise** + +In `cli_run.rs`: +- `run_real_unspecced_symbol_refuses_with_exit_2` (~145): substring + `"no vetted pip/instrument spec for symbol 'NONEXISTENT'"` → + `"no recorded geometry for symbol 'NONEXISTENT'"`; update the comment ~135. +- `run_real_vetted_symbol_never_hits_the_pip_refusal` (~188): + `!stderr.contains("no vetted pip/instrument spec")` → + `!stderr.contains("no recorded geometry")`; update doc-prose ~171–179. +- `sweep_real_unspecced_symbol_refuses_with_exit_2` (~1189–1195): change the symbol + from `"AAPL.US"` to `"NONEXISTENT"` (AAPL.US now HAS a sidecar and would resolve, + not refuse), substring `"no vetted pip"` → `"no recorded geometry"`; update the + doc-comment ~1183–1187. + +- [ ] **Step 7: Run the gates for this task** + +Run: `cargo test -p aura-cli --test cli_run` +Expected: PASS — `run_real_nonvetted_symbol_resolves_pip_from_sidecar` is GREEN +(USDJPY resolves), `run_real_vetted_index_pip_reaches_the_emitted_manifest` still +GREEN (GER40 sidecar pip == 1.0, byte-identical manifest), and all three updated +refusal tests pass. + +--- + +### Task 2: aura-ingest examples — repoint `pip_size_of`; literal pip in pure tests + +**Files:** +- Modify: `crates/aura-ingest/examples/shared/breakout_real.rs:71–80` +- Modify: `crates/aura-ingest/tests/ger40_breakout_blueprint.rs:40,67,68` + +- [ ] **Step 1: Repoint `pip_size_of` to the sidecar** + +Replace `pip_size_of` (breakout_real.rs:71–80): + +```rust +/// The example's pip divisor, sourced from the recorded geometry sidecar — the same +/// provider truth the CLI `--real` path uses, now provider-sourced. Panics on a +/// symbol with no sidecar (the examples only run known instruments with local data). +pub fn pip_size_of(symbol: &str) -> f64 { + aura_ingest::instrument_geometry(&aura_ingest::default_data_server(), symbol) + .unwrap_or_else(|| panic!("no recorded geometry sidecar for {symbol}")) + .pip_size +} +``` + +- [ ] **Step 2: Literal pip in the two pure-structural blueprint tests** + +In `ger40_breakout_blueprint.rs`, the two tests asserting `param_space()` topology +are contracted to run with no archive and must not read the sidecar. Replace +`pip_size_of(SYMBOL)` with a literal at `:40` (in +`param_space_is_exactly_entry_and_exit_bar_targets`): + +```rust + // pure-structural: the pip is a throwaway sizing arg, never asserted here, so a + // literal keeps this archive-independent (the topology assertion is the point). + let (bp, _taps) = ger40_breakout_blueprint(1.0, 15, 9, 0, Berlin); +``` + +and at `:67`/`:68` (in `blueprint_param_space_is_construction_deterministic`): + +```rust + let a = ger40_breakout_blueprint(1.0, 15, 9, 0, Berlin).0.param_space(); + let b = ger40_breakout_blueprint(1.0, 15, 9, 0, Berlin).0.param_space(); +``` + +(The gated `run_blueprint` helper at `:78` keeps `pip_size_of(SYMBOL)` — it runs +real data and skips when absent.) + +- [ ] **Step 3: Run the gates for this task** + +Run: `cargo test -p aura-ingest --test ger40_breakout_blueprint` +Expected: PASS — the two pure tests pass; the gated fidelity test runs (archive +present) or skips (absent). + +Run: `cargo test -p aura-ingest` +Expected: PASS — `instrument_spec` is now referenced only by its own unit tests and +the two doomed table-only test files; everything compiles and is green. + +--- + +### Task 3: aura-ingest lib — remove the authored floor; relocate the seam test + +**Files:** +- Create: `crates/aura-ingest/tests/instrument_geometry.rs` +- Delete: `crates/aura-ingest/tests/instrument_geometry_crosscheck.rs` +- Delete: `crates/aura-ingest/tests/instrument_spec_deploy_metadata.rs` +- Modify: `crates/aura-ingest/src/lib.rs` (delete 361–390, 392–397, 399–436, 453–557; fix `:353`) + +- [ ] **Step 1: Relocate the surviving seam test into a new file** + +Create `crates/aura-ingest/tests/instrument_geometry.rs` holding only the seam test +from `instrument_geometry_crosscheck.rs` (which guards the *surviving* +`aura_ingest::InstrumentGeometry` re-export from silent removal), dropping the +`instrument_spec` / `VETTED_SYMBOLS` imports: + +```rust +//! Guards the surviving provider-geometry seam: `aura_ingest::instrument_geometry` +//! and the `InstrumentGeometry` re-export must stay nameable from outside the crate +//! (a downstream broker consumes them). Dropping the re-export would break this. +use aura_ingest::{default_data_server, instrument_geometry, InstrumentGeometry}; + +#[test] +fn provider_geometry_type_is_nameable_through_the_ingest_seam() { + // Name the re-exported type explicitly (not just `_`), so removing the + // `pub use data_server::meta::InstrumentGeometry` re-export fails to compile here. + let server = default_data_server(); + let _maybe: Option = instrument_geometry(&server, "EURUSD"); +} +``` + +(Match the original test's exact body if it differs; the load-bearing property is +that it names `aura_ingest::InstrumentGeometry`.) + +- [ ] **Step 2: Delete the two table-only test files** + +Run: `git rm crates/aura-ingest/tests/instrument_geometry_crosscheck.rs crates/aura-ingest/tests/instrument_spec_deploy_metadata.rs` +Expected: both files removed from the working tree. + +- [ ] **Step 3: Remove the authored floor from the lib** + +In `crates/aura-ingest/src/lib.rs` delete: `pub struct InstrumentSpec` with its doc +(361–390), `pub const VETTED_SYMBOLS` (392–397), `pub fn instrument_spec` +(399–436), and the three unit tests `instrument_spec_lookup_by_symbol`, +`instrument_spec_pip_value_matches_contract_times_pip`, +`instrument_spec_ids_are_distinct` (453–557). Keep `instrument_geometry`, the +`pub use data_server::meta::InstrumentGeometry` re-export, `default_data_server`, +and `instrument_geometry_none_for_unknown_symbol` untouched. + +- [ ] **Step 4: Fix the broken intra-doc link** + +In the surviving `instrument_geometry` doc comment (~lib.rs:353), the reference +`[instrument_spec]` now points at a removed item and breaks `cargo doc`. Reword to +drop the link, e.g. "Returns `None` for a symbol with no recorded sidecar — the +caller refuses rather than guess a pip." + +- [ ] **Step 5: Run the gates for this task** + +Run: `cargo test --workspace` +Expected: PASS — no remaining reference to `InstrumentSpec` / `instrument_spec` / +`VETTED_SYMBOLS`; the new `instrument_geometry` test runs. + +Run: `cargo doc --workspace --no-deps 2>&1` +Expected: no warnings/errors (the `:353` intra-doc link is fixed). + +--- + +### Task 4: Ledger and prose hygiene + +**Files:** +- Modify: `docs/design/INDEX.md` (~604–609, ~709, ~817–834, ~1236, and the #124 note) +- Modify: `crates/aura-engine/src/report.rs:226–227` +- Modify: `crates/aura-ingest/examples/ger40_breakout_compare.rs:8,43,50` + +- [ ] **Step 1: Rewrite the ledger passages that assert the removed mechanism** + +In `docs/design/INDEX.md`: +- The C15 realization note added in cycle 0073 (~825–834: authored floor + + cross-check + tick_size/digits): rewrite to record that the authored floor was + removed in cycle 0074, the real-path pip is sourced from the data-server geometry + sidecar (`instrument_geometry`), and the override/floor tiers of #124 are deferred + until a consumer (the Stage-2 broker) needs them, read from the sidecar then. +- The #124 hierarchy passage: record the collapse to `recorded sidecar geometry → + refuse`, tiers 1 (authored override) and 3 (authored floor) deferred. +- The #22 realization passage (~604–609) and the #106 amendment (~1236) that name + `InstrumentSpec { pip_size } + instrument_spec(symbol)`: update to "pip resolved + from the recorded geometry sidecar (`instrument_geometry`); refuse-don't-guess on + absent geometry". Reword the C8 derive note (~709, "never imports InstrumentSpec") + to not name the removed type while keeping its (still-true) point. + +- [ ] **Step 2: Fix the engine comment** + +In `crates/aura-engine/src/report.rs:226–227`, the comment "`instrument_id` is +supplied by the caller (the engine never imports `InstrumentSpec`)" names a removed +type. Reword: "the engine never imports an instrument spec from `aura-ingest`". + +- [ ] **Step 3: Fix the example doc-prose** + +In `crates/aura-ingest/examples/ger40_breakout_compare.rs` (~8, ~43, ~50), +doc-comments naming `instrument_spec`: reword to "the recorded geometry sidecar". +(The live `pip_size_of(symbol)` call ~60 already works post-Task-2.) + +- [ ] **Step 4: Final gates** + +Run: `git grep -nE '\b(InstrumentSpec|instrument_spec|VETTED_SYMBOLS)\b' -- crates/` +Expected: NO matches under `crates/` (the symbols are gone; only `fieldtests/` — a +frozen, non-workspace snapshot left untouched by design — may still name them). + +Run: `cargo test --workspace` +Expected: PASS. + +Run: `cargo clippy --workspace --all-targets -- -D warnings` +Expected: clean. + +Run: `cargo doc --workspace --no-deps 2>&1` +Expected: clean. + +--- + +## Self-review + +1. **Spec coverage:** every spec section is implemented — CLI pip resolution + + new-behaviour RED test (Task 1), example repoint + hermetic structural tests + (Task 2), lib surface shrink + deletions + seam-test relocation (Task 3), + ledger/prose updates (Task 4). ✓ +2. **Placeholder scan:** no TBD/TODO/"implement later"/"similar to Task". ✓ +3. **Type/name consistency:** `pip_or_refuse`, `instrument_geometry`, + `InstrumentGeometry`, `pip_size_of`, `default_data_server` used identically + across tasks. ✓ +4. **Step granularity:** each step is a single 2–5-min edit or one Run command; + Task 1 is RED→GREEN within the one task (canonical TDD shape). ✓ +5. **No commit steps:** none; the orchestrator commits the iter. The `git rm` in + Task 3 is a working-tree deletion, not a commit. ✓ +6. **Pin/replacement contiguity:** the refusal substring `"no recorded geometry + for symbol ''"` appears contiguously in the `pip_or_refuse` `eprintln!` + (Task 1 Step 3) and in the assertions (Task 1 Step 6); the `eprintln!` + `\`-continuation joins to one line at runtime, so the substring is contiguous in + the emitted text. ✓ +7. **Compile-gate vs deferred-caller ordering:** consumers are repointed (Tasks + 1–2) BEFORE the definitions are removed (Task 3), so every task gate is + satisfiable; Task 3's removal threads no external caller (none remain). ✓ +8. **Verification-filter strings resolve:** Task 1's filter + `run_real_nonvetted_symbol_resolves_pip_from_sidecar` is the exact new test + name; `--test cli_run` / `--test ger40_breakout_blueprint` are existing files; + Task 4's `git grep` is unfiltered-by-test. ✓