plan: 0074 provider-sourced pip, drop authored instrument floor

refs #124
2026-06-25 18:14:36 +02:00
parent eb1097d7ca
commit 4fd047adf3
1 changed files with 385 additions and 0 deletions
@@ -0,0 +1,385 @@
+# Provider-sourced pip; drop the authored instrument floor — Implementation Plan
+
+> **Parent spec:** `docs/specs/0074-pip-from-sidecar.md`
+>
+> **For agentic workers:** REQUIRED SUB-SKILL: use the `implement` skill to run
+> this plan. Steps use `- [ ]` checkboxes for tracking.
+
+**Goal:** Resolve the real-path pip from the data-server geometry sidecar and
+remove the now-redundant authored instrument floor (`InstrumentSpec`,
+`instrument_spec`, `VETTED_SYMBOLS`) and its table-only tests, keeping only the
+one datum production consumes — `pip_size`.
+
+**Architecture:** The resolution collapses to `recorded sidecar geometry →
+refuse-don't-guess`. `aura_ingest::instrument_geometry` (the #143 accessor) is the
+sole pip source. The authored floor and its 9 fields (incl. the unconsumed
+`instrument_id`) are deleted; speculative fields revive from `InstrumentGeometry`
+when the Stage-2 broker consumes them. Tasks are dependency-ordered so every
+intermediate `cargo test` gate is satisfiable: consumers are repointed (Tasks
+1–2) *before* the definitions are removed (Task 3). Each task is self-contained
+green (RED→GREEN within the task per TDD).
+
+**Tech Stack:** aura-cli (real-data pip resolution), aura-ingest (lib surface +
+example fixtures), the design ledger.
+
+---
+
+**Files this plan creates or modifies:**
+
+- Modify: `crates/aura-cli/src/main.rs:726–806` — `instrument_spec_or_refuse`→`pip_or_refuse` (sidecar-sourced `f64`); reorder both call sites (server before pip)
+- Modify: `crates/aura-cli/tests/cli_run.rs` — new RED test (non-vetted symbol resolves from sidecar) + refusal-message-pin updates + AAPL.US→NONEXISTENT premise fix
+- Modify: `crates/aura-ingest/examples/shared/breakout_real.rs:71–80` — `pip_size_of` repointed to the sidecar
+- Modify: `crates/aura-ingest/tests/ger40_breakout_blueprint.rs:40,67,68` — literal pip in the two pure-structural tests
+- Modify: `crates/aura-ingest/src/lib.rs` — remove `InstrumentSpec` (361–390), `VETTED_SYMBOLS` (392–397), `instrument_spec` (399–436), 3 unit tests (453–557); fix the `:353` intra-doc link
+- Delete: `crates/aura-ingest/tests/instrument_geometry_crosscheck.rs` (floor cross-check, moot)
+- Delete: `crates/aura-ingest/tests/instrument_spec_deploy_metadata.rs` (asserts removed fields)
+- Create: `crates/aura-ingest/tests/instrument_geometry.rs` — relocated seam test guarding the surviving `InstrumentGeometry` re-export
+- Modify: `docs/design/INDEX.md` — C15 realization note (~825–834), #124 hierarchy note, #22 realization (~604–609), #106 amendment (~1236)
+- Modify: `crates/aura-engine/src/report.rs:226–227` — comment naming the removed `InstrumentSpec`
+- Modify: `crates/aura-ingest/examples/ger40_breakout_compare.rs:8,43,50` — doc-prose naming `instrument_spec`
+
+---
+
+### Task 1: aura-cli — resolve pip from the sidecar (RED → GREEN)
+
+**Files:**
+- Modify: `crates/aura-cli/tests/cli_run.rs` (new test after `run_real_vetted_index_pip_reaches_the_emitted_manifest` ~line 249; message pins ~145, ~188, ~1189–1195)
+- Modify: `crates/aura-cli/src/main.rs:726–806`
+
+- [ ] **Step 1: Write the failing test (RED)**
+
+Add this test — it captures the directive's whole point: the authored table no
+longer gates; any symbol with recorded data resolves. USDJPY was never in the
+vetted 5-symbol table but has a sidecar (`pip 0.01`) and bars on this host.
+
+```rust
+/// Property: the authored instrument table no longer gates real runs — a symbol
+/// absent from the (removed) vetted floor but carrying a recorded geometry sidecar
+/// resolves its pip from the sidecar and runs. USDJPY (never vetted; JPY pip 0.01)
+/// renders the JPY pip in the manifest, proving the pip is sourced from the provider
+/// geometry, not a hand-authored table entry. Gated: skips cleanly when USDJPY has
+/// no local geometry/data (no sidecar → "no recorded geometry"; sidecar but no bars
+/// → "no local data").
+#[test]
+fn run_real_nonvetted_symbol_resolves_pip_from_sidecar() {
+    // FX trades continuously; the GER40 Sept-2024 window also has USDJPY bars.
+    const FROM_MS: &str = "1725148800000";
+    const TO_MS: &str = "1727740799999";
+    let out = std::process::Command::new(env!("CARGO_BIN_EXE_aura"))
+        .args(["run", "--real", "USDJPY", "--from", FROM_MS, "--to", TO_MS])
+        .output()
+        .expect("spawn aura");
+
+    // Skip on a host without USDJPY geometry/data — either refusal is a clean skip,
+    // never the old table-miss "no vetted pip".
+    if out.status.code() == Some(2) {
+        let stderr = String::from_utf8_lossy(&out.stderr);
+        if stderr.contains("no recorded geometry") || stderr.contains("no local data") {
+            eprintln!("skip: no local USDJPY geometry/data");
+            return;
+        }
+    }
+
+    assert_eq!(out.status.code(), Some(0), "USDJPY should resolve and run; exit: {:?}, stderr: {}",
+               out.status, String::from_utf8_lossy(&out.stderr));
+    let stdout = String::from_utf8(out.stdout).expect("utf-8 stdout");
+    assert!(
+        stdout.contains("pip_size=0.01"),
+        "USDJPY must render the JPY sidecar pip (0.01) in the manifest, got: {}",
+        stdout.trim_end()
+    );
+}
+```
+
+- [ ] **Step 2: Run the test to verify it fails RED**
+
+Run: `cargo test -p aura-cli --test cli_run run_real_nonvetted_symbol_resolves_pip_from_sidecar`
+Expected: FAIL — current code refuses USDJPY (`instrument_spec("USDJPY")` is `None`
+→ exit 2 with "no vetted pip/instrument spec"), which matches neither skip
+substring, so `assert_eq!(.., Some(0))` fails. (Archive present on this host, so it
+does not skip.)
+
+- [ ] **Step 3: Replace the helper with a sidecar-sourced resolver**
+
+Replace `instrument_spec_or_refuse` (main.rs:726–738):
+
+```rust
+/// Resolve the per-instrument pip from the recorded geometry sidecar, or refuse
+/// (stderr + exit 2) when the symbol has no recorded geometry — the single home of
+/// the guessed-pip refusal, shared by `open_real_source` and `from_choice`. Reads
+/// the symbol's geometry metadata (not bar data) before the run source opens, so an
+/// instrument with no recorded geometry refuses without a guessed pip; the pip is
+/// the provider's recorded value, honest by construction.
+fn pip_or_refuse(server: &std::sync::Arc<data_server::DataServer>, symbol: &str) -> f64 {
+    match aura_ingest::instrument_geometry(server, symbol) {
+        Some(geo) => geo.pip_size,
+        None => {
+            eprintln!(
+                "aura: no recorded geometry for symbol '{symbol}' at {} — \
+                 refusing to run a real instrument with a guessed pip",
+                data_server::DEFAULT_DATA_PATH
+            );
+            std::process::exit(2);
+        }
+    }
+}
+```
+
+- [ ] **Step 4: Reorder `open_real_source` (server before pip)**
+
+In `open_real_source` (main.rs:767–787): build the server first, resolve the pip,
+return `pip` in the tuple (was `spec.pip_size`).
+
+```rust
+    // Per-instrument pip from the recorded sidecar; resolved BEFORE bar-data access
+    // so an instrument with no geometry refuses without touching the archive.
+    let server = std::sync::Arc::new(data_server::DataServer::new(data_server::DEFAULT_DATA_PATH));
+    let pip = pip_or_refuse(&server, symbol);
+    if !server.has_symbol(symbol) {
+        no_real_data(symbol);
+    }
+    // ... probe_window + M1FieldSource::open unchanged ...
+    (source, window, pip)
+```
+
+- [ ] **Step 5: Reorder `DataSource::from_choice`**
+
+In `from_choice` (main.rs:794–805): build the server first, resolve the pip, set
+`pip` in `DataSource::Real`. Update the doc-comment at :792
+(`instrument_spec_or_refuse` → `pip_or_refuse`).
+
+```rust
+            DataChoice::Real { symbol, from_ms, to_ms } => {
+                let server = std::sync::Arc::new(data_server::DataServer::new(data_server::DEFAULT_DATA_PATH));
+                let pip = pip_or_refuse(&server, &symbol);
+                if !server.has_symbol(&symbol) {
+                    no_real_data(&symbol);
+                }
+                DataSource::Real { server, symbol, from_ms, to_ms, pip }
+            }
+```
+
+- [ ] **Step 6: Update the refusal-message pins + the AAPL.US premise**
+
+In `cli_run.rs`:
+- `run_real_unspecced_symbol_refuses_with_exit_2` (~145): substring
+  `"no vetted pip/instrument spec for symbol 'NONEXISTENT'"` →
+  `"no recorded geometry for symbol 'NONEXISTENT'"`; update the comment ~135.
+- `run_real_vetted_symbol_never_hits_the_pip_refusal` (~188):
+  `!stderr.contains("no vetted pip/instrument spec")` →
+  `!stderr.contains("no recorded geometry")`; update doc-prose ~171–179.
+- `sweep_real_unspecced_symbol_refuses_with_exit_2` (~1189–1195): change the symbol
+  from `"AAPL.US"` to `"NONEXISTENT"` (AAPL.US now HAS a sidecar and would resolve,
+  not refuse), substring `"no vetted pip"` → `"no recorded geometry"`; update the
+  doc-comment ~1183–1187.
+
+- [ ] **Step 7: Run the gates for this task**
+
+Run: `cargo test -p aura-cli --test cli_run`
+Expected: PASS — `run_real_nonvetted_symbol_resolves_pip_from_sidecar` is GREEN
+(USDJPY resolves), `run_real_vetted_index_pip_reaches_the_emitted_manifest` still
+GREEN (GER40 sidecar pip == 1.0, byte-identical manifest), and all three updated
+refusal tests pass.
+
+---
+
+### Task 2: aura-ingest examples — repoint `pip_size_of`; literal pip in pure tests
+
+**Files:**
+- Modify: `crates/aura-ingest/examples/shared/breakout_real.rs:71–80`
+- Modify: `crates/aura-ingest/tests/ger40_breakout_blueprint.rs:40,67,68`
+
+- [ ] **Step 1: Repoint `pip_size_of` to the sidecar**
+
+Replace `pip_size_of` (breakout_real.rs:71–80):
+
+```rust
+/// The example's pip divisor, sourced from the recorded geometry sidecar — the same
+/// provider truth the CLI `--real` path uses, now provider-sourced. Panics on a
+/// symbol with no sidecar (the examples only run known instruments with local data).
+pub fn pip_size_of(symbol: &str) -> f64 {
+    aura_ingest::instrument_geometry(&aura_ingest::default_data_server(), symbol)
+        .unwrap_or_else(|| panic!("no recorded geometry sidecar for {symbol}"))
+        .pip_size
+}
+```
+
+- [ ] **Step 2: Literal pip in the two pure-structural blueprint tests**
+
+In `ger40_breakout_blueprint.rs`, the two tests asserting `param_space()` topology
+are contracted to run with no archive and must not read the sidecar. Replace
+`pip_size_of(SYMBOL)` with a literal at `:40` (in
+`param_space_is_exactly_entry_and_exit_bar_targets`):
+
+```rust
+    // pure-structural: the pip is a throwaway sizing arg, never asserted here, so a
+    // literal keeps this archive-independent (the topology assertion is the point).
+    let (bp, _taps) = ger40_breakout_blueprint(1.0, 15, 9, 0, Berlin);
+```
+
+and at `:67`/`:68` (in `blueprint_param_space_is_construction_deterministic`):
+
+```rust
+    let a = ger40_breakout_blueprint(1.0, 15, 9, 0, Berlin).0.param_space();
+    let b = ger40_breakout_blueprint(1.0, 15, 9, 0, Berlin).0.param_space();
+```
+
+(The gated `run_blueprint` helper at `:78` keeps `pip_size_of(SYMBOL)` — it runs
+real data and skips when absent.)
+
+- [ ] **Step 3: Run the gates for this task**
+
+Run: `cargo test -p aura-ingest --test ger40_breakout_blueprint`
+Expected: PASS — the two pure tests pass; the gated fidelity test runs (archive
+present) or skips (absent).
+
+Run: `cargo test -p aura-ingest`
+Expected: PASS — `instrument_spec` is now referenced only by its own unit tests and
+the two doomed table-only test files; everything compiles and is green.
+
+---
+
+### Task 3: aura-ingest lib — remove the authored floor; relocate the seam test
+
+**Files:**
+- Create: `crates/aura-ingest/tests/instrument_geometry.rs`
+- Delete: `crates/aura-ingest/tests/instrument_geometry_crosscheck.rs`
+- Delete: `crates/aura-ingest/tests/instrument_spec_deploy_metadata.rs`
+- Modify: `crates/aura-ingest/src/lib.rs` (delete 361–390, 392–397, 399–436, 453–557; fix `:353`)
+
+- [ ] **Step 1: Relocate the surviving seam test into a new file**
+
+Create `crates/aura-ingest/tests/instrument_geometry.rs` holding only the seam test
+from `instrument_geometry_crosscheck.rs` (which guards the *surviving*
+`aura_ingest::InstrumentGeometry` re-export from silent removal), dropping the
+`instrument_spec` / `VETTED_SYMBOLS` imports:
+
+```rust
+//! Guards the surviving provider-geometry seam: `aura_ingest::instrument_geometry`
+//! and the `InstrumentGeometry` re-export must stay nameable from outside the crate
+//! (a downstream broker consumes them). Dropping the re-export would break this.
+use aura_ingest::{default_data_server, instrument_geometry, InstrumentGeometry};
+
+#[test]
+fn provider_geometry_type_is_nameable_through_the_ingest_seam() {
+    // Name the re-exported type explicitly (not just `_`), so removing the
+    // `pub use data_server::meta::InstrumentGeometry` re-export fails to compile here.
+    let server = default_data_server();
+    let _maybe: Option<InstrumentGeometry> = instrument_geometry(&server, "EURUSD");
+}
+```
+
+(Match the original test's exact body if it differs; the load-bearing property is
+that it names `aura_ingest::InstrumentGeometry`.)
+
+- [ ] **Step 2: Delete the two table-only test files**
+
+Run: `git rm crates/aura-ingest/tests/instrument_geometry_crosscheck.rs crates/aura-ingest/tests/instrument_spec_deploy_metadata.rs`
+Expected: both files removed from the working tree.
+
+- [ ] **Step 3: Remove the authored floor from the lib**
+
+In `crates/aura-ingest/src/lib.rs` delete: `pub struct InstrumentSpec` with its doc
+(361–390), `pub const VETTED_SYMBOLS` (392–397), `pub fn instrument_spec`
+(399–436), and the three unit tests `instrument_spec_lookup_by_symbol`,
+`instrument_spec_pip_value_matches_contract_times_pip`,
+`instrument_spec_ids_are_distinct` (453–557). Keep `instrument_geometry`, the
+`pub use data_server::meta::InstrumentGeometry` re-export, `default_data_server`,
+and `instrument_geometry_none_for_unknown_symbol` untouched.
+
+- [ ] **Step 4: Fix the broken intra-doc link**
+
+In the surviving `instrument_geometry` doc comment (~lib.rs:353), the reference
+`[instrument_spec]` now points at a removed item and breaks `cargo doc`. Reword to
+drop the link, e.g. "Returns `None` for a symbol with no recorded sidecar — the
+caller refuses rather than guess a pip."
+
+- [ ] **Step 5: Run the gates for this task**
+
+Run: `cargo test --workspace`
+Expected: PASS — no remaining reference to `InstrumentSpec` / `instrument_spec` /
+`VETTED_SYMBOLS`; the new `instrument_geometry` test runs.
+
+Run: `cargo doc --workspace --no-deps 2>&1`
+Expected: no warnings/errors (the `:353` intra-doc link is fixed).
+
+---
+
+### Task 4: Ledger and prose hygiene
+
+**Files:**
+- Modify: `docs/design/INDEX.md` (~604–609, ~709, ~817–834, ~1236, and the #124 note)
+- Modify: `crates/aura-engine/src/report.rs:226–227`
+- Modify: `crates/aura-ingest/examples/ger40_breakout_compare.rs:8,43,50`
+
+- [ ] **Step 1: Rewrite the ledger passages that assert the removed mechanism**
+
+In `docs/design/INDEX.md`:
+- The C15 realization note added in cycle 0073 (~825–834: authored floor +
+  cross-check + tick_size/digits): rewrite to record that the authored floor was
+  removed in cycle 0074, the real-path pip is sourced from the data-server geometry
+  sidecar (`instrument_geometry`), and the override/floor tiers of #124 are deferred
+  until a consumer (the Stage-2 broker) needs them, read from the sidecar then.
+- The #124 hierarchy passage: record the collapse to `recorded sidecar geometry →
+  refuse`, tiers 1 (authored override) and 3 (authored floor) deferred.
+- The #22 realization passage (~604–609) and the #106 amendment (~1236) that name
+  `InstrumentSpec { pip_size } + instrument_spec(symbol)`: update to "pip resolved
+  from the recorded geometry sidecar (`instrument_geometry`); refuse-don't-guess on
+  absent geometry". Reword the C8 derive note (~709, "never imports InstrumentSpec")
+  to not name the removed type while keeping its (still-true) point.
+
+- [ ] **Step 2: Fix the engine comment**
+
+In `crates/aura-engine/src/report.rs:226–227`, the comment "`instrument_id` is
+supplied by the caller (the engine never imports `InstrumentSpec`)" names a removed
+type. Reword: "the engine never imports an instrument spec from `aura-ingest`".
+
+- [ ] **Step 3: Fix the example doc-prose**
+
+In `crates/aura-ingest/examples/ger40_breakout_compare.rs` (~8, ~43, ~50),
+doc-comments naming `instrument_spec`: reword to "the recorded geometry sidecar".
+(The live `pip_size_of(symbol)` call ~60 already works post-Task-2.)
+
+- [ ] **Step 4: Final gates**
+
+Run: `git grep -nE '\b(InstrumentSpec|instrument_spec|VETTED_SYMBOLS)\b' -- crates/`
+Expected: NO matches under `crates/` (the symbols are gone; only `fieldtests/` — a
+frozen, non-workspace snapshot left untouched by design — may still name them).
+
+Run: `cargo test --workspace`
+Expected: PASS.
+
+Run: `cargo clippy --workspace --all-targets -- -D warnings`
+Expected: clean.
+
+Run: `cargo doc --workspace --no-deps 2>&1`
+Expected: clean.
+
+---
+
+## Self-review
+
+1. **Spec coverage:** every spec section is implemented — CLI pip resolution +
+   new-behaviour RED test (Task 1), example repoint + hermetic structural tests
+   (Task 2), lib surface shrink + deletions + seam-test relocation (Task 3),
+   ledger/prose updates (Task 4). ✓
+2. **Placeholder scan:** no TBD/TODO/"implement later"/"similar to Task". ✓
+3. **Type/name consistency:** `pip_or_refuse`, `instrument_geometry`,
+   `InstrumentGeometry`, `pip_size_of`, `default_data_server` used identically
+   across tasks. ✓
+4. **Step granularity:** each step is a single 2–5-min edit or one Run command;
+   Task 1 is RED→GREEN within the one task (canonical TDD shape). ✓
+5. **No commit steps:** none; the orchestrator commits the iter. The `git rm` in
+   Task 3 is a working-tree deletion, not a commit. ✓
+6. **Pin/replacement contiguity:** the refusal substring `"no recorded geometry
+   for symbol '<sym>'"` appears contiguously in the `pip_or_refuse` `eprintln!`
+   (Task 1 Step 3) and in the assertions (Task 1 Step 6); the `eprintln!`
+   `\`-continuation joins to one line at runtime, so the substring is contiguous in
+   the emitted text. ✓
+7. **Compile-gate vs deferred-caller ordering:** consumers are repointed (Tasks
+   1–2) BEFORE the definitions are removed (Task 3), so every task gate is
+   satisfiable; Task 3's removal threads no external caller (none remain). ✓
+8. **Verification-filter strings resolve:** Task 1's filter
+   `run_real_nonvetted_symbol_resolves_pip_from_sidecar` is the exact new test
+   name; `--test cli_run` / `--test ger40_breakout_blueprint` are existing files;
+   Task 4's `git grep` is unfiltered-by-test. ✓