Aura/docs/plans/0074-pip-from-sidecar.md

Provider-sourced pip; drop the authored instrument floor — Implementation Plan

Parent spec: docs/specs/0074-pip-from-sidecar.md

For agentic workers: REQUIRED SUB-SKILL: use the implement skill to run this plan. Steps use - [ ] checkboxes for tracking.

Goal: Resolve the real-path pip from the data-server geometry sidecar and remove the now-redundant authored instrument floor (InstrumentSpec, instrument_spec, VETTED_SYMBOLS) and its table-only tests, keeping only the one datum production consumes — pip_size.

Architecture: The resolution collapses to recorded sidecar geometry → refuse-don't-guess. aura_ingest::instrument_geometry (the #143 accessor) is the sole pip source. The authored floor and its 9 fields (incl. the unconsumed instrument_id) are deleted; speculative fields revive from InstrumentGeometry when the Stage-2 broker consumes them. Tasks are dependency-ordered so every intermediate cargo test gate is satisfiable: consumers are repointed (Tasks 1–2) before the definitions are removed (Task 3). Each task is self-contained green (RED→GREEN within the task per TDD).

Tech Stack: aura-cli (real-data pip resolution), aura-ingest (lib surface + example fixtures), the design ledger.

---

Files this plan creates or modifies:

• Modify: crates/aura-cli/src/main.rs:726–806 — instrument_spec_or_refuse→pip_or_refuse (sidecar-sourced f64); reorder both call sites (server before pip)
• Modify: crates/aura-cli/tests/cli_run.rs — new RED test (non-vetted symbol resolves from sidecar) + refusal-message-pin updates + AAPL.US→NONEXISTENT premise fix
• Modify: crates/aura-ingest/examples/shared/breakout_real.rs:71–80 — pip_size_of repointed to the sidecar
• Modify: crates/aura-ingest/tests/ger40_breakout_blueprint.rs:40,67,68 — literal pip in the two pure-structural tests
• Modify: crates/aura-ingest/src/lib.rs — remove InstrumentSpec (361–390), VETTED_SYMBOLS (392–397), instrument_spec (399–436), 3 unit tests (453–557); fix the :353 intra-doc link
• Delete: crates/aura-ingest/tests/instrument_geometry_crosscheck.rs (floor cross-check, moot)
• Delete: crates/aura-ingest/tests/instrument_spec_deploy_metadata.rs (asserts removed fields)
• Create: crates/aura-ingest/tests/instrument_geometry.rs — relocated seam test guarding the surviving InstrumentGeometry re-export
• Modify: docs/design/INDEX.md — C15 realization note (~825–834), #124 hierarchy note, #22 realization (~604–609), #106 amendment (~1236)
• Modify: crates/aura-engine/src/report.rs:226–227 — comment naming the removed InstrumentSpec
• Modify: crates/aura-ingest/examples/ger40_breakout_compare.rs:8,43,50 — doc-prose naming instrument_spec

---

Task 1: aura-cli — resolve pip from the sidecar (RED → GREEN)

Files:

• Modify: crates/aura-cli/tests/cli_run.rs (new test after run_real_vetted_index_pip_reaches_the_emitted_manifest ~line 249; message pins ~145, ~188, ~1189–1195)

• Modify: crates/aura-cli/src/main.rs:726–806

• Step 1: Write the failing test (RED)

Add this test — it captures the directive's whole point: the authored table no longer gates; any symbol with recorded data resolves. USDJPY was never in the vetted 5-symbol table but has a sidecar (pip 0.01) and bars on this host.

/// Property: the authored instrument table no longer gates real runs — a symbol
/// absent from the (removed) vetted floor but carrying a recorded geometry sidecar
/// resolves its pip from the sidecar and runs. USDJPY (never vetted; JPY pip 0.01)
/// renders the JPY pip in the manifest, proving the pip is sourced from the provider
/// geometry, not a hand-authored table entry. Gated: skips cleanly when USDJPY has
/// no local geometry/data (no sidecar → "no recorded geometry"; sidecar but no bars
/// → "no local data").
#[test]
fn run_real_nonvetted_symbol_resolves_pip_from_sidecar() {
    // FX trades continuously; the GER40 Sept-2024 window also has USDJPY bars.
    const FROM_MS: &str = "1725148800000";
    const TO_MS: &str = "1727740799999";
    let out = std::process::Command::new(env!("CARGO_BIN_EXE_aura"))
        .args(["run", "--real", "USDJPY", "--from", FROM_MS, "--to", TO_MS])
        .output()
        .expect("spawn aura");

    // Skip on a host without USDJPY geometry/data — either refusal is a clean skip,
    // never the old table-miss "no vetted pip".
    if out.status.code() == Some(2) {
        let stderr = String::from_utf8_lossy(&out.stderr);
        if stderr.contains("no recorded geometry") || stderr.contains("no local data") {
            eprintln!("skip: no local USDJPY geometry/data");
            return;
        }
    }

    assert_eq!(out.status.code(), Some(0), "USDJPY should resolve and run; exit: {:?}, stderr: {}",
               out.status, String::from_utf8_lossy(&out.stderr));
    let stdout = String::from_utf8(out.stdout).expect("utf-8 stdout");
    assert!(
        stdout.contains("pip_size=0.01"),
        "USDJPY must render the JPY sidecar pip (0.01) in the manifest, got: {}",
        stdout.trim_end()
    );
}

• Step 2: Run the test to verify it fails RED

Run: cargo test -p aura-cli --test cli_run run_real_nonvetted_symbol_resolves_pip_from_sidecar Expected: FAIL — current code refuses USDJPY (instrument_spec("USDJPY") is None → exit 2 with "no vetted pip/instrument spec"), which matches neither skip substring, so assert_eq!(.., Some(0)) fails. (Archive present on this host, so it does not skip.)

• Step 3: Replace the helper with a sidecar-sourced resolver

Replace instrument_spec_or_refuse (main.rs:726–738):

/// Resolve the per-instrument pip from the recorded geometry sidecar, or refuse
/// (stderr + exit 2) when the symbol has no recorded geometry — the single home of
/// the guessed-pip refusal, shared by `open_real_source` and `from_choice`. Reads
/// the symbol's geometry metadata (not bar data) before the run source opens, so an
/// instrument with no recorded geometry refuses without a guessed pip; the pip is
/// the provider's recorded value, honest by construction.
fn pip_or_refuse(server: &std::sync::Arc<data_server::DataServer>, symbol: &str) -> f64 {
    match aura_ingest::instrument_geometry(server, symbol) {
        Some(geo) => geo.pip_size,
        None => {
            eprintln!(
                "aura: no recorded geometry for symbol '{symbol}' at {} — \
                 refusing to run a real instrument with a guessed pip",
                data_server::DEFAULT_DATA_PATH
            );
            std::process::exit(2);
        }
    }
}

• Step 4: Reorder open_real_source (server before pip)

In open_real_source (main.rs:767–787): build the server first, resolve the pip, return pip in the tuple (was spec.pip_size).

    // Per-instrument pip from the recorded sidecar; resolved BEFORE bar-data access
    // so an instrument with no geometry refuses without touching the archive.
    let server = std::sync::Arc::new(data_server::DataServer::new(data_server::DEFAULT_DATA_PATH));
    let pip = pip_or_refuse(&server, symbol);
    if !server.has_symbol(symbol) {
        no_real_data(symbol);
    }
    // ... probe_window + M1FieldSource::open unchanged ...
    (source, window, pip)

• Step 5: Reorder DataSource::from_choice

In from_choice (main.rs:794–805): build the server first, resolve the pip, set pip in DataSource::Real. Update the doc-comment at :792 (instrument_spec_or_refuse → pip_or_refuse).

            DataChoice::Real { symbol, from_ms, to_ms } => {
                let server = std::sync::Arc::new(data_server::DataServer::new(data_server::DEFAULT_DATA_PATH));
                let pip = pip_or_refuse(&server, &symbol);
                if !server.has_symbol(&symbol) {
                    no_real_data(&symbol);
                }
                DataSource::Real { server, symbol, from_ms, to_ms, pip }
            }

• Step 6: Update the refusal-message pins + the AAPL.US premise

In cli_run.rs:

• run_real_unspecced_symbol_refuses_with_exit_2 (~145): substring "no vetted pip/instrument spec for symbol 'NONEXISTENT'" → "no recorded geometry for symbol 'NONEXISTENT'"; update the comment ~135.

• run_real_vetted_symbol_never_hits_the_pip_refusal (~188): !stderr.contains("no vetted pip/instrument spec") → !stderr.contains("no recorded geometry"); update doc-prose ~171–179.

• sweep_real_unspecced_symbol_refuses_with_exit_2 (~1189–1195): change the symbol from "AAPL.US" to "NONEXISTENT" (AAPL.US now HAS a sidecar and would resolve, not refuse), substring "no vetted pip" → "no recorded geometry"; update the doc-comment ~1183–1187.

• Step 7: Run the gates for this task

Run: cargo test -p aura-cli --test cli_run Expected: PASS — run_real_nonvetted_symbol_resolves_pip_from_sidecar is GREEN (USDJPY resolves), run_real_vetted_index_pip_reaches_the_emitted_manifest still GREEN (GER40 sidecar pip == 1.0, byte-identical manifest), and all three updated refusal tests pass.

---

Task 2: aura-ingest examples — repoint pip_size_of; literal pip in pure tests

Files:

• Modify: crates/aura-ingest/examples/shared/breakout_real.rs:71–80

• Modify: crates/aura-ingest/tests/ger40_breakout_blueprint.rs:40,67,68

• Step 1: Repoint pip_size_of to the sidecar

Replace pip_size_of (breakout_real.rs:71–80):

/// The example's pip divisor, sourced from the recorded geometry sidecar — the same
/// provider truth the CLI `--real` path uses, now provider-sourced. Panics on a
/// symbol with no sidecar (the examples only run known instruments with local data).
pub fn pip_size_of(symbol: &str) -> f64 {
    aura_ingest::instrument_geometry(&aura_ingest::default_data_server(), symbol)
        .unwrap_or_else(|| panic!("no recorded geometry sidecar for {symbol}"))
        .pip_size
}

• Step 2: Literal pip in the two pure-structural blueprint tests

In ger40_breakout_blueprint.rs, the two tests asserting param_space() topology are contracted to run with no archive and must not read the sidecar. Replace pip_size_of(SYMBOL) with a literal at :40 (in param_space_is_exactly_entry_and_exit_bar_targets):

    // pure-structural: the pip is a throwaway sizing arg, never asserted here, so a
    // literal keeps this archive-independent (the topology assertion is the point).
    let (bp, _taps) = ger40_breakout_blueprint(1.0, 15, 9, 0, Berlin);

and at :67/:68 (in blueprint_param_space_is_construction_deterministic):

    let a = ger40_breakout_blueprint(1.0, 15, 9, 0, Berlin).0.param_space();
    let b = ger40_breakout_blueprint(1.0, 15, 9, 0, Berlin).0.param_space();

(The gated run_blueprint helper at :78 keeps pip_size_of(SYMBOL) — it runs real data and skips when absent.)

• Step 3: Run the gates for this task

Run: cargo test -p aura-ingest --test ger40_breakout_blueprint Expected: PASS — the two pure tests pass; the gated fidelity test runs (archive present) or skips (absent).

Run: cargo test -p aura-ingest Expected: PASS — instrument_spec is now referenced only by its own unit tests and the two doomed table-only test files; everything compiles and is green.

---

Task 3: aura-ingest lib — remove the authored floor; relocate the seam test

Files:

• Create: crates/aura-ingest/tests/instrument_geometry.rs

• Delete: crates/aura-ingest/tests/instrument_geometry_crosscheck.rs

• Delete: crates/aura-ingest/tests/instrument_spec_deploy_metadata.rs

• Modify: crates/aura-ingest/src/lib.rs (delete 361–390, 392–397, 399–436, 453–557; fix :353)

• Step 1: Relocate the surviving seam test into a new file

Create crates/aura-ingest/tests/instrument_geometry.rs holding only the seam test from instrument_geometry_crosscheck.rs (which guards the surviving aura_ingest::InstrumentGeometry re-export from silent removal), dropping the instrument_spec / VETTED_SYMBOLS imports:

//! Guards the surviving provider-geometry seam: `aura_ingest::instrument_geometry`
//! and the `InstrumentGeometry` re-export must stay nameable from outside the crate
//! (a downstream broker consumes them). Dropping the re-export would break this.
use aura_ingest::{default_data_server, instrument_geometry, InstrumentGeometry};

#[test]
fn provider_geometry_type_is_nameable_through_the_ingest_seam() {
    // Name the re-exported type explicitly (not just `_`), so removing the
    // `pub use data_server::meta::InstrumentGeometry` re-export fails to compile here.
    let server = default_data_server();
    let _maybe: Option<InstrumentGeometry> = instrument_geometry(&server, "EURUSD");
}

(Match the original test's exact body if it differs; the load-bearing property is that it names aura_ingest::InstrumentGeometry.)

• Step 2: Delete the two table-only test files

Run: git rm crates/aura-ingest/tests/instrument_geometry_crosscheck.rs crates/aura-ingest/tests/instrument_spec_deploy_metadata.rs Expected: both files removed from the working tree.

• Step 3: Remove the authored floor from the lib

In crates/aura-ingest/src/lib.rs delete: pub struct InstrumentSpec with its doc (361–390), pub const VETTED_SYMBOLS (392–397), pub fn instrument_spec (399–436), and the three unit tests instrument_spec_lookup_by_symbol, instrument_spec_pip_value_matches_contract_times_pip, instrument_spec_ids_are_distinct (453–557). Keep instrument_geometry, the pub use data_server::meta::InstrumentGeometry re-export, default_data_server, and instrument_geometry_none_for_unknown_symbol untouched.

• Step 4: Fix the broken intra-doc link

In the surviving instrument_geometry doc comment (~lib.rs:353), the reference [instrument_spec] now points at a removed item and breaks cargo doc. Reword to drop the link, e.g. "Returns None for a symbol with no recorded sidecar — the caller refuses rather than guess a pip."

• Step 5: Run the gates for this task

Run: cargo test --workspace Expected: PASS — no remaining reference to InstrumentSpec / instrument_spec / VETTED_SYMBOLS; the new instrument_geometry test runs.

Run: cargo doc --workspace --no-deps 2>&1 Expected: no warnings/errors (the :353 intra-doc link is fixed).

---

Task 4: Ledger and prose hygiene

Files:

• Modify: docs/design/INDEX.md (~604–609, ~709, ~817–834, ~1236, and the #124 note)

• Modify: crates/aura-engine/src/report.rs:226–227

• Modify: crates/aura-ingest/examples/ger40_breakout_compare.rs:8,43,50

• Step 1: Rewrite the ledger passages that assert the removed mechanism

In docs/design/INDEX.md:

• The C15 realization note added in cycle 0073 (~825–834: authored floor + cross-check + tick_size/digits): rewrite to record that the authored floor was removed in cycle 0074, the real-path pip is sourced from the data-server geometry sidecar (instrument_geometry), and the override/floor tiers of #124 are deferred until a consumer (the Stage-2 broker) needs them, read from the sidecar then.

• The #124 hierarchy passage: record the collapse to recorded sidecar geometry → refuse, tiers 1 (authored override) and 3 (authored floor) deferred.

• The #22 realization passage (~604–609) and the #106 amendment (~1236) that name InstrumentSpec { pip_size } + instrument_spec(symbol): update to "pip resolved from the recorded geometry sidecar (instrument_geometry); refuse-don't-guess on absent geometry". Reword the C8 derive note (~709, "never imports InstrumentSpec") to not name the removed type while keeping its (still-true) point.

• Step 2: Fix the engine comment

In crates/aura-engine/src/report.rs:226–227, the comment "instrument_id is supplied by the caller (the engine never imports InstrumentSpec)" names a removed type. Reword: "the engine never imports an instrument spec from aura-ingest".

• Step 3: Fix the example doc-prose

In crates/aura-ingest/examples/ger40_breakout_compare.rs (~8, ~43, ~50), doc-comments naming instrument_spec: reword to "the recorded geometry sidecar". (The live pip_size_of(symbol) call ~60 already works post-Task-2.)

• Step 4: Final gates

Run: git grep -nE '\b(InstrumentSpec|instrument_spec|VETTED_SYMBOLS)\b' -- crates/ Expected: NO matches under crates/ (the symbols are gone; only fieldtests/ — a frozen, non-workspace snapshot left untouched by design — may still name them).

Run: cargo test --workspace Expected: PASS.

Run: cargo clippy --workspace --all-targets -- -D warnings Expected: clean.

Run: cargo doc --workspace --no-deps 2>&1 Expected: clean.

---

Self-review

1. Spec coverage: every spec section is implemented — CLI pip resolution + new-behaviour RED test (Task 1), example repoint + hermetic structural tests (Task 2), lib surface shrink + deletions + seam-test relocation (Task 3), ledger/prose updates (Task 4). ✓
2. Placeholder scan: no TBD/TODO/"implement later"/"similar to Task". ✓
3. Type/name consistency: pip_or_refuse, instrument_geometry, InstrumentGeometry, pip_size_of, default_data_server used identically across tasks. ✓
4. Step granularity: each step is a single 2–5-min edit or one Run command; Task 1 is RED→GREEN within the one task (canonical TDD shape). ✓
5. No commit steps: none; the orchestrator commits the iter. The git rm in Task 3 is a working-tree deletion, not a commit. ✓
6. Pin/replacement contiguity: the refusal substring "no recorded geometry for symbol '<sym>'" appears contiguously in the pip_or_refuse eprintln! (Task 1 Step 3) and in the assertions (Task 1 Step 6); the eprintln! \-continuation joins to one line at runtime, so the substring is contiguous in the emitted text. ✓
7. Compile-gate vs deferred-caller ordering: consumers are repointed (Tasks 1–2) BEFORE the definitions are removed (Task 3), so every task gate is satisfiable; Task 3's removal threads no external caller (none remain). ✓
8. Verification-filter strings resolve: Task 1's filter run_real_nonvetted_symbol_resolves_pip_from_sidecar is the exact new test name; --test cli_run / --test ger40_breakout_blueprint are existing files; Task 4's git grep is unfiltered-by-test. ✓