diff --git a/docs/specs/0074-pip-from-sidecar.md b/docs/specs/0074-pip-from-sidecar.md new file mode 100644 index 0000000..eace519 --- /dev/null +++ b/docs/specs/0074-pip-from-sidecar.md @@ -0,0 +1,238 @@ +# Provider-sourced pip; drop the authored instrument floor — Design Spec + +**Date:** 2026-06-25 +**Status:** Draft — awaiting user spec review +**Authors:** orchestrator + Claude + +## Goal + +aura must define and take over only the instrument reference data it currently +consumes, and source it from the data-server geometry sidecars rather than a +hand-maintained table. A grep of every `InstrumentSpec` field *read* across +`crates/` shows production code consumes exactly one datum — `pip_size` — at the +real-data path. Every other field (`contract_size`, `pip_value_per_lot`, +`min_lot`, `lot_step`, `quote_currency`, `tick_size`, `digits`, **and +`instrument_id`**) is constructed but never read outside the table's own tests. + +This cycle resolves the pip from the per-symbol sidecar +(`instrument_geometry(server, symbol).pip_size`) and removes the now-redundant +authored floor (`InstrumentSpec`, `instrument_spec()`, `VETTED_SYMBOLS`) together +with the tests that exist only to validate it. The decision and its grounding are +recorded on the reference issue (#124, reconciliation comment). + +Deferred, not deleted (per the user directive "build it back in when the broker +needs it"): the speculative fields are revived from `InstrumentGeometry` — which +already carries `digits`, `lotSize`, `tickSize`, `quoteAsset` — when the Stage-2 +realistic-broker milestone actually consumes them. No knowledge is lost; the +sidecar remains the source. + +## Architecture + +The instrument-reference resolution hierarchy of #124 (authored override > +recorded metadata > authored floor > refuse) collapses, for what is built now, to +its two ends: **recorded sidecar geometry → refuse-don't-guess**. The middle +authored floor (tier 3) and the authored override (tier 1) are future work, built +when a consumer needs them. The `instrument_geometry` accessor and the +`InstrumentGeometry` re-export added in #143 are the surviving seam and become the +sole pip source. + +The `instrument_id` question (the directive's "find out"): the position-event +table needs an `instrument_id`, but it already takes one as a **caller-supplied +`i64` parameter** — `derive_position_events(record, instrument_id)` +(report.rs:231) — and the engine deliberately never imports `InstrumentSpec` +(report.rs:226). `InstrumentSpec.instrument_id` has no production reader; it is +dropped with the struct. No symbol→id registry is consumed anywhere today, so +none is built; when one is needed it is added then. + +Invariants preserved: **refuse-don't-guess** (a symbol with no sidecar → exit 2, +unchanged); **honest-pip-by-construction** (now the provider's recorded value +instead of a hand-authored one — strictly more honest); **determinism / C6** (the +sidecar is recorded data read once at bootstrap, never a live mid-replay call); +the engine's `(Timestamp, Scalar)` seam is untouched (geometry stays beside the +hot path, C7/C15). + +## Concrete code shapes + +### Delivered — CLI pip resolution (the production path) + +`crates/aura-cli/src/main.rs`. Before — table lookup, server-free, refuses on an +un-vetted symbol: + +```rust +// ~line 730 +fn instrument_spec_or_refuse(symbol: &str) -> aura_ingest::InstrumentSpec { + match aura_ingest::instrument_spec(symbol) { + Some(s) => s, + None => { + eprintln!("aura: no vetted pip/instrument spec for symbol '{symbol}' — \ + refusing to run a real instrument with a guessed pip \ + (add it to the instrument table)"); + std::process::exit(2); + } + } +} +``` + +After — resolve the pip from the recorded sidecar; refuse when absent. The pip is +the only datum any caller used, so the helper returns `f64`: + +```rust +// the single home of the guessed-pip refusal, now sidecar-sourced. Reads the +// per-symbol geometry metadata (not bar data) BEFORE the run source opens, so an +// instrument with no recorded geometry refuses without touching the archive — +// the pip stays honest by construction (the provider's value, never guessed). +fn pip_or_refuse(server: &std::sync::Arc, symbol: &str) -> f64 { + match aura_ingest::instrument_geometry(server, symbol) { + Some(geo) => geo.pip_size, + None => { + eprintln!("aura: no recorded geometry for symbol '{symbol}' at {} — \ + refusing to run a real instrument with a guessed pip", + data_server::DEFAULT_DATA_PATH); + std::process::exit(2); + } + } +} +``` + +Both call sites build the server first, then resolve the pip (the refusal stays +ahead of bar-data access). `open_real_source` (~line 767): + +```rust +// before: spec lookup, THEN server +// let spec = instrument_spec_or_refuse(symbol); +// let server = Arc::new(DataServer::new(DEFAULT_DATA_PATH)); +// after: server first, then pip from its sidecar +let server = std::sync::Arc::new(data_server::DataServer::new(data_server::DEFAULT_DATA_PATH)); +let pip = pip_or_refuse(&server, symbol); +if !server.has_symbol(symbol) { no_real_data(symbol); } +// ... probe_window + open as today; return (source, window, pip) +``` + +`DataSource::from_choice` (~line 794) takes the same reorder: build `server`, +`let pip = pip_or_refuse(&server, &symbol);`, then `has_symbol`, then +`DataSource::Real { server, symbol, from_ms, to_ms, pip }`. + +### Delivered — aura-ingest library surface + +`crates/aura-ingest/src/lib.rs`. **Removed:** `pub struct InstrumentSpec` (all 9 +fields), `pub fn instrument_spec`, `pub const VETTED_SYMBOLS`, and the three unit +tests that only exercise them (`instrument_spec_lookup_by_symbol`, +`instrument_spec_pip_value_matches_contract_times_pip`, +`instrument_spec_ids_are_distinct`). **Kept unchanged:** the pip source — + +```rust +pub use data_server::meta::InstrumentGeometry; + +/// Recorded per-symbol geometry at the ingestion edge: the provider's structural +/// truth (digits, pip/tick size, lot size, quote currency), read from the +/// dataset sidecar. Beside the hot path (C7/C15); never enters the engine seam. +pub fn instrument_geometry(server: &Arc, symbol: &str) -> Option { + server.symbol_meta(symbol) +} +``` + +and its unit test `instrument_geometry_none_for_unknown_symbol`. + +### Delivered — example fixtures + +`crates/aura-ingest/examples/shared/breakout_real.rs`. `pip_size_of` resolves +from the sidecar (real-data examples require the archive anyway): + +```rust +/// The example's pip divisor, sourced from the recorded geometry sidecar — the +/// same provider truth the CLI `--real` path uses (#98), now provider-sourced. +/// Panics on a symbol with no sidecar (examples only run known instruments with +/// local data). +pub fn pip_size_of(symbol: &str) -> f64 { + aura_ingest::instrument_geometry(&aura_ingest::default_data_server(), symbol) + .unwrap_or_else(|| panic!("no recorded geometry sidecar for {symbol}")) + .pip_size +} +``` + +The **pure-structural** blueprint tests assert on `param_space()` topology, not on +the pip, and are contracted to "run everywhere" (no archive). They pass a literal +in place of the sidecar read so they stay hermetic — +`crates/aura-ingest/tests/ger40_breakout_blueprint.rs`: + +```rust +// param_space_is_exactly_entry_and_exit_bar_targets (#37: "No data needed") +// and blueprint_param_space_is_construction_deterministic: the pip is a throwaway +// sizing arg here, never asserted, so a literal keeps these archive-independent. +let (bp, _taps) = ger40_breakout_blueprint(1.0 /* GER40 pip: index point */, 15, 9, 0, Berlin); +``` + +Real-data / archive-gated consumers (`ger40_breakout_world.rs`, the example +binaries, the gated blueprint fidelity test) keep calling `pip_size_of` — they +skip cleanly when the archive is absent and run with it present, as today. The +cross-instrument compare example (`ger40_breakout_compare.rs`, variable `symbol`) +keeps `pip_size_of(symbol)` and is now per-symbol sidecar-sourced. + +### Removed — table-only test files + +`crates/aura-ingest/tests/instrument_geometry_crosscheck.rs` (cross-checks the +removed floor against the provider — moot once the floor is gone) and +`crates/aura-ingest/tests/instrument_spec_deploy_metadata.rs` (asserts on the +removed fields). + +## Components + +- **aura-cli** — `pip_or_refuse` replaces `instrument_spec_or_refuse`; two call + sites reordered (server before pip). +- **aura-ingest (lib)** — public surface shrinks to `instrument_geometry` + + `InstrumentGeometry`; the authored floor and its tests are removed. +- **aura-ingest (examples/tests)** — `pip_size_of` repointed to the sidecar; + pure-structural blueprint tests use a literal pip; two table-only test files + deleted. +- **Design ledger** (`docs/design/INDEX.md`) — the C15 realization note added in + 0073 (authored floor + cross-check + tick_size/digits) is rewritten to record + the floor's removal and provider-sourced pip; the #124 hierarchy note records + the collapse to "sidecar → refuse" with the override/floor tiers deferred. + +## Data flow + +`aura --real ` → build `DataServer` → `pip_or_refuse(server, symbol)` +reads `.meta.json` via `symbol_meta` → `pip_size` (or exit 2) → +threaded into the real `Source` / SimBroker exactly as before. The +position-event table's `instrument_id` continues to arrive as a caller parameter, +untouched. + +## Error handling + +A symbol with no readable/parseable sidecar → `instrument_geometry` returns +`None` → `pip_or_refuse` prints the refusal and `exit(2)`, before any bar-data +access. Same exit code and pre-data ordering as the removed helper; only the +message and the source change. Any test pinning the old refusal substring is +updated to the new message. + +## Testing strategy + +- **RED → GREEN (behaviour):** a test that `pip_or_refuse` (or the real path) for + a symbol with a sidecar yields the sidecar's `pip_size`, and that an unknown + symbol refuses (exit 2). The existing real-path tests already ratify "pip is + threaded into the real source"; this cycle re-sources it without changing that + observable. +- **Deletions:** removing `InstrumentSpec`/`instrument_spec`/`VETTED_SYMBOLS` and + their tests is safe by the consumption grep (no production reader). The compile + itself enumerates any missed consumer. +- **Hermetic structural tests stay hermetic:** the two pure blueprint tests must + still pass with the archive absent (literal pip, no sidecar read). +- **Gates:** `cargo test --workspace` green; `cargo clippy --workspace + --all-targets -- -D warnings` clean. On this host the archive is present, so the + sidecar-sourced paths actually execute. + +## Acceptance criteria + +1. **Removes redundancy / improves correctness (the project criterion):** the + authored floor duplicated the provider geometry byte-for-byte (proven by the + #143 cross-check); removing it deletes a drift surface and sources the pip from + the broker's recorded truth. +2. **Reintroduces no failure class the core constraints forbid:** refuse-don't- + guess preserved (exit 2 on absent geometry); determinism preserved (sidecar is + recorded data read at bootstrap, C6); the engine `(Timestamp, Scalar)` seam + untouched (C4/C7); no look-ahead introduced (static metadata). +3. **Defines only what is needed:** the only instrument datum any current path + consumes — `pip_size` — is the only one defined/sourced; the rest is deferred + to the Stage-2 broker that will consume it, read from the sidecar then. +4. `cargo test --workspace` and the clippy gate pass; the pure-structural tests + pass with and without the local archive.