Aura/docs/specs/0074-pip-from-sidecar.md

Provider-sourced pip; drop the authored instrument floor — Design Spec

Date: 2026-06-25 Status: Draft — awaiting user spec review Authors: orchestrator + Claude

Goal

aura must define and take over only the instrument reference data it currently consumes, and source it from the data-server geometry sidecars rather than a hand-maintained table. A grep of every InstrumentSpec field read across crates/ shows production code consumes exactly one datum — pip_size — at the real-data path. Every other field (contract_size, pip_value_per_lot, min_lot, lot_step, quote_currency, tick_size, digits, and instrument_id) is constructed but never read outside the table's own tests.

This cycle resolves the pip from the per-symbol sidecar (instrument_geometry(server, symbol).pip_size) and removes the now-redundant authored floor (InstrumentSpec, instrument_spec(), VETTED_SYMBOLS) together with the tests that exist only to validate it. The decision and its grounding are recorded on the reference issue (#124, reconciliation comment).

Deferred, not deleted (per the user directive "build it back in when the broker needs it"): the speculative fields are revived from InstrumentGeometry — which already carries digits, lotSize, tickSize, quoteAsset — when the Stage-2 realistic-broker milestone actually consumes them. No knowledge is lost; the sidecar remains the source.

Architecture

The instrument-reference resolution hierarchy of #124 (authored override > recorded metadata > authored floor > refuse) collapses, for what is built now, to its two ends: recorded sidecar geometry → refuse-don't-guess. The middle authored floor (tier 3) and the authored override (tier 1) are future work, built when a consumer needs them. The instrument_geometry accessor and the InstrumentGeometry re-export added in #143 are the surviving seam and become the sole pip source.

The instrument_id question (the directive's "find out"): the position-event table needs an instrument_id, but it already takes one as a caller-supplied i64 parameter — derive_position_events(record, instrument_id) (report.rs:231) — and the engine deliberately never imports InstrumentSpec (report.rs:226). InstrumentSpec.instrument_id has no production reader; it is dropped with the struct. No symbol→id registry is consumed anywhere today, so none is built; when one is needed it is added then.

Invariants preserved: refuse-don't-guess (a symbol with no sidecar → exit 2, unchanged); honest-pip-by-construction (now the provider's recorded value instead of a hand-authored one — strictly more honest); determinism / C6 (the sidecar is recorded data read once at bootstrap, never a live mid-replay call); the engine's (Timestamp, Scalar) seam is untouched (geometry stays beside the hot path, C7/C15).

Concrete code shapes

Delivered — CLI pip resolution (the production path)

crates/aura-cli/src/main.rs. Before — table lookup, server-free, refuses on an un-vetted symbol:

// ~line 730
fn instrument_spec_or_refuse(symbol: &str) -> aura_ingest::InstrumentSpec {
    match aura_ingest::instrument_spec(symbol) {
        Some(s) => s,
        None => {
            eprintln!("aura: no vetted pip/instrument spec for symbol '{symbol}' — \
                       refusing to run a real instrument with a guessed pip \
                       (add it to the instrument table)");
            std::process::exit(2);
        }
    }
}

After — resolve the pip from the recorded sidecar; refuse when absent. The pip is the only datum any caller used, so the helper returns f64:

// the single home of the guessed-pip refusal, now sidecar-sourced. Reads the
// per-symbol geometry metadata (not bar data) BEFORE the run source opens, so an
// instrument with no recorded geometry refuses without touching the archive —
// the pip stays honest by construction (the provider's value, never guessed).
fn pip_or_refuse(server: &std::sync::Arc<data_server::DataServer>, symbol: &str) -> f64 {
    match aura_ingest::instrument_geometry(server, symbol) {
        Some(geo) => geo.pip_size,
        None => {
            eprintln!("aura: no recorded geometry for symbol '{symbol}' at {} — \
                       refusing to run a real instrument with a guessed pip",
                      data_server::DEFAULT_DATA_PATH);
            std::process::exit(2);
        }
    }
}

Both call sites build the server first, then resolve the pip (the refusal stays ahead of bar-data access). open_real_source (~line 767):

// before: spec lookup, THEN server
//   let spec = instrument_spec_or_refuse(symbol);
//   let server = Arc::new(DataServer::new(DEFAULT_DATA_PATH));
// after: server first, then pip from its sidecar
let server = std::sync::Arc::new(data_server::DataServer::new(data_server::DEFAULT_DATA_PATH));
let pip = pip_or_refuse(&server, symbol);
if !server.has_symbol(symbol) { no_real_data(symbol); }
// ... probe_window + open as today; return (source, window, pip)

DataSource::from_choice (~line 794) takes the same reorder: build server, let pip = pip_or_refuse(&server, &symbol);, then has_symbol, then DataSource::Real { server, symbol, from_ms, to_ms, pip }.

Delivered — aura-ingest library surface

crates/aura-ingest/src/lib.rs. Removed: pub struct InstrumentSpec (all 9 fields), pub fn instrument_spec, pub const VETTED_SYMBOLS, and the three unit tests that only exercise them (instrument_spec_lookup_by_symbol, instrument_spec_pip_value_matches_contract_times_pip, instrument_spec_ids_are_distinct). Kept unchanged: the pip source —

pub use data_server::meta::InstrumentGeometry;

/// Recorded per-symbol geometry at the ingestion edge: the provider's structural
/// truth (digits, pip/tick size, lot size, quote currency), read from the
/// dataset sidecar. Beside the hot path (C7/C15); never enters the engine seam.
pub fn instrument_geometry(server: &Arc<DataServer>, symbol: &str) -> Option<InstrumentGeometry> {
    server.symbol_meta(symbol)
}

and its unit test instrument_geometry_none_for_unknown_symbol.

Delivered — example fixtures

crates/aura-ingest/examples/shared/breakout_real.rs. pip_size_of resolves from the sidecar (real-data examples require the archive anyway):

/// The example's pip divisor, sourced from the recorded geometry sidecar — the
/// same provider truth the CLI `--real` path uses (#98), now provider-sourced.
/// Panics on a symbol with no sidecar (examples only run known instruments with
/// local data).
pub fn pip_size_of(symbol: &str) -> f64 {
    aura_ingest::instrument_geometry(&aura_ingest::default_data_server(), symbol)
        .unwrap_or_else(|| panic!("no recorded geometry sidecar for {symbol}"))
        .pip_size
}

The pure-structural blueprint tests assert on param_space() topology, not on the pip, and are contracted to "run everywhere" (no archive). They pass a literal in place of the sidecar read so they stay hermetic — crates/aura-ingest/tests/ger40_breakout_blueprint.rs:

// param_space_is_exactly_entry_and_exit_bar_targets (#37: "No data needed")
// and blueprint_param_space_is_construction_deterministic: the pip is a throwaway
// sizing arg here, never asserted, so a literal keeps these archive-independent.
let (bp, _taps) = ger40_breakout_blueprint(1.0 /* GER40 pip: index point */, 15, 9, 0, Berlin);

Real-data / archive-gated consumers (ger40_breakout_world.rs, the example binaries, the gated blueprint fidelity test) keep calling pip_size_of — they skip cleanly when the archive is absent and run with it present, as today. The cross-instrument compare example (ger40_breakout_compare.rs, variable symbol) keeps pip_size_of(symbol) and is now per-symbol sidecar-sourced.

Removed — table-only test files

crates/aura-ingest/tests/instrument_geometry_crosscheck.rs (cross-checks the removed floor against the provider — moot once the floor is gone) and crates/aura-ingest/tests/instrument_spec_deploy_metadata.rs (asserts on the removed fields).

Components

• aura-cli — pip_or_refuse replaces instrument_spec_or_refuse; two call sites reordered (server before pip).
• aura-ingest (lib) — public surface shrinks to instrument_geometry + InstrumentGeometry; the authored floor and its tests are removed.
• aura-ingest (examples/tests) — pip_size_of repointed to the sidecar; pure-structural blueprint tests use a literal pip; two table-only test files deleted.
• Design ledger (docs/design/INDEX.md) — the C15 realization note added in 0073 (authored floor + cross-check + tick_size/digits) is rewritten to record the floor's removal and provider-sourced pip; the #124 hierarchy note records the collapse to "sidecar → refuse" with the override/floor tiers deferred.

Data flow

aura --real <SYMBOL> → build DataServer → pip_or_refuse(server, symbol) reads <SYMBOL>.meta.json via symbol_meta → pip_size (or exit 2) → threaded into the real Source / SimBroker exactly as before. The position-event table's instrument_id continues to arrive as a caller parameter, untouched.

Error handling

A symbol with no readable/parseable sidecar → instrument_geometry returns None → pip_or_refuse prints the refusal and exit(2), before any bar-data access. Same exit code and pre-data ordering as the removed helper; only the message and the source change. Any test pinning the old refusal substring is updated to the new message.

Testing strategy

• RED → GREEN (behaviour): a test that pip_or_refuse (or the real path) for a symbol with a sidecar yields the sidecar's pip_size, and that an unknown symbol refuses (exit 2). The existing real-path tests already ratify "pip is threaded into the real source"; this cycle re-sources it without changing that observable.
• Deletions: removing InstrumentSpec/instrument_spec/VETTED_SYMBOLS and their tests is safe by the consumption grep (no production reader). The compile itself enumerates any missed consumer.
• Hermetic structural tests stay hermetic: the two pure blueprint tests must still pass with the archive absent (literal pip, no sidecar read).
• Gates: cargo test --workspace green; cargo clippy --workspace --all-targets -- -D warnings clean. On this host the archive is present, so the sidecar-sourced paths actually execute.

Acceptance criteria

1. Removes redundancy / improves correctness (the project criterion): the authored floor duplicated the provider geometry byte-for-byte (proven by the #143 cross-check); removing it deletes a drift surface and sources the pip from the broker's recorded truth.
2. Reintroduces no failure class the core constraints forbid: refuse-don't- guess preserved (exit 2 on absent geometry); determinism preserved (sidecar is recorded data read at bootstrap, C6); the engine (Timestamp, Scalar) seam untouched (C4/C7); no look-ahead introduced (static metadata).
3. Defines only what is needed: the only instrument datum any current path consumes — pip_size — is the only one defined/sourced; the rest is deferred to the Stage-2 broker that will consume it, read from the sidecar then.
4. cargo test --workspace and the clippy gate pass; the pure-structural tests pass with and without the local archive.