Brummel/Aura
1
0
Fork 0
Code Issues 31 Pull Requests Actions Packages Projects Releases Wiki Activity

Instrument-reference & cost resolution hierarchy: recorded dataset-metadata tier (DataServer seam); authored override always wins #124

New Issue
Closed
opened 2026-06-22 18:45:13 +02:00 by Brummel · 6 comments
Brummel commented 2026-06-22 18:45:13 +02:00
Owner
Copy Link
Copy Source

The authored layer for instrument reference data and cost models (instrument_spec() + the realistic broker's spread/commission models) is permanent and authoritative-by-choice, not a provider stand-in. This issue adds the recorded dataset-metadata tier as the MIDDLE tier of the resolution hierarchy — never above the explicit authored override.

Resolution hierarchy (precedence high → low)

1. Explicit authored override  (project/run; Rust, later Aura.toml)         — always wins
2. Recorded dataset metadata   (provider-sourced at curate time,            — THIS issue
                                 surfaced by the DataServer/Source seam, C6/C3/C7)
3. Vetted authored fallback    (instrument_spec() table, #113)              — permanent floor
4. Refuse-don't-guess          (exit 2)                                     — absolute floor

Why a tier, not a replacement

  • A data source can never be guaranteed to ship metadata — tier 3/4 must always exist.
  • When it does ship metadata, it may be wrong/unrealistic — the canonical case is the spread: a feed (e.g. Pepperstone) often records a constant ~2-pip spread decoupled from the real, higher, volatile historical spread. The recorded value is an input the authored override supersedes; trusting it blindly biases a realistic broker optimistic.
  • Therefore recorded values are opt-in per dataset/field, and the explicit authored override always wins.

Mechanics

  • The Pepperstone NFS files today are bars-only (M1Columns: OHLC + per-bar spread + volume); no instrument-structural metadata is carried. "Provider-sourced" means: at dataset curate time, record the symbol's structural spec (from the provider's symbol info) into the dataset metadata, which the DataServer then surfaces at the ingestion edge.
  • A resolver at the ingestion edge composes the four tiers and hands the resolved InstrumentSpec / cost model to the broker + derivation (which already consume an injected spec per #113/#116).

depends on: #71 (DataServer/Source seam — its home), #113 (authored layer), #120 (cost models)
context: surfaced refining the realistic-broker milestone; the recorded-metadata channel on the source seam is World-II work, separate from the broker milestone.

The authored layer for instrument reference data and cost models (`instrument_spec()` + the realistic broker's spread/commission models) is **permanent and authoritative-by-choice**, not a provider stand-in. This issue adds the **recorded dataset-metadata tier** as the MIDDLE tier of the resolution hierarchy — never above the explicit authored override. ## Resolution hierarchy (precedence high → low) ``` 1. Explicit authored override (project/run; Rust, later Aura.toml) — always wins 2. Recorded dataset metadata (provider-sourced at curate time, — THIS issue surfaced by the DataServer/Source seam, C6/C3/C7) 3. Vetted authored fallback (instrument_spec() table, #113) — permanent floor 4. Refuse-don't-guess (exit 2) — absolute floor ``` ## Why a tier, not a replacement - A data source can **never be guaranteed** to ship metadata — tier 3/4 must always exist. - When it does ship metadata, it may be wrong/unrealistic — the **canonical case is the spread**: a feed (e.g. Pepperstone) often records a constant ~2-pip spread decoupled from the real, higher, volatile historical spread. The recorded value is an **input** the authored override supersedes; trusting it blindly biases a realistic broker optimistic. - Therefore recorded values are **opt-in per dataset/field**, and the explicit authored override always wins. ## Mechanics - The Pepperstone NFS files today are **bars-only** (`M1Columns`: OHLC + per-bar spread + volume); no instrument-structural metadata is carried. "Provider-sourced" means: at dataset **curate** time, record the symbol's structural spec (from the provider's symbol info) into the dataset metadata, which the `DataServer` then surfaces at the ingestion edge. - A **resolver at the ingestion edge** composes the four tiers and hands the resolved `InstrumentSpec` / cost model to the broker + derivation (which already consume an injected spec per #113/#116). depends on: #71 (DataServer/Source seam — its home), #113 (authored layer), #120 (cost models) context: surfaced refining the realistic-broker milestone; the recorded-metadata channel on the source seam is World-II work, separate from the broker milestone.
Brummel added the feature label 2026-06-22 18:45:13 +02:00
Brummel commented 2026-06-22 19:50:56 +02:00
Author
Owner
Copy Link
Copy Source

Design input from cycle 0063 (broker-foundation) — quote_currency representation

The authored vetted floor landed in 0063 (#113) types quote_currency as
&'static str (crates/aura-ingest/src/lib.rs), keeping InstrumentSpec: Copy
and fitting the static Rust-authored table.

An adversarial review of 0063 flagged that this field cannot hold a
runtime-sourced currency (a recorded dataset-metadata value or an Aura.toml
override — exactly this issue's tier). When #124 lands the resolver, it must pick
one of:

  1. Distinct resolved type — keep InstrumentSpec as the &'static str
    authored floor and have the resolver produce a separate
    ResolvedInstrumentSpec (owned String/Cow currency). No break to the
    authored table; the broker/derivation consume the resolved type.
  2. Widen the field — change InstrumentSpec.quote_currency to
    Cow<'static, str> (or String), dropping Copy. One type throughout; the
    authored literals become Cow::Borrowed.

This is a #124 design decision, deliberately not pre-empted in 0063. Recording it
here so the resolver design weighs it rather than discovering the &'static str
floor by surprise. The C15 "extensible without a signature break" promise is about
additive fields (honored in 0063); the currency type transition is this tier's
concern.

## Design input from cycle 0063 (broker-foundation) — quote_currency representation The authored vetted floor landed in 0063 (#113) types `quote_currency` as `&'static str` (`crates/aura-ingest/src/lib.rs`), keeping `InstrumentSpec: Copy` and fitting the static Rust-authored table. An adversarial review of 0063 flagged that this field cannot hold a **runtime-sourced** currency (a recorded dataset-metadata value or an Aura.toml override — exactly this issue's tier). When #124 lands the resolver, it must pick one of: 1. **Distinct resolved type** — keep `InstrumentSpec` as the `&'static str` authored floor and have the resolver produce a separate `ResolvedInstrumentSpec` (owned `String`/`Cow` currency). No break to the authored table; the broker/derivation consume the resolved type. 2. **Widen the field** — change `InstrumentSpec.quote_currency` to `Cow<'static, str>` (or `String`), dropping `Copy`. One type throughout; the authored literals become `Cow::Borrowed`. This is a #124 design decision, deliberately not pre-empted in 0063. Recording it here so the resolver design weighs it rather than discovering the `&'static str` floor by surprise. The C15 "extensible without a signature break" promise is about *additive fields* (honored in 0063); the currency *type* transition is this tier's concern.
Brummel commented 2026-06-25 13:59:53 +02:00
Author
Owner
Copy Link
Copy Source

Provider metadata now physically exists — supersedes the bars-only premise

The Mechanics section above states "the Pepperstone NFS files today are bars-only
(M1Columns: OHLC + per-bar spread + volume); no instrument-structural metadata is
carried". That is now superseded: the export pipeline writes one per-symbol
geometry sidecar <SYMBOL>.meta.json beside the bars. Verified: 30 files in
/mnt/tickdata/Pepperstone, all schemaVersion 2, carrying digits, pipSize,
tickSize, lotSize, baseAsset, quoteAsset — exactly the "symbol's
structural spec recorded at curate time" this issue anticipated for tier 2.

The concrete realization of that tier is now filed as two cross-referenced issues,
not folded into this one (which stays the hierarchy/precedence design):

  • http://192.168.178.103:3000/Brummel/data-server/issues/3 — the upstream
    reader: parse the sidecars into neutral InstrumentGeometry so the provider's
    filename/schema/quirks (DE40/USTEC naming, 1E-05, null-for-NaN) stay
    inside the broker-facing library.
  • #143 — the aura-ingest consumer: slot the neutral geometry as tier 2,
    cross-check it against the vetted floor (#113), add tick_size/digits, and
    extend quote_currency past FX. It defers to this issue on precedence and on
    the quote_currency type fork raised in the comment below.

Scope note: the sidecar is geometry only. It deliberately excludes the
time-varying / account-dependent fields — including the spread, which is this
issue's canonical "recorded value the authored override supersedes" case. So the
sidecars satisfy tier 2 for structural geometry (pip/lot/tick/digits/currency),
not for cost models; the spread/commission tier-2 channel remains open here.

## Provider metadata now physically exists — supersedes the bars-only premise The Mechanics section above states "the Pepperstone NFS files today are bars-only (M1Columns: OHLC + per-bar spread + volume); no instrument-structural metadata is carried". That is now superseded: the export pipeline writes one per-symbol geometry sidecar `<SYMBOL>.meta.json` beside the bars. Verified: 30 files in `/mnt/tickdata/Pepperstone`, all `schemaVersion` 2, carrying `digits`, `pipSize`, `tickSize`, `lotSize`, `baseAsset`, `quoteAsset` — exactly the "symbol's structural spec recorded at curate time" this issue anticipated for tier 2. The concrete realization of that tier is now filed as two cross-referenced issues, not folded into this one (which stays the hierarchy/precedence design): - `http://192.168.178.103:3000/Brummel/data-server/issues/3` — the upstream reader: parse the sidecars into neutral `InstrumentGeometry` so the provider's filename/schema/quirks (`DE40`/`USTEC` naming, `1E-05`, `null`-for-NaN) stay inside the broker-facing library. - `#143` — the `aura-ingest` consumer: slot the neutral geometry as tier 2, cross-check it against the vetted floor (`#113`), add `tick_size`/`digits`, and extend `quote_currency` past FX. It defers to this issue on precedence and on the `quote_currency` type fork raised in the comment below. Scope note: the sidecar is **geometry only**. It deliberately excludes the time-varying / account-dependent fields — including the **spread**, which is this issue's canonical "recorded value the authored override supersedes" case. So the sidecars satisfy tier 2 for *structural geometry* (pip/lot/tick/digits/currency), not for cost models; the spread/commission tier-2 channel remains open here.
Brummel commented 2026-06-25 16:32:04 +02:00
Author
Owner
Copy Link
Copy Source

Tier-2 source now reachable at the ingestion edge (via #143)

#143 (closed, cycle 0073) landed the recorded dataset-metadata tier-2 source:
aura_ingest::instrument_geometry(server, symbol) -> Option<InstrumentGeometry> over
data-server's symbol_meta, returning neutral broker-agnostic geometry
(digits, pip_size, tick_size, lot_size, base, quote). The raw provider JSON stays
inside the broker-facing library; aura consumes only the neutral type.

What it deliberately left to this issue:

  • the tier-composing resolver (authored override > recorded metadata > vetted
    floor > refuse) — #143 surfaced the tier-2 data, not the precedence engine;
  • the quote_currency type-fork (distinct ResolvedInstrumentSpec vs. widening
    to Cow) — the floor's quote_currency is still &'static str; the runtime-sourced
    String currency lives only in InstrumentGeometry for now;
  • the spread/commission tier-2 channel — the sidecar is geometry-only (it excludes
    the time-varying/account-dependent fields, the spread among them), so cost-model
    tier-2 remains open here.
## Tier-2 *source* now reachable at the ingestion edge (via #143) #143 (closed, cycle 0073) landed the recorded dataset-metadata **tier-2 source**: `aura_ingest::instrument_geometry(server, symbol) -> Option<InstrumentGeometry>` over data-server's `symbol_meta`, returning neutral broker-agnostic geometry (`digits, pip_size, tick_size, lot_size, base, quote`). The raw provider JSON stays inside the broker-facing library; aura consumes only the neutral type. What it deliberately left to **this** issue: - the **tier-composing resolver** (authored override > recorded metadata > vetted floor > refuse) — #143 surfaced the tier-2 *data*, not the precedence engine; - the **`quote_currency` type-fork** (distinct `ResolvedInstrumentSpec` vs. widening to `Cow`) — the floor's `quote_currency` is still `&'static str`; the runtime-sourced String currency lives only in `InstrumentGeometry` for now; - the **spread/commission tier-2 channel** — the sidecar is geometry-only (it excludes the time-varying/account-dependent fields, the spread among them), so cost-model tier-2 remains open here.
Brummel commented 2026-06-25 17:48:51 +02:00
Author
Owner
Copy Link
Copy Source

Design reconciliation (specify) — minimal, provider-sourced reframe

User directive (this run): aura takes over and defines only what is
currently needed; the data-server now supplies symbol geometry, so geometry
must no longer be hard-coded in aura. Find out whether an instrument_id is
needed. This resolves the shape of this issue's resolver. The forks below are
listed open here; this records their resolution before the spec is written.

Consumption investigation (the empirical basis)

Grep of every InstrumentSpec field read across crates/ (reads, not
construction):

  • pip_size — the only field read in production: aura-cli
    (open_real_source / DataSource::from_choice thread spec.pip_size into the
    real source / SimBroker) and the aura-ingest example pip_size_of.
  • instrument_id — no production reader. The position-event table carries an
    instrument_id, but as a caller-supplied parameter
    derive_position_events(record, instrument_id) (report.rs:231); the engine
    never imports InstrumentSpec (report.rs:226), and stage1_r_e2e.rs asserts
    id == 7, an id the floor never registers (it only has 1–5). The field is
    decoupled and unused.
  • contract_size, pip_value_per_lot, min_lot, lot_step,
    quote_currency, tick_size, digits     — constructed only; read only in the
    table's own unit tests. Speculative, anticipating the unbuilt Stage-2 broker.

Forks resolved

  • Fork: resolution shape → collapse, for what is built now, to
    sidecar geometry → refuse. The four-tier override > metadata > floor > refuse hierarchy is deferred, not deleted: tier 1 (authored override) and
    tier 3 (authored floor) are built when a consumer needs them (the Stage-2
    broker overriding the spread). Tier 2 (provider-sourced) becomes the primary
    and only currently-built source; tier 4 (refuse-don't-guess) survives.
    Basis: derived — user directive (YAGNI) + the floor's values were
    byte-identical duplication of the provider (the #143 cross-check proved it).
  • Fork: InstrumentSpec struct → drop the struct, instrument_spec(),
    and VETTED_SYMBOLS. Pip is sourced from InstrumentGeometry.pip_size (the
    #143 instrument_geometry accessor, which survives). Basis: derived — only
    pip_size is read in production.
  • Fork: instrument_id (the directive's "find out") → not needed; dropped
    with the struct.     The position-event table keeps its caller-supplied
    instrument_id (a separate, real, engine-side concept). No production code maps
    symbol → id today; when such a registry is needed it is built then. Basis:
    derived — grep shows no production reader.
  • Fork: quote_currency type (the 0063 &'static str vs owned fork raised
    above) → moot now (field dropped). Revived from InstrumentGeometry.quote
    (owned String) when Stage 2 needs currency — no Copy concern, since it is
    read off the sidecar, never a static table. Basis: derived — the fork only
    existed because of the authored floor that is now gone.

Scope of this run vs. what stays open here

This run implements the currently-needed slice: pip resolved from the
provider sidecar at the aura-ingest/CLI edge, and the dead authored floor removed.
#124 stays open for the deferred tiers — the authored override and the
cost-model / spread tier-2 channel — which the Stage-2 realistic-broker milestone
will need. #140 (hand-register four more symbols into the floor) is obsolete
and closed: pip now comes from whichever symbols ship a sidecar (30 present).
#143's tick_size/digits fields, added last cycle and never consumed, are
removed forward as part of this rework (no history rewrite).

## Design reconciliation (specify) — minimal, provider-sourced reframe User directive (this run): *aura takes over and defines only what is **currently** needed; the data-server now supplies symbol geometry, so geometry must no longer be hard-coded in aura. Find out whether an `instrument_id` is needed.* This resolves the shape of this issue's resolver. The forks below are listed open here; this records their resolution before the spec is written. ### Consumption investigation (the empirical basis) Grep of every `InstrumentSpec` field read across `crates/` (reads, not construction): - **`pip_size`** — the *only* field read in production: `aura-cli` (`open_real_source` / `DataSource::from_choice` thread `spec.pip_size` into the real source / SimBroker) and the `aura-ingest` example `pip_size_of`. - **`instrument_id`** — no production reader. The position-event table carries an `instrument_id`, but as a **caller-supplied parameter** `derive_position_events(record, instrument_id)` (report.rs:231); the engine *never* imports `InstrumentSpec` (report.rs:226), and `stage1_r_e2e.rs` asserts `id == 7`, an id the floor never registers (it only has 1–5). The field is decoupled and unused. - **`contract_size`, `pip_value_per_lot`, `min_lot`, `lot_step`, `quote_currency`, `tick_size`, `digits`** — constructed only; read only in the table's own unit tests. Speculative, anticipating the unbuilt Stage-2 broker. ### Forks resolved - **Fork: resolution shape** → collapse, for what is built now, to **sidecar geometry → refuse**. The four-tier `override > metadata > floor > refuse` hierarchy is *deferred*, not deleted: tier 1 (authored override) and tier 3 (authored floor) are built when a consumer needs them (the Stage-2 broker overriding the spread). Tier 2 (provider-sourced) becomes the primary and only currently-built source; tier 4 (refuse-don't-guess) survives. Basis: **derived** — user directive (YAGNI) + the floor's values were byte-identical duplication of the provider (the #143 cross-check proved it). - **Fork: `InstrumentSpec` struct** → **drop** the struct, `instrument_spec()`, and `VETTED_SYMBOLS`. Pip is sourced from `InstrumentGeometry.pip_size` (the #143 `instrument_geometry` accessor, which survives). Basis: **derived** — only `pip_size` is read in production. - **Fork: `instrument_id` (the directive's "find out")** → **not needed; dropped with the struct.** The position-event table keeps its caller-supplied `instrument_id` (a separate, real, engine-side concept). No production code maps symbol → id today; when such a registry is needed it is built then. Basis: **derived** — grep shows no production reader. - **Fork: `quote_currency` type** (the 0063 `&'static str` vs owned fork raised above) → **moot now** (field dropped). Revived from `InstrumentGeometry.quote` (owned `String`) when Stage 2 needs currency — no `Copy` concern, since it is read off the sidecar, never a static table. Basis: **derived** — the fork only existed because of the authored floor that is now gone. ### Scope of this run vs. what stays open here This run implements the **currently-needed slice**: pip resolved from the provider sidecar at the aura-ingest/CLI edge, and the dead authored floor removed. **#124 stays open** for the deferred tiers — the authored override and the cost-model / spread tier-2 channel — which the Stage-2 realistic-broker milestone will need. **#140** (hand-register four more symbols into the floor) is obsolete and closed: pip now comes from whichever symbols ship a sidecar (30 present). **#143**'s `tick_size`/`digits` fields, added last cycle and never consumed, are removed forward as part of this rework (no history rewrite).
Brummel commented 2026-06-28 11:59:36 +02:00
Author
Owner
Copy Link
Copy Source

Note: the 2026-06-28 C10 rework (29cdc8c) retires the realistic broker, so this hierarchy's eventual consumer is now the cost-model graph's currency→R normalization at ingestion (#148) and the live/deploy edge (#149), not a Stage-2 realistic broker. The authored-override-wins resolution itself stands. Keeping open.

Note: the 2026-06-28 C10 rework (`29cdc8c`) retires the realistic broker, so this hierarchy's eventual consumer is now the cost-model graph's currency→R normalization at ingestion (#148) and the live/deploy edge (#149), not a Stage-2 realistic broker. The authored-override-wins resolution itself stands. Keeping open.
claude commented 2026-07-21 10:36:44 +02:00
Collaborator
Copy Link
Copy Source

This issue's resolution hierarchy was scoped around two consumers, both retired or removed since filing.

  1. instrument_spec() / InstrumentSpec — the authored tier-1/tier-3 floor this issue names — was deleted in cycle 0074 (commit 7e4b91a, "source real-path pip from the geometry sidecar; drop the authored floor"): a consumption grep showed production code read only the pip_size field, so the six-field deploy-grade struct, instrument_spec(), and VETTED_SYMBOLS were removed as dead code. Only the tier-2 source survives: instrument_geometry() (defined crates/aura-ingest/src/lib.rs:411, re-exported as InstrumentGeometry), which returns Option<InstrumentGeometry> and refuses (exit 2) on None — no override tier above it, no authored floor below it. After the #295 shell-boundary extraction, the caller lives in crates/aura-runner/src/member.rs (moved out of aura-cli, logic unchanged).

  2. "The realistic broker's spread/commission models" — the issue's other named consumer — no longer exist. Design commit 29cdc8c ("design(C10): retire the realistic broker…", 2026-06-28) reworked C10 in docs/design/INDEX.md: an authored-friction historical broker is now explicitly forbidden ("modelling an authored-friction 'realistic broker' over historical data" — current Forbids clause). No RealisticBroker type exists anywhere under crates/. Cost is now a composable C9 graph of cost nodes in R (crates/aura-strategy/src/constant_cost.rs, carry_cost.rs, vol_slippage_cost.rs, cost_sum.rs), none of which currently reads instrument geometry.

The ledger already records this collapse (docs/design/INDEX.md:1275-1293, "Realization (instrument specs, 2026-06, #22 -> #124)"): the override/floor tiers of the #124 hierarchy "are deferred until a consumer … needs them", and "#124 collapses to recorded sidecar geometry -> refuse in the meantime" — i.e. the design this issue asked for has no live consumer today, and the consumer it was written against is gone, not merely pending.

Closing as superseded. If a future cost-model node needs currency/pip normalization from instrument reference data, that resolution design belongs to the C9 cost-graph consumer that actually needs it (e.g. a currency-cost node under crates/aura-strategy/), scoped against the cost model's real requirements rather than a revival of this issue's authored-override/broker-cost framing.

This issue's resolution hierarchy was scoped around two consumers, both retired or removed since filing. 1. `instrument_spec()` / `InstrumentSpec` — the authored tier-1/tier-3 floor this issue names — was deleted in cycle 0074 (commit `7e4b91a`, "source real-path pip from the geometry sidecar; drop the authored floor"): a consumption grep showed production code read only the `pip_size` field, so the six-field deploy-grade struct, `instrument_spec()`, and `VETTED_SYMBOLS` were removed as dead code. Only the tier-2 source survives: `instrument_geometry()` (defined `crates/aura-ingest/src/lib.rs:411`, re-exported as `InstrumentGeometry`), which returns `Option<InstrumentGeometry>` and refuses (exit 2) on `None` — no override tier above it, no authored floor below it. After the #295 shell-boundary extraction, the caller lives in `crates/aura-runner/src/member.rs` (moved out of `aura-cli`, logic unchanged). 2. "The realistic broker's spread/commission models" — the issue's other named consumer — no longer exist. Design commit `29cdc8c` ("design(C10): retire the realistic broker…", 2026-06-28) reworked C10 in `docs/design/INDEX.md`: an authored-friction historical broker is now explicitly forbidden ("modelling an authored-friction 'realistic broker' over historical data" — current Forbids clause). No `RealisticBroker` type exists anywhere under `crates/`. Cost is now a composable C9 graph of cost nodes in R (`crates/aura-strategy/src/constant_cost.rs`, `carry_cost.rs`, `vol_slippage_cost.rs`, `cost_sum.rs`), none of which currently reads instrument geometry. The ledger already records this collapse (`docs/design/INDEX.md:1275-1293`, "Realization (instrument specs, 2026-06, #22 -> #124)"): the override/floor tiers of the #124 hierarchy "are deferred until a consumer … needs them", and "#124 collapses to recorded sidecar geometry -> refuse in the meantime" — i.e. the design this issue asked for has no live consumer today, and the consumer it was written against is gone, not merely pending. Closing as superseded. If a future cost-model node needs currency/pip normalization from instrument reference data, that resolution design belongs to the C9 cost-graph consumer that actually needs it (e.g. a currency-cost node under `crates/aura-strategy/`), scoped against the cost model's real requirements rather than a revival of this issue's authored-override/broker-cost framing.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
BLOCKER
Core functionality blocked; preempts other selection
 bug
Observable misbehaviour
 feature
Substantial extension; no full spec required
 idea
Droppable; no commitment
No Label feature
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
claude (Claude) Brummel
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Brummel/Aura#124
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?