Brummel/Aura
1
0
Fork 0
Code Issues 31 Pull Requests Actions Packages Projects Releases Wiki Activity

Give dataset identity room for a second provider and tick resolution #274

New Issue
Open
opened 2026-07-15 10:35:47 +02:00 by claude · 1 comment
claude commented 2026-07-15 10:35:47 +02:00
Collaborator
Copy Link
Copy Source

Today a dataset is identified by a bare symbol string — "FRA40", "GER40". Two axes are silently collapsed into that one string, and both are set to change: which provider supplied the data, and at which resolution it was recorded. Neither has any representation today — not in the selection syntax, not in the store, not in the file index.

The two collapsed axes

  • Provider. There is one data provider today — Pepperstone, the bars-only NFS archive #124 describes ("the Pepperstone NFS files today are bars-only … no instrument-structural metadata is carried", 2026-06-22) — so the symbol string uniquely names a dataset. The moment a second provider carries an overlapping instrument — GER40 from Pepperstone and from another source — "GER40" no longer identifies one dataset, and no authored reference can say which.
  • Resolution. The archive carries M1 bars, so "the FRA40 data" is unambiguous. Claim (from the project owner; not independently confirmed — the archive root was unreachable to inspect for curated tick files at filing time): real tick data exists beside the M1 bars. Once tick is curated in, "FRA40" no longer names one series but must select between M1 and tick.

Where the flat key sits (verified against the engine source)

The symbol string is the shared identity across the pipeline — but not uniformly, and the precise spread bounds any migration:

  • Campaign / research documents carry it directly: data.instruments: Vec<String> (crates/aura-research/src/lib.rs:475).
  • The monthly file index keys on it: list_m1_months(data_path, symbol) scans for <symbol>_YYYY_MM.m1 files (crates/aura-ingest/src/lib.rs:460).
  • The #272 fault surface names the faulting member by it: CellRealization.instrument: String (crates/aura-registry/src/lineage.rs:149).
  • The content-addressed store embeds it but does not key on it: the store key is the SHA256 of the canonical document bytes (content_id_of, crates/aura-research/src/lib.rs:712), and the symbol lives inside those bytes (a campaign doc's instruments). Changing the identity shape therefore re-serializes those docs to new content ids — the store is in the blast radius as content, not as key.
  • Blueprints do not carry it at all: they are instrument-agnostic topology; a bare run binds the instrument at run time via the --real <SYM> flag (crates/aura-cli/src/main.rs:553), and only campaign documents name instruments statically.

So a move from symbol to a structured identity is a core-vocabulary migration touching authored campaign documents, the file index, the fault surface, and (by re-hashing) the store at once — while leaving blueprints untouched. That cross-surface reach is why it reads as milestone-scale, not a single issue's worth of work.

Filed now, not built now

Building any of this today would be premature: there is one provider and one resolution, so a multi-provider selector that always says "Pepperstone" is ballast, and a guessed identity shape is an abstraction that has to be torn out when the real requirement arrives. What is recorded now is the analysis — which axes collapse, where the flat key sits, why it is expensive — because that analysis is cheap while fresh and costly to reconstruct once a second provider or a tick import forces the question under time pressure.

Not the inspect surface's problem

The aura data info reshaping (#273) deliberately carries no provider or resolution field and no stub for either. That verb inspects a dataset; it does not select one, and its JSON output is additively extensible — a provider or resolutions field can appear there for free once the axes are real, in the shape reality then dictates. The identity/selection problem is upstream of the inspect surface and is what this issue holds.

Open questions (not decided here)

  • Is the provider key provider:symbol, a separate field, or a list (one instrument recorded by several providers)?
  • Is resolution a third axis of the same key, or a per-dataset attribute selected at run time?
  • How does an authored campaign document select among providers/resolutions without breaking the single-provider, single-resolution documents that exist today (a default-when-unambiguous rule)?
Today a dataset is identified by a bare symbol string — `"FRA40"`, `"GER40"`. Two axes are silently collapsed into that one string, and both are set to change: which **provider** supplied the data, and at which **resolution** it was recorded. Neither has any representation today — not in the selection syntax, not in the store, not in the file index. ## The two collapsed axes - **Provider.** There is one data provider today — Pepperstone, the bars-only NFS archive #124 describes ("the Pepperstone NFS files today are bars-only … no instrument-structural metadata is carried", 2026-06-22) — so the symbol string uniquely names a dataset. The moment a second provider carries an overlapping instrument — `GER40` from Pepperstone *and* from another source — `"GER40"` no longer identifies one dataset, and no authored reference can say which. - **Resolution.** The archive carries M1 bars, so "the FRA40 data" is unambiguous. Claim (from the project owner; not independently confirmed — the archive root was unreachable to inspect for curated tick files at filing time): real tick data exists beside the M1 bars. Once tick is curated in, `"FRA40"` no longer names one series but must select between M1 and tick. ## Where the flat key sits (verified against the engine source) The symbol string is the shared identity across the pipeline — but not uniformly, and the precise spread bounds any migration: - **Campaign / research documents** carry it directly: `data.instruments: Vec<String>` (`crates/aura-research/src/lib.rs:475`). - **The monthly file index** keys on it: `list_m1_months(data_path, symbol)` scans for `<symbol>_YYYY_MM.m1` files (`crates/aura-ingest/src/lib.rs:460`). - **The #272 fault surface** names the faulting member by it: `CellRealization.instrument: String` (`crates/aura-registry/src/lineage.rs:149`). - **The content-addressed store** embeds it but does *not* key on it: the store key is the SHA256 of the canonical document bytes (`content_id_of`, `crates/aura-research/src/lib.rs:712`), and the symbol lives inside those bytes (a campaign doc's `instruments`). Changing the identity shape therefore re-serializes those docs to new content ids — the store is in the blast radius as *content*, not as *key*. - **Blueprints do not carry it at all**: they are instrument-agnostic topology; a bare run binds the instrument at run time via the `--real <SYM>` flag (`crates/aura-cli/src/main.rs:553`), and only campaign documents name instruments statically. So a move from `symbol` to a structured identity is a core-vocabulary migration touching authored campaign documents, the file index, the fault surface, and (by re-hashing) the store at once — while leaving blueprints untouched. That cross-surface reach is why it reads as milestone-scale, not a single issue's worth of work. ## Filed now, not built now Building any of this today would be premature: there is one provider and one resolution, so a multi-provider selector that always says "Pepperstone" is ballast, and a guessed identity shape is an abstraction that has to be torn out when the real requirement arrives. What is recorded now is the analysis — which axes collapse, where the flat key sits, why it is expensive — because that analysis is cheap while fresh and costly to reconstruct once a second provider or a tick import forces the question under time pressure. ## Not the inspect surface's problem The `aura data info` reshaping (#273) deliberately carries no `provider` or `resolution` field and no stub for either. That verb *inspects* a dataset; it does not *select* one, and its JSON output is additively extensible — a `provider` or `resolutions` field can appear there for free once the axes are real, in the shape reality then dictates. The identity/selection problem is upstream of the inspect surface and is what this issue holds. ## Open questions (not decided here) - Is the provider key `provider:symbol`, a separate field, or a list (one instrument recorded by several providers)? - Is resolution a third axis of the same key, or a per-dataset attribute selected at run time? - How does an authored campaign document select among providers/resolutions without breaking the single-provider, single-resolution documents that exist today (a default-when-unambiguous rule)?
claude added the feature label 2026-07-15 10:35:47 +02:00
claude commented 2026-07-21 10:36:39 +02:00
Author
Collaborator
Copy Link
Copy Source

Re-anchoring note after the #295 shell-boundary extraction (merged 4ed6455): four of this issue's five source citations are unchanged and still accurate verbatim — crates/aura-research/src/lib.rs:475 (pub instruments: Vec<String>), crates/aura-research/src/lib.rs:712 (content_id_of), crates/aura-ingest/src/lib.rs:460 (list_m1_months), and crates/aura-registry/src/lineage.rs:148 (CellRealization.instrument, off by one from the original #149 due to unrelated edits). None of aura-research, aura-ingest, or aura-registry were touched by #295's extraction.

The fifth citation has drifted. crates/aura-cli/src/main.rs:553 no longer names the --real <SYM> flag; that line is now mc_member_line, an unrelated report-formatting helper. The --real resolution now lives at data_choice_from (crates/aura-cli/src/main.rs:~1576), and the DataChoice/DataSource type it constructs was moved out of aura-cli into the new aura-runner crate as part of #295 (crates/aura-runner/src/family.rs, imported at crates/aura-cli/src/main.rs:56). Binding an instrument at run time is therefore no longer a single-crate concern: the flag parse stays in aura-cli, but the type it produces, and the family builders that consume it, live in aura-runner.

This does not change the issue's thrust. The decided document-first direction (#300) does not list --real among the flags migrating into document vocabulary (that list is --axis, --from/--to, --stop-length/--stop-k, --select, --block-len, --resamples, --seeds) — instrument selection is already document-expressible today via DataSection.instruments, so --real remains CLI sugar for the ad-hoc single-instrument case rather than a flag needing new document vocabulary invented for it. Read together with the ledger's C25 amendment (text artifacts are the canonical layer; every control surface is a projection over them), the natural home for a future provider/resolution axis is DataSection in aura-research, not a new CLI flag family — which answers this issue's open question about where the axis belongs, without deciding its shape (provider:symbol vs. separate field vs. list), which remains genuinely open. See also the C26 ledger entry (docs/design/INDEX.md, aura-runner's binding module) for the adjacent post-#295 crate split on column binding.

Re-anchoring note after the #295 shell-boundary extraction (merged 4ed6455): four of this issue's five source citations are unchanged and still accurate verbatim — `crates/aura-research/src/lib.rs:475` (`pub instruments: Vec<String>`), `crates/aura-research/src/lib.rs:712` (`content_id_of`), `crates/aura-ingest/src/lib.rs:460` (`list_m1_months`), and `crates/aura-registry/src/lineage.rs:148` (`CellRealization.instrument`, off by one from the original #149 due to unrelated edits). None of aura-research, aura-ingest, or aura-registry were touched by #295's extraction. The fifth citation has drifted. `crates/aura-cli/src/main.rs:553` no longer names the `--real <SYM>` flag; that line is now `mc_member_line`, an unrelated report-formatting helper. The `--real` resolution now lives at `data_choice_from` (`crates/aura-cli/src/main.rs:~1576`), and the `DataChoice`/`DataSource` type it constructs was moved out of aura-cli into the new `aura-runner` crate as part of #295 (`crates/aura-runner/src/family.rs`, imported at `crates/aura-cli/src/main.rs:56`). Binding an instrument at run time is therefore no longer a single-crate concern: the flag parse stays in aura-cli, but the type it produces, and the family builders that consume it, live in aura-runner. This does not change the issue's thrust. The decided document-first direction (#300) does not list `--real` among the flags migrating into document vocabulary (that list is --axis, --from/--to, --stop-length/--stop-k, --select, --block-len, --resamples, --seeds) — instrument selection is already document-expressible today via `DataSection.instruments`, so `--real` remains CLI sugar for the ad-hoc single-instrument case rather than a flag needing new document vocabulary invented for it. Read together with the ledger's C25 amendment (text artifacts are the canonical layer; every control surface is a projection over them), the natural home for a future provider/resolution axis is `DataSection` in aura-research, not a new CLI flag family — which answers this issue's open question about where the axis belongs, without deciding its shape (provider:symbol vs. separate field vs. list), which remains genuinely open. See also the C26 ledger entry (docs/design/INDEX.md, `aura-runner`'s `binding` module) for the adjacent post-#295 crate split on column binding.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
BLOCKER
Core functionality blocked; preempts other selection
 bug
Observable misbehaviour
 feature
Substantial extension; no full spec required
 idea
Droppable; no commitment
No Label feature
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
claude (Claude) Brummel
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Brummel/Aura#274
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?