f0aadb54f80c327ed564be06cf47ab8ddaec2dcd
22 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 claude
|
529a8af0d6 |
feat(cli,ingest): aura data coverage — month-gap inventory for a symbol
aura data coverage <SYMBOL> prints the archive's monthly file-index coverage: a span line (first..last present month) and either an explicit 'no gaps' line or one 'missing: YYYY-MM..YYYY-MM' line per interior contiguous hole. The Copper failure mode — a campaign aborting mid-run because a first/last-bounds check passed while the window start had no data — is now visible before any run (the real archive reports 'Copper missing: 2017-04..2019-09'). An unknown symbol (no archive files) refuses on stderr with exit 1; informational absence stays prose + exit 0 per the verb corpus convention. Mechanics: list_m1_months goes pub in aura-ingest (archive_extent deliberately walks past gaps and cannot report them); a new two-level 'aura data' clap namespace follows the Nodes precedent; the archive root resolves through the normal project path (paths.data override, data-server default otherwise). Pure report fn with unit tests for the refusal, the gapless span, and the collapsed-range shape; the headline e2e runs hostless over a new gapped synthetic-archive fixture. refs #264 (cut 2, aura data list, follows separately) |
||
|
claude
|
1ccce9ad32 |
fix(test): anchor all test sandboxes on reclaimable pid-free names
Every hand-rolled test-sandbox helper keyed its directory on
std::process::id() under env::temp_dir(), so the pre-create wipe never
matched a prior run's path: one directory stranded per helper site per
test-binary invocation, >50k dirs / 15.5 GiB on the 16 GiB tmpfs by
2026-07-13.
The remedy extends commit 84e1075's knob-lab pattern to every site:
fixed, tag-keyed, PID-FREE names under the build-tree tmp anchor, each
site keeping its pre-create remove_dir_all — which now genuinely
reclaims the previous run. Integration/bench targets use
env!("CARGO_TARGET_TMPDIR"); src-internal unit-test helpers (cargo sets
that var only for integration/bench targets) derive the same anchor
from env!("CARGO_MANIFEST_DIR") + ../../target/tmp. Residue is bounded
to one directory per site, on disk instead of the tmpfs, per checkout.
One deliberate exception: project_new.rs's tmp() stays under
env::temp_dir() because two of its tests (new_outside_a_work_tree_*)
need a base genuinely detached from any git work tree, and target/
lives inside the checkout's work tree. Its name carries a per-checkout
hash discriminator instead, so concurrently running checkouts (which
share /tmp) cannot race on the fixed name while each run still
reclaims its predecessor.
tempfile-RAII was considered and rejected (issue #258 decision log):
it cannot cover the process-lifetime OnceLock scaffolds (statics never
drop), adds a dependency, and still strands kill-leftovers on the
tmpfs.
Verified: full workspace suite green (incl. the new RED regression
test temp_cwd_sandbox_does_not_leak_under_env_temp_dir), clippy clean,
sandboxes observed under target/tmp with stable names across runs.
Stale pre-fix PID-keyed dirs in /tmp are not swept by this change and
age out only by manual cleanup.
closes #258
|
||
|
claude
|
a895891ab1 |
perf(ingest,cli): derive archive bounds from the monthly file index — probe_window drain retired
probe_window drained a source's entire close column just to learn its first and last bar timestamp; both live callers (open_real_source and campaign_window_ms on the campaign trunk) now resolve through aura-ingest's new archive_extent(): list the symbol's SYM_YYYY_MM.m1 files, load only the boundary months via the existing loader seam, and walk across gap months forward/backward — O(2 file loads) typical instead of O(archive). Empty-window refusal semantics are unchanged (pinned). Two characterization tests captured the resolved windows against the OLD implementation over the synthetic per-test archive and stay green unchanged after the swap. Measured: the probe itself drops 366ms -> 13ms (~28x) on the full 2014-2026 GER40 archive; end-to-end command wall-clock moves only ~2-3% because the sim loop dominates — the fix removes an O(archive-size) term, not the dominant cost. closes #252 |
||
 Brummel
|
595c98b264 |
feat(cli,ingest): binding module + generalized column openers (#231 tasks 1-2)
crates/aura-cli/src/binding.rs: the closed column vocabulary (open/high/ low/close/spread/volume, price as the close alias) with ResolvedBinding — one ordered plan both halves of the single-price weld will share. resolve_binding applies override-wins-over-name-default, refuses unknown roles / bad override values / the price+close double-alias / a rebound reserved close role, synthesizes the broker-only close entry, and sorts into canonical M1Field order (the C4 merge tie-break contract). 12 exact-prose unit tests. Module-wide dead_code allow is scaffolding until the wrap seam consumes it next. aura-ingest: open_columns / open_columns_window open any field set in the given order (None if any field lacks an overlapping archive file); open_ohlc now delegates with the fixed OHLC order — behaviour-identical, still guarded by the open_ohlc_seam bit-identity test. Post-review polish: RESERVED_CLOSE_ROLE constant couples the synthesized close's two sites; dead_code note reworded condition-based. Verified: binding 12/12, ingest absence test green, full workspace suite green, clippy -D warnings clean; independent quality review approved. refs #231 |
||
|
Brummel
|
b7aaa0ba59 |
feat(cli): dissolve the real-data blueprint sweep into the campaign path (#210 T4)
aura sweep <bp.json> --real ... --axis ... now runs as thin sugar over its generated campaign document through the one campaign executor: the dispatch registers the canonical blueprint (topology_hash IS content_id_of over the same canonical bytes, so the strategy ref resolves against the existing single store write), strips the CLI's wrapped axis names to the raw campaign namespace (#203 bind convention), converts the probed archive window back through the ingest seam's own epoch_ns_to_unix_ms (made pub; C3 — the seam owns the ms/ns convention, never an inline division), and hands off to verb_sugar::run_sweep_sugar. The inline real-data arm is gone; the synthetic arm and --list-axes are untouched. campaign_run gains RunPresentation {Full, MemberLinesOnly}: run_campaign keeps Full (record line unchanged, project gate unchanged); the sugar path runs MemberLinesOnly — the generated doc's emit already limits stdout to family_table member lines, the mode suppresses only the final record line. The record append is identical in both modes; no project gate on the by-id path (the dissolved verb works project-less, exactly as before). The characterization pin survives the re-cut with exactly the one sanctioned assertion flip (instrument-absence -> instrument==GER40) and now also pins the generated artifacts: one process + one campaign document auto-registered (dedup on a second identical invocation), raw axis keys in the stored document, the stored window a ms sub-range of the request, C1 at the member-report level with the registry's run-suffix increment isolated. Suite 1055/0, clippy clean, gated real-data paths executed locally. refs #210 |
||
|
Brummel
|
7e4b91ab5f |
feat(0074): source real-path pip from the geometry sidecar; drop the authored floor
The data-server now ships per-symbol geometry sidecars, so aura no longer hard-codes instrument geometry. A consumption grep showed production reads exactly one `InstrumentSpec` field — `pip_size` — at the real-data path; the other eight (incl. `instrument_id`) were constructed but never read. - aura-cli resolves the real-path pip from the recorded sidecar (`instrument_geometry` -> `InstrumentGeometry.pip_size`), refusing (exit 2) on absent geometry. The authored table no longer gates: any symbol with a sidecar + bars now runs (new test: USDJPY, never vetted, resolves its JPY pip 0.01 from the sidecar). - Removed the redundant authored floor: `InstrumentSpec`, `instrument_spec`, `VETTED_SYMBOLS`, and the table-only tests. The `InstrumentGeometry` seam (the surviving pip source) and its nameability guard are kept. - `instrument_id` (the "find out"): no production consumer. The position-event table's `instrument_id` is a decoupled caller parameter; the engine never imports an instrument spec. Dropped with the struct. - The example `pip_size_of` is re-sourced to the sidecar; the pure-structural blueprint tests use a literal pip to stay archive-independent. Speculative deploy-grade fields and the #124 override/floor tiers are deferred to the Stage-2 broker that will consume them, read from the sidecar geometry then. Ledger (C8/C15/#22/#106) updated; #124 collapses to "recorded sidecar geometry -> refuse" in the meantime. Verified: cargo test --workspace (all green), clippy -D warnings clean, cargo doc clean, with the local archive present so the sidecar paths run. refs #124 |
||
|
Brummel
|
dda20abc28 |
audit(0073): cycle close — lift tier-2 geometry to ledger, finish VETTED_SYMBOLS, drop ephemera
Cycle-close audit: drift-clean on all load-bearing invariants (C7/C8/C15 hold; geometry stays non-scalar metadata, the Source seam still carries only (Timestamp, Scalar); scope boundary held — no resolver, quote_currency still &'static str, both #124's). Tidy: completed the VETTED_SYMBOLS consolidation the feat commit over-claimed — migrated the two remaining inline vetted-symbol arrays in the crate-internal unit tests (instrument_spec_pip_value_matches_contract_times_pip, instrument_spec_ids_are_distinct) to the const, so #140 now extends one list. Ledger: lifted this cycle's durable rationale into the C15 realization note (docs/design/INDEX.md) — the floor now carries tick_size/digits, and the recorded-metadata tier-2 source (instrument_geometry over symbol_meta -> neutral InstrumentGeometry, cross-checked against the authored floor) now exists at the ingestion edge; the tier-composing resolver remains #124. Drop the ephemeral cycle spec + plan (docs/specs, docs/plans) per the lifecycle convention; durable rationale now in the ledger + the #143 thread. refs #143 |
||
|
Brummel
|
5b7cd5eb2d |
feat(0073): consume DataServer geometry — cross-check vetted floor + tick_size/digits
Make data-server's neutral InstrumentGeometry a first-class, broker-agnostic input at the aura-ingest edge: re-export InstrumentGeometry + a thin instrument_geometry(server, symbol) accessor over symbol_meta (the recorded dataset-metadata tier 2 of #124's hierarchy; the tier-composing resolver stays #124). Add the two deferred deploy-grade InstrumentSpec fields tick_size + digits (additive, struct stays Copy; C15), authored for the 5 vetted symbols from the verified sidecars. A new real-data cross-check asserts the authored floor agrees with provider geometry (pip_size, contract_size==lot_size, the C10 pip_value invariant, tick_size, digits, quote_currency), iterating the vetted set so #140's additions are covered for free; skip-if-archive-absent. Scope (derived, recorded on #143): no runtime resolver and no quote_currency type change — both remain #124's. 'quote_currency beyond FX' is satisfied at the geometry tier (InstrumentGeometry.quote: String), not by widening the &'static str floor. Beyond the plan (implementer additions, reviewed and accepted): hoisted the vetted-symbol list to one pub const VETTED_SYMBOLS, collapsing the three drift-prone copies plan-recon flagged (#140 now extends one constant); a tick_size == 10^(-digits) consistency guard (verified to hold across all 30 provider sidecars, so it will not false-trip on #140); a re-export-nameable seam test guarding the InstrumentGeometry re-export from silent removal. Verification: cargo test --workspace all green; cargo clippy --workspace --all-targets -D warnings clean. The cross-check ran against the live Pepperstone archive (not skipped) — all 5 vetted sidecars match the authored table. refs #143 |
||
|
Brummel
|
b91c89f964 |
feat(broker-foundation): InstrumentSpec deploy metadata + PositionEvent schema
The realistic-broker milestone's (C10 A-side) two foundation value types — no broker, no derivation yet, fully tested. #113 — InstrumentSpec gains six deploy-grade fields (instrument_id, contract_size, pip_value_per_lot, min_lot, lot_step, quote_currency) and the vetted table is populated for GER40/FRA40/EURUSD/GBPUSD/USDCAD. The struct stays Copy (quote_currency is &'static str); the _ => None refuse-don't-guess arm is the permanent floor. This is the milestone's permanent authored floor (recorded-metadata tier is forward work, #124). #114 — PositionAction { Buy, Sell, Close } (closed enum, serde-encoded as a bare i64 0/1/2 via From/TryFrom) + PositionEvent in aura-engine beside RunMetrics, faithful to the C10 column spec: event_ts, action, position_id, instrument_id, unsigned volume; no open_ts; direction is the action (no signed-volume trick). Re-exported from the crate root. Derived fork decisions (recorded on #113): quote_currency &'static str; action as ordinal i64; stable instrument_id + overridable vetted values. Verified by the orchestrator: cargo test --workspace = 458 passed / 0 failed; clippy --all-targets -D warnings clean. An adversarial 4-lens review (ledger faithfulness, downstream soundness, serde/wire robustness, test honesty) returned 3 SOUND + 1 CONCERN: quote_currency &'static str cannot hold a runtime-sourced currency at the #124 resolver — not a blocker (#115/#116 read only the authored floor), logged on #124 as a deliberate deferred design choice. The implementer's deviation from the plan snippet (vec![..] -> [..]) avoids clippy::useless_vec; semantics identical. Two tester-authored consumer-boundary E2E tests added beyond the inline units: instrument_spec_deploy_metadata.rs (public resolution seam) and position_event_table.rs (persisted bare-int wire shape + reversal table). closes #113 #114 |
||
|
Brummel
|
26bec6d64a |
feat: per-instrument pip metadata channel for the sim-optimal broker
The data-server symbol stream carries only price bars, no instrument metadata.
So SimBroker was constructed with one global pip_size literal (0.0001, correct
only for 5-decimal FX), and `aura run --real <symbol>` emitted wrong-unit pip
equity for any non-FX instrument (AAPL.US, BTCUSD off by orders of magnitude).
Add the missing channel to *specify* per-instrument pip:
- aura-ingest: `InstrumentSpec { pip_size }` + `instrument_spec(symbol)` — a
typed, Rust-authored vetted lookup (NOT Aura.toml; the later Aura.toml schema
can subsume it). Seeded GER40/FRA40 = 1.0 (index points), EURUSD/GBPUSD/USDCAD
= 0.0001 (5-dp FX majors); None for an un-specced symbol.
- aura-cli: `sample_harness` and `sim_optimal_manifest` gain a `pip_size`
parameter; `run_sample_real` looks the pip up by symbol BEFORE any data access
and refuses (exit 2) an un-specced instrument rather than guessing — honest by
construction. The looked-up pip reaches both the broker divisor and the
manifest broker label (`format!`). Every synthetic caller passes the unchanged
0.0001, now named `SYNTHETIC_PIP_SIZE`. SimBroker (aura-std) is untouched.
Scope: narrowed to the CLI real path — the actual bug. The runnable GER40
examples already bake the correct 1.0 and are NOT buggy; centralizing them
through the lookup is deferred to #98.
Decisions (user-directed, recorded on #22): metadata channel = typed
InstrumentSpec at the source edge; un-specced real symbol refuses; seed =
GER40/FRA40 + FX majors; example refactor deferred. The spec went through the
grounding-check gate; the auto-sign panel surfaced the refusal-behaviour and
scope as genuine design decisions, escalated and resolved by the user.
Tests: instrument_spec unit; manifest pip rendering (1.0 -> "1", 0.0001
unchanged); non-gated CLI refusal (exit 2, no stdout leak, vetted symbol clears
the lookup); gated GER40 binary-boundary honesty (pip_size=1 in the emitted
manifest, deterministic). The pre-existing run_sample_real determinism test was
retargeted AAPL.US -> GER40 (AAPL.US is un-specced and would now refuse at the
lookup); the AAPL.US bounded-window streaming property still lives in
aura-ingest/tests/streaming_seam.rs (literal source, no spec lookup).
Verified: cargo build --workspace, clippy --all-targets -D warnings, and
cargo test --workspace all green; the gated GER40 path ran with local data.
closes #22
|
||
|
Brummel
|
8c9a1b4630 |
feat(aura-ingest): consolidate the real-data source-open seam
A real OHLC source now builds from aura-ingest alone, in the engine-native epoch-ns Timestamp currency, with the one ms<->ns crossing owned by the seam. Library surface (src/lib.rs), all additive — the ms-based `open` and `unix_ms_to_epoch_ns` are untouched, so every existing call site is preserved byte-for-byte: - epoch_ns_to_unix_ms (private): the seam-owned inverse; consumers never convert (#80). - M1FieldSource::open_window: the Timestamp-window mirror of `open`, mapping each bound through the private inverse and delegating to `open`. - open_ohlc: the canonical OHLC bundle opener — four M1FieldSources in the fixed open/high/low/close C4 merge order, sharing one Arc<DataServer> (one cache, C12); the order lives in one vetted place (#92). - default_data_server + `pub use data_server::{DataServer, DEFAULT_DATA_PATH}`: a real-data source builds without naming the external data_server crate (#81). Consumer migration: the shared breakout_real.rs helpers and every GER40 example/test move to the Timestamp-native surface — open_ohlc_sources removed, utc_month_window_ms -> utc_month_window (Timestamp), report_from_trace retyped, the per-consumer ns_to_ms hand-divides deleted, the direct data_server imports replaced by the re-exports. ger40_breakout_compare.rs (which opened OHLC by its own hand-loop, reached by neither retyped helper) is migrated by inspection. Return type is Vec, not [_; 4]: Harness::run consumes Vec<Box<dyn Source>>, so it feeds straight in; order-safety comes from the single helper either way. Tests: a hermetic round-trip pinning the inverse; a gated behaviour-preservation test (open_ohlc Timestamp path == ms-path open, bit-identical recorded series); a hermetic absent-archive fixture pinning the file-level None contract and the #81 re-export firewall. Verified: cargo build --workspace --all-targets clean; cargo test --workspace green (the gated GER40 tests ran against the local archive — open_ohlc_seam proves byte-identity on real data, not a skip); clippy --all-targets -D warnings clean. Acceptance greps pass: open_ohlc_sources gone, no consumer-side ns_to_ms, no data_server import in any migrated GER40 consumer, epoch_ns_to_unix_ms private. closes #80, #81, #92 |
||
|
Brummel
|
f66036bd61 |
feat(aura-engine): Source::resident_records — probe ring residency via the trait
The residency probe was an inherent method on the concrete M1FieldSource, so a Box<dyn Source> consumer could not read it without downcasting (the adjacent friction noted in #95). Hoist it onto the Source trait as resident_records(&self) -> Option<usize> with a default of None, mirroring bounds()'s Option-for-unknown idiom: None = "does not report" (the eager VecSource, which has no streaming ring), distinct from Some(0) = "reports, holds none" (e.g. exhausted). M1FieldSource overrides it to return Some(chunk len). Callers updated (streaming_seam, the two deep-dive fieldtest bins). Tests: a hermetic VecSource default-is-None check, and a &dyn Source probe in the gated residency test proving M1FieldSource residency is readable without a downcast. refs #95 |
||
|
Brummel
|
0d7c2c16f4 |
docs: narrow streaming residency claim from whole-process RSS to the source ring
The C12 realization and the 0041 spec claimed streaming residency is O(one chunk) at the process level ("a 20-year window streams in the same memory as a one-day one"). That holds only for the aura source ring (M1FieldSource::resident_records(), bounded by one chunk and correctly tested); whole-process RSS grows O(records-touched) because data-server's FileCache retains each window's parsed chunks (~56 B/record) read-only for the pass. Reproduced: 6 MiB @ 1mo -> 173 MiB @ ~10y, linear.
That retention is the replay-many optimization (load-once; close+volume share one parse via the field-agnostic FileKey), not a leak, and no always-on eviction can bound a single forward pass without regressing it. This narrows the docs/comments to the true per-source-ring property and names the FileCache per-window retention as the real, replay-amortized process cost. The streaming_seam assert is unchanged (it was always the ring bound; only its labelling confused ring vs process). The deferred opt-in non-retaining streaming read path is parked as Brummel/data-server#2.
closes #95
|
||
|
Brummel
|
6390093f93 |
refactor(aura-ingest): chunk-direct M1 transpose, drop the AoS intermediate
load_m1_window collected every bar into an intermediate Vec<M1Parsed> (a full AoS materialization, un-presized so it grew by reallocation) before transpose_m1 copied it again into the SoA columns — ~2x peak load memory plus realloc churn. Transpose each chunk straight into the columns instead, via a new shared M1Columns::extend_from_bars (reserves per chunk for amortized growth). One fewer full copy, halved peak. Behaviour-preserving: the resulting M1Columns is byte-identical and transpose_m1's / load_m1_window's signatures are unchanged. Pinned by a new parity test that needs no live DataServer. closes #78 |
||
|
Brummel
|
cd3d1ca9ed |
refactor(aura-core): split Scalar into a tag-free Cell + ScalarKind
Motivation
----------
`Scalar` was a tagged enum (I64/F64/Bool/Ts), so every scalar value
physically carried its own kind tag. But the kind is already known from
the schema/port/column the value flows through (C7: the type is a
property of the column, not of the value — the hot path is already
columnar `Column<T>`, and `AnyColumn::get` *reconstructs* the tag from
the column on the way out). The per-value tag was therefore redundant
with the kind the surrounding context already holds.
That redundancy had three costs:
* It baked an implicit `match` (a branch) into every function that read
a Scalar payload — even where the caller statically knew the type.
The tag could never be exploited away.
* Size: a tagged enum is tag + payload = 16 bytes (f64/i64 alignment),
twice the 8 bytes the value needs. A `Column<Scalar>` would be double
the memory and half the cache utilisation.
* It is the shared root of several downstream papercuts we keep hitting
— the lossy f64 manifest field, the `unreachable!` panic on a
non-numeric param, the serde-tag question — all symptoms of "the type
is baked into the value".
Change
------
Introduce `Cell`: a type-erased 64-bit word (`struct Cell(u64)`) that is
not readable without external type context. It is constructed per base
type (`from_i64/from_f64/from_bool/from_ts`) and read only by naming the
type at the call site (`i64()/f64()/bool()/ts()`) — each a branch-free
bit-cast. The hot path resolves the kind once at the boundary (from the
schema) and then reads natively, with no per-value branch. `Cell` knows
nothing of `Scalar` or `ScalarKind`; the dependency is strictly one-way,
and it lives in its own `cell.rs` (more is planned on top of it).
`Scalar` becomes `struct { kind: ScalarKind, cell: Cell }` — the
self-describing form for the dynamic boundaries (builder binding,
serialization, rendering), built on top of `Cell`. Its `as_*` accessors
now `debug_assert` the kind and return the native value (free in
release); calling the wrong accessor is a caller bug, not a checked
`Option`. The variant constructors `Scalar::I64(..)` become associated
fns `Scalar::i64(..)`.
`PartialEq` is hand-written (not derived) to preserve the former enum's
value semantics: kinds must match, then native payloads compare, so f64
keeps IEEE-754 behaviour (`NaN != NaN`, `+0.0 == -0.0`) and a kind
mismatch is never equal even when the raw words coincide. A fixture
(`scalar_eq_is_value_not_bitwise`) pins exactly the cases where bit- and
value-equality diverge, so it can't silently regress. `Cell`'s own
`Eq`/`Hash` stay bitwise — correct for a raw word.
The change is behaviour-preserving: Scalar's observable behaviour is
identical to the pre-Cell enum (the value-equality fixture proves it);
only the internal representation changed. The ~440 call sites across the
workspace are a mechanical constructor rename plus ~12 destructuring
sites (match-arms / `let`-patterns) rewritten to `kind()` + `as_*`.
Verified: cargo build --workspace --all-targets, cargo clippy --workspace
--all-targets -- -D warnings, cargo test --workspace — all green.
|
||
|
Brummel
|
1d9555c468 |
feat(aura-engine,aura-registry): producer-supplied window + registry lineage (0045 iter 1)
Iteration 1 of spec 0045 — the engine + registry core for storing orchestration families as linked records. aura-engine: add Source::bounds() — the inclusive (from, to) data extent a producer will stream, known without materialization (#71 firewall) — and a free window_of() that folds the union extent across a run's sources. This is the producer-supplied replacement for the prices.first()/.last() Vec scans at the CLI call sites (those migrate in iteration 2); a lazy producer now reports its window so stored lineage stays byte-identical whether a source is eager or streamed. aura-ingest: M1FieldSource stores its requested [from_ms, to_ms] window at open and reports it via bounds() (normalized to epoch-ns; None when open-ended — the archive-extent query is a deferred non-goal). aura-registry: new lineage module — FamilyKind / FamilyRunRecord (a RunReport stamped with family_id + kind + ordinal) / Family, plus Registry::append_family (assigns family_id = "{name}-{counter}" via a per-name generate-and-check counter, writes members to a sibling families.jsonl) and load_family_members; group_families re-derives the families (the round-trip), and the three *_member_reports extractors pull the per-kind member reports. The flat runs.jsonl store and its append/load/rank_by/optimize API are byte-for-byte unchanged (a test pins the two stores' disjointness). C9 preserved: aura-engine gains no registry dependency. New dep: serde derive on aura-registry (per-case policy, same basis as serde_json already is). Verification: cargo build --workspace; cargo test --workspace (228 green, incl. 3 new engine bounds/window_of, 1 data-gated ingest bounds run against real archive data, 5 new registry lineage round-trip/counter/disjointness tests); cargo clippy --workspace --all-targets -D warnings clean. The CLI surface (aura mc, family-aware aura runs, the window_of migration) is iteration 2. refs #70 |
||
|
Brummel
|
682e459554 |
audit: cycle 0041 — drift-clean (Source ingestion seam)
Architect drift review over 8b330e3..HEAD (the Source-seam cycle #71 plus the interim #67 optimize work and refactors). No semantic or contract drift: the seam faithfully preserves C3 (ms→epoch-ns at the one ingestion boundary), C4 (tie-by-source-index, byte-for-byte), and C7 (field-per-source SoA); the full suite is byte-identical green and the gated streaming tests pass on real data. Regression gate: `cargo test --workspace` green (the project declares no dedicated regression script; the suite is the gate). Tidied stale prose the cycle left behind (no behaviour change): - docs/design/INDEX.md (C12): replaced the cycle-0011 "deliberate eager gap" status note with a cycle-0041 realization note — the producer `Source` seam + streaming `M1FieldSource` now deliver per-source O(one-chunk) streaming; precisely scopes what remains open (cross-*sim* Arc<[T]> window sharing, still unbuilt until the orchestration families #66/#68/#69 consume it). - aura-core/src/lib.rs: dropped the now-shipped "Source trait + data-server ingestion" from the module doc's "still to come" list (aura-engine's twin line was fixed in the cycle; aura-core's was missed). - aura-ingest/src/lib.rs: module header now documents both coexisting paths (eager load_m1_window/M1Columns vs lazy streaming M1FieldSource). - aura-registry/src/lib.rs: rank_by/optimize doc comments linked to the private `metric_cmp` (a public→private intra-doc-link warning from the #67 commit); demoted to plain code spans, so `cargo doc --workspace` is now warning-clean. |
||
|
Brummel
|
8158e97dc6 |
feat(aura-ingest): streaming M1FieldSource — lazy data-server window through the seam
Completes the Source ingestion seam (plan Tasks 3-4): a real, lazily-streamed
data-server source proving the producer seam against the actual data source,
and closing the cycle-0011 deliberate eager-materialization gap.
- M1Field + a free decode(field, bar) — a pure projection of one M1 bar into
(epoch-ns ts, Scalar), reusing unix_ms_to_epoch_ns so ms->ns stays the one
C3 normalization point. Pure ⇒ hermetically unit-tested without an iterator.
- M1FieldSource: a streaming Source over a data-server M1 window. Holds one
Arc<[M1Parsed]> chunk (a zero-copy clone of the cache's chunk) + a cursor +
a pre-decoded head (so peek(&self) is cheap over a &mut next_chunk refill);
constructs each Scalar per-pull, refills when the chunk drains. Resident
footprint is O(one chunk), independent of window length.
- aura-engine moved [dev-dependencies] -> [dependencies] (M1FieldSource is a
library item implementing aura_engine::Source, not a test-only type).
- A gated streaming integration test (skips where local data is absent):
* end-to-end backtest through the seam to a RunReport, C1-deterministic
(two runs bit-identical);
* residency predicate — driving the source the way run() does (peek/next)
over the *unbounded* archive (~1.77M bars locally), peak resident_records
stays <= one data-server chunk (CHUNK_SIZE) while total >> CHUNK_SIZE:
O(one chunk), NOT O(window). A O(window) source would peak at `total`;
* two-field sharing (close + volume over one window, same ts axis).
The eager load_m1_window / close_stream path is kept (still valid for bounded
loads); the gap is closed by a streaming path now existing, not by deleting the
eager one. The residency test drives the full archive rather than the plan's
narrow 2006-08 window, which holds only ~21 bars locally — too few for the
> CHUNK_SIZE non-vacuity precondition; the unbounded archive is the stronger
"independent of window length" proof.
Verified: cargo build/test/clippy --workspace clean; the 3 gated tests ran
against real local data and passed; aura-ingest/aura-engine docs clean (two
pre-existing aura-registry doc-link warnings are from an earlier commit, not
this cycle).
closes #71
|
||
|
Brummel
|
eec876975c |
feat(engine): serde foundation + amend C16 dependency policy (cycle D / iter 1)
Iteration 1 of the run-registry cycle (#33): lay the dependency + serde foundation the registry's typed read-path needs, and amend the contract that forbade it. Contract change (load-bearing — C18/C16). C16's blanket "zero-external- dependency by commitment" + "aura-ingest is the sole external-dependency firewall" framing is struck and replaced by a considered, per-case dependency policy: dependencies are admitted by deliberate review (what a crate pulls in vs. what it buys), with particular scrutiny for anything entering the frozen deploy artifact (C13); standard vetted crates (serde, rayon, ...) pass that review and are used wherever they do the job, including in the bot; hand-rolling what a vetted crate does is the anti-pattern. C16's engine/project split + three-tier node reuse are unchanged. The five source comments that asserted the struck framing (aura-engine/aura-ingest Cargo.toml + aura-ingest/report.rs docs) are rewritten to match; no live source or ledger text still asserts it. Per-case justification for serde (the policy now requires one): it gives the run-report types a typed (de)serialization path — exactly what ranking/compare over stored runs needs, and what the writer-only hand-rolled JSON (C14) could never provide; closes the typed-read-path gap (#17). Its closure is tiny, ubiquitous, heavily audited, and its output deterministic (C1-safe). serde_json is test-only this iteration (dev-dependency); no production serde_json yet. Changes: [workspace.dependencies] serde (derive) + serde_json; serde derives on Timestamp (aura-core) and RunMetrics/RunManifest/RunReport (aura-engine), with serde_json round-trip tests (window renders as a [from,to] integer array via the transparent Timestamp newtype). The hand-rolled to_json writers are untouched (still drive stdout). No aura-registry crate, no SweepPoint change — those are iterations 2 and 3. Verified: cargo test --workspace green (incl. the two new round-trip tests); clippy --workspace --all-targets -D warnings clean; no surviving assertion of the struck zero-dep framing in live source or the ledger. refs #33 |
||
|
Brummel
|
ac4a8c68cf |
docs(aura-ingest): precise None contract for load_m1_window
The rustdoc claimed 'None if the symbol has no data in [from_ms, to_ms]', implying a bar-level None. Empirically (cycle-0011 fieldtest) None is file-level: it propagates data-server's own None, returned only when no archived file overlaps the window (far-future window or unknown symbol). A window that overlaps a loaded file but holds zero bars returns Some(M1Columns) with empty columns, not None. Document the distinction precisely (chosen over collapsing in-coverage- empty into None): None propagates data-server's one file-level meaning unchanged -- 'no data source to read from' -- rather than overloading it with a second aura-level 'source present but empty here' meaning. So Some/None means 'data source present' vs 'nothing to read from', not 'has bars' vs 'has none'; a consumer testing for an empty window must check cols.close.is_empty(). Doc-only, no behaviour change; ledger untouched (the contract is not referenced there). closes #18 |
||
|
Brummel
|
8bd5d0a3f9 |
fix(aura-ingest): express unbounded M1 window bounds via Option<i64>
load_m1_window took (from_ms: i64, to_ms: i64) and forwarded
Some(from_ms)/Some(to_ms) into stream_m1_windowed, throwing away the
underlying API's per-bound Option<i64> (None = unbounded). There was thus
no way to express an open upper bound, and the natural i64::MAX sentinel
overflowed chrono's from_timestamp_millis inside data-server's coarse
unix_ms_to_year_month file filter, panicking the whole ingest.
Thread Option<i64> through both bounds unchanged, so None = unbounded
matches the underlying contract and 'to the end of history' needs no
sentinel. Rejected the alternative of clamping a too-large bound to a
ceiling constant: a ceiling is not actually unbounded (it silently drops
any post-ceiling bar) and reintroduces a magic literal; restoring the
Option makes the open bound structurally expressible instead.
The committed RED test (
|
||
|
Brummel
|
a24729e97f |
feat(aura-ingest): data-server M1 ingestion at the one merge boundary
aura's first real data source (closes #7, Walking-skeleton milestone). A new aura-ingest workspace crate transposes data-server's AoS M1Parsed records into SoA base columns (C7), normalizes Unix-ms to canonical epoch-ns at the one ingestion boundary (C3), and feeds the existing k-way merge a real close-price stream so a backtest runs over real bars, deterministically (C1). Surface: - unix_ms_to_epoch_ns(time_ms) -> Timestamp (= ms * 1_000_000), the single C3 unit normalization. - M1Columns: the OHLCV bar as a bundle of base columns (open/high/low/close/ spread: f64, volume: i64, ts: epoch-ns) — SoA, C7. - transpose_m1(&[M1Parsed]) -> M1Columns: pure AoS->SoA (C1). - M1Columns::close_stream() -> Vec<(Timestamp, Scalar)>: the price input the SMA-cross sample strategy consumes. - load_m1_window(server, symbol, from_ms, to_ms): drains data-server's chronological chunks, transposes once at the boundary. Design (per spec 0011): - Eager materialization, not a lazy/shared Source abstraction: C12's cross-sim Arc<[T]> sharing has no consumer until the multi-sim orchestration cycle; deferred, the transpose logic carries over. - aura-ingest is the external-dependency firewall. data-server transitively pulls chrono + regex + zip (+ their trees) into Cargo.lock; isolating it in this one crate keeps aura-core/std/engine zero-external-dependency and the frozen deploy artifact (C6 replays recorded streams, never re-ingests) clean. cargo test --workspace now needs a populated cargo cache (one Gitea fetch, done). - Scope: boundary + tests only, no aura run CLI arg surface (M1 first; tick and the CLI wiring are follow-ups). Tests: 6 hermetic unit tests on hand-built M1Parsed (normalization, field-wise transpose, purity, close_stream order, empty edges) + one gated integration test (tests/real_bars.rs) that runs the cycle-0007 sample harness over real AAPL.US 2006-08 close bars and asserts finite metrics + two-run bit-identical JSON (C1); it skips cleanly where /mnt/tickdata is absent. On this machine it ran the real path. Workspace gates green: test (86), clippy -D warnings, doc -D warnings. Minor: transient unused-import warnings in the new lib.rs across the additive build steps resolved once load_m1_window consumed the imports; final -D warnings gate clean. |