Brummel/Aura
1
0
Fork 0
Code Issues 31 Pull Requests Actions Packages Projects Releases Wiki Activity

Give measurement its first deflatable metric (tap-to-scalar reduction + null model) #290

New Issue
Closed
opened 2026-07-20 11:47:01 +02:00 by claude · 4 comments
claude commented 2026-07-20 11:47:01 +02:00
Collaborator
Copy Link
Copy Source

Precondition-seeder for the deferred half of #147 (the registry deflation-vocabulary genericity, C28 phase 6). The deferral's condition is a genuine second metric consumer; this records the concrete candidate shape.

A measurement run today produces declared taps only -- MeasurementReport (crates/aura-engine/src/report.rs:119) carries taps: Vec<String> and no scalar metric -- so measurement output can be inspected but not ranked, swept, or deflated. The missing piece is a post-run reduction from a recorded tap series to one scalar per run, paired with its own null model. Candidates:

  • Information coefficient -- corr(signal_t, forward_return_t+h); null model: permutation of the signal against the returns (or analytically N(0, 1/sqrt(n))). The forward-return alignment is an offline join over recorded traces (shift + join_on_ts) after the run -- no in-graph future read, so C2 (causality) is untouched.
  • Directional hit-rate -- the share of correctly-signed predictions; null model: Binomial(n, 0.5).
  • Signal autocorrelation / persistence -- null model: white-noise surrogate.

Any one of these, wired through sweep selection, turns optimize_deflated's hard-coded two-arm null-model branch (crates/aura-registry/src/lib.rs:791 -- R block-bootstrap vs total_pips dispersion floor) into a genuine two-implementor seam; at that point the (metric, null-model) injection of #147 item 2 is demand-driven instead of speculative, and the remaining deferral lifts.

Precondition-seeder for the deferred half of #147 (the registry deflation-vocabulary genericity, C28 phase 6). The deferral's condition is a genuine second metric consumer; this records the concrete candidate shape. A measurement run today produces declared taps only -- `MeasurementReport` (`crates/aura-engine/src/report.rs:119`) carries `taps: Vec<String>` and no scalar metric -- so measurement output can be inspected but not ranked, swept, or deflated. The missing piece is a post-run reduction from a recorded tap series to one scalar per run, paired with its own null model. Candidates: - **Information coefficient** -- `corr(signal_t, forward_return_t+h)`; null model: permutation of the signal against the returns (or analytically `N(0, 1/sqrt(n))`). The forward-return alignment is an offline join over recorded traces (shift + `join_on_ts`) after the run -- no in-graph future read, so C2 (causality) is untouched. - **Directional hit-rate** -- the share of correctly-signed predictions; null model: `Binomial(n, 0.5)`. - **Signal autocorrelation / persistence** -- null model: white-noise surrogate. Any one of these, wired through sweep selection, turns `optimize_deflated`'s hard-coded two-arm null-model branch (`crates/aura-registry/src/lib.rs:791` -- R block-bootstrap vs `total_pips` dispersion floor) into a genuine two-implementor seam; at that point the (metric, null-model) injection of #147 item 2 is demand-driven instead of speculative, and the remaining deferral lifts.
claude added the idea label 2026-07-20 11:47:01 +02:00
claude self-assigned this 2026-07-20 14:38:04 +02:00
claude commented 2026-07-20 14:38:53 +02:00
Author
Collaborator
Copy Link
Copy Source

Un-deferral proceeding; #290 sequenced before #147 item 2 (2026-07-20)

The deferral of #147 item 2 (the registry deflation-vocabulary genericity) rested on a single unmet precondition: a genuine second metric consumer, so the abstraction is demand-driven rather than a one-implementor speculation. This run builds that consumer here (#290), then extracts the abstraction against two real implementors.

User direction (2026-07-20): lift the twice-ratified deferral of #147 item 2 (2026-06-27 in the #147 body; re-ratified in the 2026-07-02 comment on #147) by building measurement its first deflatable metric first. The call to un-defer is the user's, taken in this run.

Sequencing decision (derived). Two orderings were open:

  • (A) design #290 and the #147-item-2 abstraction in one spec;
  • (B) ship #290 as its own cycle first, then extract #147 item 2 against two real implementors.

Chosen: (B). Basis (derived): the project's demand-driven-abstraction principle — designing the abstraction against one real plus one still-being-built implementor is the speculative posture the deferral itself was written to avoid. Both cycles run in the same session; #147 item 2 becomes autonomously-doable the moment #290's second implementor lands.

Metric selection (derived lean, finalised in spec production). Of the three candidates in the body: signal autocorrelation measures persistence, not predictive skill (wrong target); directional hit-rate discards conviction magnitude (a coarse special case). The information coefficient (corr(signal_t, forward_return_t+h)) uses sign and magnitude, is the standard signal-quality yardstick, and carries a permutation null model genuinely distinct from the R block-bootstrap. Its forward-return alignment is an offline join over recorded traces (no in-graph future read — C2 untouched). The C7 relationship ("signal quality is measured in R") is complementary, not contradictory: R scores the stopped strategy on the execution layer; the information coefficient scores the raw pre-execution signal on the measurement layer. This lean is validated against C2/C7 by the spec grounding-check.

Registry surfaces the item-2 extraction touches (verified against the current tree): optimize_deflated (crates/aura-registry/src/lib.rs:784, the two-arm null-model branch), the string-keyed Metric vocabulary (:502) and metric_cmp (:564) / rank_by (:578). Measurement current output carries tap names only (MeasurementReport, crates/aura-engine/src/report.rs:119).

Design ready for spec production.

## Un-deferral proceeding; #290 sequenced before #147 item 2 (2026-07-20) The deferral of #147 item 2 (the registry deflation-vocabulary genericity) rested on a single unmet precondition: a genuine second metric consumer, so the abstraction is demand-driven rather than a one-implementor speculation. This run builds that consumer here (#290), then extracts the abstraction against two real implementors. **User direction (2026-07-20):** lift the twice-ratified deferral of #147 item 2 (2026-06-27 in the #147 body; re-ratified in the 2026-07-02 comment on #147) by building measurement its first deflatable metric first. The call to un-defer is the user's, taken in this run. **Sequencing decision (derived).** Two orderings were open: - (A) design #290 and the #147-item-2 abstraction in one spec; - (B) ship #290 as its own cycle first, then extract #147 item 2 against two real implementors. Chosen: **(B)**. Basis (derived): the project's demand-driven-abstraction principle — designing the abstraction against one real plus one still-being-built implementor is the speculative posture the deferral itself was written to avoid. Both cycles run in the same session; #147 item 2 becomes autonomously-doable the moment #290's second implementor lands. **Metric selection (derived lean, finalised in spec production).** Of the three candidates in the body: signal autocorrelation measures persistence, not predictive skill (wrong target); directional hit-rate discards conviction magnitude (a coarse special case). The **information coefficient** (corr(signal_t, forward_return_t+h)) uses sign and magnitude, is the standard signal-quality yardstick, and carries a permutation null model genuinely distinct from the R block-bootstrap. Its forward-return alignment is an offline join over recorded traces (no in-graph future read — C2 untouched). The C7 relationship ("signal quality is measured in R") is complementary, not contradictory: R scores the stopped strategy on the execution layer; the information coefficient scores the raw pre-execution signal on the measurement layer. This lean is validated against C2/C7 by the spec grounding-check. Registry surfaces the item-2 extraction touches (verified against the current tree): `optimize_deflated` (crates/aura-registry/src/lib.rs:784, the two-arm null-model branch), the string-keyed `Metric` vocabulary (:502) and `metric_cmp` (:564) / `rank_by` (:578). Measurement current output carries tap names only (`MeasurementReport`, crates/aura-engine/src/report.rs:119). Design ready for spec production.
claude commented 2026-07-20 14:50:43 +02:00
Author
Collaborator
Copy Link
Copy Source

Design refinement: #290 delivers a standalone post-run reduction; the registry wiring is #147 item 2 (2026-07-20)

Recon of the current tree (drift-checked this run) refines the shape implied by the earlier sequencing comment, which framed #290 as adding an IC arm inside optimize_deflateds null-model branch. The tree shows that is not cleanly separable from #147 item 2: optimize_deflated / rank_by / metric_cmp (crates/aura-registry/src/lib.rs:784/578/564) read the concrete RunReport<RunMetrics> and branch on the trading Metric enum (is_r_metric, :603). Giving them an IC arm requires the metric payload to carry IC — which is exactly the domain-supplied-score + injected-null-model abstraction of #147 item 2. Adding a hard-coded third arm now only to abstract it away next cycle is the churn the sequencing avoids.

Refined cut.

  • #290 (this cycle) delivers IC as a standalone post-run reduction over recorded traces. A measurement blueprint declares a tap on the signal and a tap on the price/close; a new reduction reads both persisted trace columns (runs/traces/<name>/<tap>.json, a ColumnarTrace of ts[] + columns[]), computes the forward-return-aligned information coefficient corr(signal_t, forward_return_{t+h}) and its permutation-null significance, and reports the scalar. The generic pieces — Pearson correlation and a Fisher-Yates permutation over SplitMix64 — land in aura-analysis (the domain-free statistics foundation, beside resample_block); the IC semantics (forward-return horizon h, the offline join) live in the reduction command. The run path (run_measurement, crates/aura-cli/src/main.rs:1852) is untouched, so existing measurement and strategy run output stays byte-identical (C18). The forward-return read close_{t+h} is over a completed run recorded trace, not an in-graph node: C2 governs in-graph node access (a node sees only the past) and does not reach a post-run reduction.
  • #147 item 2 (next cycle) abstracts optimize_deflated / rank_by / metric_cmp over an injected (score, null-model), then wires IC through sweep selection (cross-trials deflation). This is now demand-driven: two structurally distinct null-model computations exist in aura-analysis — the moving-block bootstrap behind R (resample_block) and the permutation behind IC — so the injected-null-model seam is validated against two real implementors, not one. It realizes the ledger architecture stated in C28 ("a named metric vocabulary supplied by measurement and backtest instead of baked in R-only ... the statistics injected from aura-analysis").

Basis (derived): layer-honesty (the reduction is process-column work over run artifacts, C28), byte-identity safety (C18), and avoiding add-arm-then-abstract-it churn. The two-consumer payoff that lifts #147 item 2 deferral is the permutation null existing as a real, tested computation.

Sub-forks resolved (derived). Metric = Pearson IC. Null = one-sided permutation, overfit_probability = (count(null_ic >= raw_ic) + 1)/(n_perm + 1) (mirrors the R-arm formula at lib.rs:807, higher-is-better). Forward-return horizon default h = 1 bar. The permutation seed is recorded in the output for reproducibility (C18 forbids results not reproducible from a recorded manifest).

Design ready for spec production.

## Design refinement: #290 delivers a standalone post-run reduction; the registry wiring is #147 item 2 (2026-07-20) Recon of the current tree (drift-checked this run) refines the shape implied by the earlier sequencing comment, which framed #290 as adding an IC arm inside `optimize_deflated`s null-model branch. The tree shows that is not cleanly separable from #147 item 2: `optimize_deflated` / `rank_by` / `metric_cmp` (crates/aura-registry/src/lib.rs:784/578/564) read the concrete `RunReport<RunMetrics>` and branch on the trading `Metric` enum (`is_r_metric`, :603). Giving them an IC arm requires the metric payload to carry IC — which is exactly the domain-supplied-score + injected-null-model abstraction of #147 item 2. Adding a hard-coded third arm now only to abstract it away next cycle is the churn the sequencing avoids. **Refined cut.** - **#290 (this cycle)** delivers IC as a standalone post-run reduction over recorded traces. A measurement blueprint declares a tap on the signal and a tap on the price/close; a new reduction reads both persisted trace columns (`runs/traces/<name>/<tap>.json`, a `ColumnarTrace` of `ts[]` + `columns[]`), computes the forward-return-aligned information coefficient `corr(signal_t, forward_return_{t+h})` and its permutation-null significance, and reports the scalar. The generic pieces — Pearson correlation and a Fisher-Yates permutation over `SplitMix64` — land in `aura-analysis` (the domain-free statistics foundation, beside `resample_block`); the IC semantics (forward-return horizon `h`, the offline join) live in the reduction command. The run path (`run_measurement`, crates/aura-cli/src/main.rs:1852) is untouched, so existing measurement and strategy run output stays byte-identical (C18). The forward-return read `close_{t+h}` is over a completed run recorded trace, not an in-graph node: C2 governs in-graph node access (a node sees only the past) and does not reach a post-run reduction. - **#147 item 2 (next cycle)** abstracts `optimize_deflated` / `rank_by` / `metric_cmp` over an injected (score, null-model), then wires IC through sweep selection (cross-trials deflation). This is now demand-driven: two structurally distinct null-model computations exist in `aura-analysis` — the moving-block bootstrap behind R (`resample_block`) and the permutation behind IC — so the injected-null-model seam is validated against two real implementors, not one. It realizes the ledger architecture stated in C28 ("a named metric vocabulary supplied by measurement and backtest instead of baked in R-only ... the statistics injected from aura-analysis"). **Basis (derived):** layer-honesty (the reduction is process-column work over run artifacts, C28), byte-identity safety (C18), and avoiding add-arm-then-abstract-it churn. The two-consumer payoff that lifts #147 item 2 deferral is the permutation null existing as a real, tested computation. **Sub-forks resolved (derived).** Metric = Pearson IC. Null = one-sided permutation, `overfit_probability = (count(null_ic >= raw_ic) + 1)/(n_perm + 1)` (mirrors the R-arm formula at lib.rs:807, higher-is-better). Forward-return horizon default h = 1 bar. The permutation seed is recorded in the output for reproducibility (C18 forbids results not reproducible from a recorded manifest). Design ready for spec production.
claude commented 2026-07-20 14:59:24 +02:00
Author
Collaborator
Copy Link
Copy Source

Spec auto-signed on a grounding-check PASS (2026-07-20)

The measurement-IC spec (the standalone post-run Information Coefficient reduction refined in the comment above) was produced and auto-signed under /boss. The signature is the grounding-check PASS: an independent fresh-context agent verdict that every load-bearing assumption the spec makes about the current tree is ratified by a named, currently-green test —

  • MeasurementReport is { manifest, taps } with no metric field, and aura run on a tapped, no-bias blueprint dispatches to the measurement path that persists the tap trace;
  • taps persist as ColumnarTrace under runs/traces/<run>/<tap>.json with a trace-store read round-trip;
  • the aura-analysis statistics kernel (SplitMix64, resample_block, MetricStats, quantile) is pub and deterministic-by-seed — the base the new pearson_corr / permute extend;
  • the deflation null computes (count(null >= raw) + 1)/(n + 1), the formula the IC permutation null mirrors;
  • the aura run output golden is byte-identical (the run path is untouched this cycle).

No human signed; a user veto reverts it forward (the spec is a git-ignored working file, never committed). The design is settled and ready for plan production.

## Spec auto-signed on a grounding-check PASS (2026-07-20) The measurement-IC spec (the standalone post-run Information Coefficient reduction refined in the comment above) was produced and auto-signed under /boss. The signature is the grounding-check PASS: an independent fresh-context agent verdict that every load-bearing assumption the spec makes about the current tree is ratified by a named, currently-green test — - `MeasurementReport` is `{ manifest, taps }` with no metric field, and `aura run` on a tapped, no-`bias` blueprint dispatches to the measurement path that persists the tap trace; - taps persist as `ColumnarTrace` under `runs/traces/<run>/<tap>.json` with a trace-store read round-trip; - the `aura-analysis` statistics kernel (`SplitMix64`, `resample_block`, `MetricStats`, `quantile`) is `pub` and deterministic-by-seed — the base the new `pearson_corr` / `permute` extend; - the deflation null computes `(count(null >= raw) + 1)/(n + 1)`, the formula the IC permutation null mirrors; - the `aura run` output golden is byte-identical (the run path is untouched this cycle). No human signed; a user veto reverts it forward (the spec is a git-ignored working file, never committed). The design is settled and ready for plan production.
claude commented 2026-07-20 16:43:54 +02:00
Author
Collaborator
Copy Link
Copy Source

Cycle shipped on the run branch; awaiting the user-reserved merge (2026-07-20)

The Information Coefficient reduction is implemented and committed on branch worktree-issue-290-measure-metric (commit a9d36dd). aura measure ic <run> --signal <tap> --price <tap> [--horizon] [--permutations] [--seed] reads a persisted run two tap traces, computes corr(signal_t, forward_return_{t+h}) and a one-sided permutation-null overfit probability; pearson_corr + permute landed in aura-analysis (the domain-free foundation), the IC semantics + the verb in aura-cli (the shell). The run path (run_measurement), MeasurementReport, the trace store, and the registry vocabulary are untouched.

Verified independently by the orchestrator: cargo test --workspace green — the new aura-analysis unit tests, the aura-cli IC unit tests (engineered signal significant / varying-uncorrelated not significant / constant + degenerate floor / exact-ts alignment / determinism), and the measure_ic E2E (well-formedness, determinism, unknown-run and missing-tap error paths); cargo clippy --workspace --all-targets -D warnings clean; the C18 byte-identity golden (run_prints_json_and_exits_zero) unchanged. The spec was auto-signed on a grounding-check PASS (no human signature). Cycle-close audit: ratified against C2/C10/C18/C28, with two medium debts noted — measurement-domain logic accreting in the shell (the ledger-recorded "no measurement crate yet" debt, C28 Status), and an undefended duplicate-timestamp assumption in the offline join (filed as #294).

This lands the second, structurally distinct null-model computation (a permutation null, beside the R moving-block bootstrap), meeting the item-2 deferral precondition on #147. The closes #290 marker fires when the branch merges.

## Cycle shipped on the run branch; awaiting the user-reserved merge (2026-07-20) The Information Coefficient reduction is implemented and committed on branch `worktree-issue-290-measure-metric` (commit a9d36dd). `aura measure ic <run> --signal <tap> --price <tap> [--horizon] [--permutations] [--seed]` reads a persisted run two tap traces, computes `corr(signal_t, forward_return_{t+h})` and a one-sided permutation-null overfit probability; `pearson_corr` + `permute` landed in aura-analysis (the domain-free foundation), the IC semantics + the verb in aura-cli (the shell). The run path (`run_measurement`), `MeasurementReport`, the trace store, and the registry vocabulary are untouched. Verified independently by the orchestrator: `cargo test --workspace` green — the new aura-analysis unit tests, the aura-cli IC unit tests (engineered signal significant / varying-uncorrelated not significant / constant + degenerate floor / exact-ts alignment / determinism), and the `measure_ic` E2E (well-formedness, determinism, unknown-run and missing-tap error paths); `cargo clippy --workspace --all-targets -D warnings` clean; the C18 byte-identity golden (`run_prints_json_and_exits_zero`) unchanged. The spec was auto-signed on a grounding-check PASS (no human signature). Cycle-close audit: ratified against C2/C10/C18/C28, with two medium debts noted — measurement-domain logic accreting in the shell (the ledger-recorded "no measurement crate yet" debt, C28 Status), and an undefended duplicate-timestamp assumption in the offline join (filed as #294). This lands the second, structurally distinct null-model computation (a permutation null, beside the R moving-block bootstrap), meeting the item-2 deferral precondition on #147. The `closes #290` marker fires when the branch merges.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
BLOCKER
Core functionality blocked; preempts other selection
 bug
Observable misbehaviour
 feature
Substantial extension; no full spec required
 idea
Droppable; no commitment
No Label idea
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
claude (Claude) Brummel
No Assignees claude
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Brummel/Aura#290
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?