Brummel
a263f1f51e
Consumer-perspective milestone fieldtest over the seven shipped items: ganged authoring end to end (#61), consumer-blueprint render (#28), param-bearing scaffold starter (#183), outside-project refusal (#218), default stops (#217), gate-free plateau walkforward (#215), --trace honesty (#168) — all working. Red half: aura reproduce DIVERGES 0/N on a dissolved-sweep (campaign-minted) family and PANICS (main.rs:1123, non-empty OOS window) on a WalkForward family — the C18/C1 reproduce promise breaks exactly on campaign-pipeline families, invisible to per-cycle fieldtests. Both routed to debug this run. Friction/gap findings filed as forward issues. Fixture corpus: from-scratch ganged mid-channel op-script + built blueprint, throwaway project, verbatim transcript. Report spec (git-ignored working file): docs/specs/fieldtest-triage-harvest-milestone.md refs #61
147 lines
8.8 KiB
Plaintext
147 lines
8.8 KiB
Plaintext
Milestone fieldtest — Triage harvest 2026-07 (surface honesty & authoring ergonomics)
|
|
Binary under test: ./target/debug/aura, built from HEAD 29bf601 via `cargo build --workspace`.
|
|
Data: real GER40, 2025 window --from 1735689600000 --to 1767225599000.
|
|
Project: throwaway_proj/ scaffolded by `aura new` (cdylib `cargo build`'t once).
|
|
Public interface only: README, authoring-guide, glossary, design INDEX, --help, examples/*.json.
|
|
|
|
================================================================================
|
|
TASK 1 — author a ganged strategy from scratch (#61, #28)
|
|
================================================================================
|
|
|
|
Op-script authored: triage_1_ganged_midchannel.ops.json
|
|
A mid-channel breakout: bias = price - (RollingMax(N)+RollingMin(N))/2, with the
|
|
two Donchian window lengths ganged into ONE public knob `channel_length`.
|
|
mid.factor bound 0.5, bias.scale bound 1.0 → the gang is the sole open axis.
|
|
|
|
First attempt (feed grouped near the sources, referencing dev before it was added):
|
|
$ aura graph build < triage_1_ganged_midchannel.ops.json
|
|
aura: op 3 (feed): unknown node identifier "dev" [exit 1]
|
|
-> CLEAN diagnostic: names the op index (3) AND the missing identifier ("dev").
|
|
My authoring-order mistake; reordered so all `add`s precede the `feed`.
|
|
|
|
Second attempt (all nodes declared first):
|
|
$ aura graph build < triage_1_ganged_midchannel.ops.json > triage_1_ganged_midchannel.bp.json [exit 0]
|
|
|
|
$ aura graph introspect --params triage_1_ganged_midchannel.bp.json
|
|
channel_length:I64 <- ONE axis (two lengths collapsed)
|
|
|
|
$ aura sweep triage_1_ganged_midchannel.bp.json --list-axes
|
|
graph.channel_length:I64 <- ONE wrapped axis
|
|
|
|
blueprint carries the relation:
|
|
"gangs":[{"name":"channel_length","kind":"I64",
|
|
"members":[{"node":0,"pos":0,"name":"length"},{"node":1,"pos":0,"name":"length"}]}]
|
|
|
|
$ aura graph triage_1_ganged_midchannel.bp.json > render.html [exit 0]
|
|
Embedded model carries: "gangs":[{"name":"channel_length","kind":"I64",
|
|
"members":[[0,0,"length"],[1,0,"length"]]}]
|
|
Embedded viewer JS annotates the ganged node:
|
|
const label = txt(n, k) + (gang ? `<font color="#9399b2"> (gang: ${gang.name})</font>` ...
|
|
-> render shows the node labelled "(gang: channel_length)". WORKING.
|
|
|
|
================================================================================
|
|
TASK 2 — project bootstrap + tunable starter (#183, #218)
|
|
================================================================================
|
|
|
|
$ aura new throwaway_proj
|
|
created project `throwaway_proj` (namespace `throwaway_proj`) [exit 0]
|
|
files: Aura.toml, Cargo.toml, CLAUDE.md, .gitignore, src/lib.rs, blueprints/signal.json
|
|
|
|
#183 — scaffolded starter node carries a tunable param (WORKING):
|
|
src/lib.rs: `throwaway_proj::Scale` declares
|
|
params: vec![ParamSpec { name: "factor".into(), kind: ScalarKind::F64 }]
|
|
builder closure |p| Box::new(Scale::new(p[0].f64()))
|
|
doc comment: "One-input scalar gain: emits `input * factor`."
|
|
blueprints/signal.json binds it: {"type":"throwaway_proj::Scale","bound":[{"pos":0,"name":"factor",...,"value":{"F64":1.0}}]}
|
|
-> a copyable, self-documenting parameterized template.
|
|
NOTE: the enumerated public docs (authoring-guide) teach the three DATA artifacts
|
|
(op-script/process/campaign); Rust NODE authoring (the ParamSpec pattern) is taught
|
|
only by the scaffolded node itself + docs/project-layout.md (referenced by the
|
|
project's CLAUDE.md, not in the guide). See spec finding.
|
|
|
|
#218 — dissolved verb OUTSIDE any project refuses cleanly (WORKING):
|
|
cwd = clean empty dir (no Aura.toml up-tree)
|
|
$ aura sweep <ganged.bp.json> --real GER40 --from ... --to ... --axis graph.channel_length=10,20
|
|
aura: sweep needs a project: strategies resolve against the project store and vocabulary
|
|
(no Aura.toml found up from /.../noproj) [exit 1]
|
|
cwd after: still empty — NO runs/ store written. WORKING.
|
|
|
|
================================================================================
|
|
TASK 3 — sweep + walkforward the ganged blueprint end to end (#61, #215, #217)
|
|
================================================================================
|
|
|
|
Inside throwaway_proj/, blueprint copied to blueprints/ganged_midchannel.json.
|
|
|
|
Sweep over the single gang axis, default stops (WORKING):
|
|
$ aura sweep blueprints/ganged_midchannel.json --real GER40 --from ... --to ... \
|
|
--axis graph.channel_length=10,20,40 --name triage_gang_sweep [exit 0]
|
|
3 members; params per member = [graph.channel_length, stop_length=3, stop_k=2.0]
|
|
(default stops applied without --stop-length/--stop-k — #217).
|
|
|
|
Walkforward, --select plateau (bare form the task names) (SURFACE MISMATCH):
|
|
$ aura walkforward ... --select plateau ...
|
|
aura: Usage: ... [--select <argmax|plateau:mean|plateau:worst>] ... [exit 2]
|
|
-> bare `plateau` refused; usage error DOES list the valid forms. friction/spec_gap.
|
|
|
|
Walkforward, --select plateau:mean, NO gate preceding, default stops (#215 WORKING):
|
|
$ aura walkforward blueprints/ganged_midchannel.json --real GER40 --from ... --to ... \
|
|
--axis graph.channel_length=10,20,40 --select plateau:mean --name triage_gang_wf [exit 0]
|
|
ACCEPTED. 9 OOS windows; each `selection` block: "mode":"PlateauMean",
|
|
neighbourhood_score + n_neighbours reported. Stitched OOS verdict + param_stability.
|
|
-> plateau selection without a preceding gate is legal (decision-4 relaxation). WORKING.
|
|
|
|
Families persist + list (WORKING):
|
|
$ aura runs families
|
|
{"family_id":"753e2fd5-0-GER40-w0-r0-s0-0","kind":"Sweep","members":3}
|
|
{"family_id":"de77bc72-0-GER40-w0-r0-s0-0","kind":"Sweep","members":3} (WF internal IS-refit sweep)
|
|
{"family_id":"de77bc72-0-GER40-w0-r0-s1-0","kind":"WalkForward","members":9}
|
|
|
|
reproduce — BUG A (DIVERGED, systematic):
|
|
$ aura reproduce 753e2fd5-0-GER40-w0-r0-s0-0
|
|
753e2fd5-... member graph.channel_length=10 ... reproduced: DIVERGED
|
|
753e2fd5-... member graph.channel_length=20 ... reproduced: DIVERGED
|
|
753e2fd5-... member graph.channel_length=40 ... reproduced: DIVERGED
|
|
reproduced 0/3 members bit-identically [exit 1]
|
|
(stable across two runs; the WF-internal sweep family de77bc72-...-s0-0 also 0/3 DIVERGED)
|
|
README: "aura reproduce <family-id> ... check it is bit-identical (the C18/C1 determinism guarantee)."
|
|
Localization: a FRESH identical sweep reproduces the persisted metrics BIT-FOR-BIT
|
|
fresh expectancy_r: 0.009163633325930131 / 0.009069046610003514 / 0.009568962981603747
|
|
persisted : 0.009163633325930131 / 0.009069046610003514 / 0.009568962981603747
|
|
=> the run is deterministic (C1 holds); `aura reproduce`'s re-derivation path
|
|
diverges from how the dissolved-sweep (campaign-generated) family was persisted.
|
|
|
|
reproduce — BUG B (PANIC on a WalkForward family):
|
|
$ aura reproduce de77bc72-0-GER40-w0-r0-s1-0
|
|
thread 'main' panicked at crates/aura-cli/src/main.rs:1123:39: non-empty OOS window [exit 101]
|
|
(missing-id path by contrast is clean: `aura reproduce no-such-family-id`
|
|
-> "aura: no such family 'no-such-family-id'" exit 1)
|
|
|
|
================================================================================
|
|
TASK 4 — surface honesty spot-checks (#168, #28)
|
|
================================================================================
|
|
|
|
#168 — --trace refuses honestly (WORKING):
|
|
$ aura sweep <bp> --real GER40 --axis graph.channel_length=10 --trace mytrace
|
|
aura: --trace is not yet available on sweep; see #224 [exit 2]
|
|
$ aura walkforward <bp> ... --trace mytrace
|
|
aura: --trace is not yet available on walkforward; see #224 [exit 2]
|
|
-> honest forward-pointer, not a silent no-op. WORKING.
|
|
|
|
#28 — bad blueprint arg to `aura graph` is a usage fault, not a silent sample (WORKING):
|
|
$ aura graph does_not_exist.json
|
|
aura: Usage: aura graph [<blueprint.json>] [build|introspect|register];
|
|
does_not_exist.json is not a readable .json blueprint [exit 2, 0 bytes stdout]
|
|
$ aura graph notabp.txt
|
|
aura: ... notabp.txt is not a readable .json blueprint [exit 2, 0 bytes stdout]
|
|
$ aura graph (no arg)
|
|
-> renders the built-in sample HTML [exit 0, ~1.43MB]
|
|
|
|
#28-adjacent friction — op-script (valid JSON, wrong schema) rendered:
|
|
$ aura graph triage_1_ganged_midchannel.ops.json
|
|
aura: ...ops.json: blueprint document is not valid JSON:
|
|
invalid type: map, expected u32 at line 2 column 2 [exit 2, 0 bytes stdout]
|
|
-> exit 2 correct (no silent sample), BUT the file IS valid JSON; the message
|
|
"not valid JSON" misdescribes a schema mismatch. And the render positional does
|
|
NOT shape-discriminate an op-script array the way sibling introspect verbs do
|
|
(--params/--content-id/--identity-id/register accept EITHER shape; render does not).
|