fieldtest: triage-harvest milestone evidence (8 working, 2 bugs, 2 friction, 1 gap)
Consumer-perspective milestone fieldtest over the seven shipped items: ganged authoring end to end (#61), consumer-blueprint render (#28), param-bearing scaffold starter (#183), outside-project refusal (#218), default stops (#217), gate-free plateau walkforward (#215), --trace honesty (#168) — all working. Red half: aura reproduce DIVERGES 0/N on a dissolved-sweep (campaign-minted) family and PANICS (main.rs:1123, non-empty OOS window) on a WalkForward family — the C18/C1 reproduce promise breaks exactly on campaign-pipeline families, invisible to per-cycle fieldtests. Both routed to debug this run. Friction/gap findings filed as forward issues. Fixture corpus: from-scratch ganged mid-channel op-script + built blueprint, throwaway project, verbatim transcript. Report spec (git-ignored working file): docs/specs/fieldtest-triage-harvest-milestone.md refs #61
This commit is contained in:
@@ -0,0 +1,146 @@
|
||||
Milestone fieldtest — Triage harvest 2026-07 (surface honesty & authoring ergonomics)
|
||||
Binary under test: ./target/debug/aura, built from HEAD 29bf601 via `cargo build --workspace`.
|
||||
Data: real GER40, 2025 window --from 1735689600000 --to 1767225599000.
|
||||
Project: throwaway_proj/ scaffolded by `aura new` (cdylib `cargo build`'t once).
|
||||
Public interface only: README, authoring-guide, glossary, design INDEX, --help, examples/*.json.
|
||||
|
||||
================================================================================
|
||||
TASK 1 — author a ganged strategy from scratch (#61, #28)
|
||||
================================================================================
|
||||
|
||||
Op-script authored: triage_1_ganged_midchannel.ops.json
|
||||
A mid-channel breakout: bias = price - (RollingMax(N)+RollingMin(N))/2, with the
|
||||
two Donchian window lengths ganged into ONE public knob `channel_length`.
|
||||
mid.factor bound 0.5, bias.scale bound 1.0 → the gang is the sole open axis.
|
||||
|
||||
First attempt (feed grouped near the sources, referencing dev before it was added):
|
||||
$ aura graph build < triage_1_ganged_midchannel.ops.json
|
||||
aura: op 3 (feed): unknown node identifier "dev" [exit 1]
|
||||
-> CLEAN diagnostic: names the op index (3) AND the missing identifier ("dev").
|
||||
My authoring-order mistake; reordered so all `add`s precede the `feed`.
|
||||
|
||||
Second attempt (all nodes declared first):
|
||||
$ aura graph build < triage_1_ganged_midchannel.ops.json > triage_1_ganged_midchannel.bp.json [exit 0]
|
||||
|
||||
$ aura graph introspect --params triage_1_ganged_midchannel.bp.json
|
||||
channel_length:I64 <- ONE axis (two lengths collapsed)
|
||||
|
||||
$ aura sweep triage_1_ganged_midchannel.bp.json --list-axes
|
||||
graph.channel_length:I64 <- ONE wrapped axis
|
||||
|
||||
blueprint carries the relation:
|
||||
"gangs":[{"name":"channel_length","kind":"I64",
|
||||
"members":[{"node":0,"pos":0,"name":"length"},{"node":1,"pos":0,"name":"length"}]}]
|
||||
|
||||
$ aura graph triage_1_ganged_midchannel.bp.json > render.html [exit 0]
|
||||
Embedded model carries: "gangs":[{"name":"channel_length","kind":"I64",
|
||||
"members":[[0,0,"length"],[1,0,"length"]]}]
|
||||
Embedded viewer JS annotates the ganged node:
|
||||
const label = txt(n, k) + (gang ? `<font color="#9399b2"> (gang: ${gang.name})</font>` ...
|
||||
-> render shows the node labelled "(gang: channel_length)". WORKING.
|
||||
|
||||
================================================================================
|
||||
TASK 2 — project bootstrap + tunable starter (#183, #218)
|
||||
================================================================================
|
||||
|
||||
$ aura new throwaway_proj
|
||||
created project `throwaway_proj` (namespace `throwaway_proj`) [exit 0]
|
||||
files: Aura.toml, Cargo.toml, CLAUDE.md, .gitignore, src/lib.rs, blueprints/signal.json
|
||||
|
||||
#183 — scaffolded starter node carries a tunable param (WORKING):
|
||||
src/lib.rs: `throwaway_proj::Scale` declares
|
||||
params: vec![ParamSpec { name: "factor".into(), kind: ScalarKind::F64 }]
|
||||
builder closure |p| Box::new(Scale::new(p[0].f64()))
|
||||
doc comment: "One-input scalar gain: emits `input * factor`."
|
||||
blueprints/signal.json binds it: {"type":"throwaway_proj::Scale","bound":[{"pos":0,"name":"factor",...,"value":{"F64":1.0}}]}
|
||||
-> a copyable, self-documenting parameterized template.
|
||||
NOTE: the enumerated public docs (authoring-guide) teach the three DATA artifacts
|
||||
(op-script/process/campaign); Rust NODE authoring (the ParamSpec pattern) is taught
|
||||
only by the scaffolded node itself + docs/project-layout.md (referenced by the
|
||||
project's CLAUDE.md, not in the guide). See spec finding.
|
||||
|
||||
#218 — dissolved verb OUTSIDE any project refuses cleanly (WORKING):
|
||||
cwd = clean empty dir (no Aura.toml up-tree)
|
||||
$ aura sweep <ganged.bp.json> --real GER40 --from ... --to ... --axis graph.channel_length=10,20
|
||||
aura: sweep needs a project: strategies resolve against the project store and vocabulary
|
||||
(no Aura.toml found up from /.../noproj) [exit 1]
|
||||
cwd after: still empty — NO runs/ store written. WORKING.
|
||||
|
||||
================================================================================
|
||||
TASK 3 — sweep + walkforward the ganged blueprint end to end (#61, #215, #217)
|
||||
================================================================================
|
||||
|
||||
Inside throwaway_proj/, blueprint copied to blueprints/ganged_midchannel.json.
|
||||
|
||||
Sweep over the single gang axis, default stops (WORKING):
|
||||
$ aura sweep blueprints/ganged_midchannel.json --real GER40 --from ... --to ... \
|
||||
--axis graph.channel_length=10,20,40 --name triage_gang_sweep [exit 0]
|
||||
3 members; params per member = [graph.channel_length, stop_length=3, stop_k=2.0]
|
||||
(default stops applied without --stop-length/--stop-k — #217).
|
||||
|
||||
Walkforward, --select plateau (bare form the task names) (SURFACE MISMATCH):
|
||||
$ aura walkforward ... --select plateau ...
|
||||
aura: Usage: ... [--select <argmax|plateau:mean|plateau:worst>] ... [exit 2]
|
||||
-> bare `plateau` refused; usage error DOES list the valid forms. friction/spec_gap.
|
||||
|
||||
Walkforward, --select plateau:mean, NO gate preceding, default stops (#215 WORKING):
|
||||
$ aura walkforward blueprints/ganged_midchannel.json --real GER40 --from ... --to ... \
|
||||
--axis graph.channel_length=10,20,40 --select plateau:mean --name triage_gang_wf [exit 0]
|
||||
ACCEPTED. 9 OOS windows; each `selection` block: "mode":"PlateauMean",
|
||||
neighbourhood_score + n_neighbours reported. Stitched OOS verdict + param_stability.
|
||||
-> plateau selection without a preceding gate is legal (decision-4 relaxation). WORKING.
|
||||
|
||||
Families persist + list (WORKING):
|
||||
$ aura runs families
|
||||
{"family_id":"753e2fd5-0-GER40-w0-r0-s0-0","kind":"Sweep","members":3}
|
||||
{"family_id":"de77bc72-0-GER40-w0-r0-s0-0","kind":"Sweep","members":3} (WF internal IS-refit sweep)
|
||||
{"family_id":"de77bc72-0-GER40-w0-r0-s1-0","kind":"WalkForward","members":9}
|
||||
|
||||
reproduce — BUG A (DIVERGED, systematic):
|
||||
$ aura reproduce 753e2fd5-0-GER40-w0-r0-s0-0
|
||||
753e2fd5-... member graph.channel_length=10 ... reproduced: DIVERGED
|
||||
753e2fd5-... member graph.channel_length=20 ... reproduced: DIVERGED
|
||||
753e2fd5-... member graph.channel_length=40 ... reproduced: DIVERGED
|
||||
reproduced 0/3 members bit-identically [exit 1]
|
||||
(stable across two runs; the WF-internal sweep family de77bc72-...-s0-0 also 0/3 DIVERGED)
|
||||
README: "aura reproduce <family-id> ... check it is bit-identical (the C18/C1 determinism guarantee)."
|
||||
Localization: a FRESH identical sweep reproduces the persisted metrics BIT-FOR-BIT
|
||||
fresh expectancy_r: 0.009163633325930131 / 0.009069046610003514 / 0.009568962981603747
|
||||
persisted : 0.009163633325930131 / 0.009069046610003514 / 0.009568962981603747
|
||||
=> the run is deterministic (C1 holds); `aura reproduce`'s re-derivation path
|
||||
diverges from how the dissolved-sweep (campaign-generated) family was persisted.
|
||||
|
||||
reproduce — BUG B (PANIC on a WalkForward family):
|
||||
$ aura reproduce de77bc72-0-GER40-w0-r0-s1-0
|
||||
thread 'main' panicked at crates/aura-cli/src/main.rs:1123:39: non-empty OOS window [exit 101]
|
||||
(missing-id path by contrast is clean: `aura reproduce no-such-family-id`
|
||||
-> "aura: no such family 'no-such-family-id'" exit 1)
|
||||
|
||||
================================================================================
|
||||
TASK 4 — surface honesty spot-checks (#168, #28)
|
||||
================================================================================
|
||||
|
||||
#168 — --trace refuses honestly (WORKING):
|
||||
$ aura sweep <bp> --real GER40 --axis graph.channel_length=10 --trace mytrace
|
||||
aura: --trace is not yet available on sweep; see #224 [exit 2]
|
||||
$ aura walkforward <bp> ... --trace mytrace
|
||||
aura: --trace is not yet available on walkforward; see #224 [exit 2]
|
||||
-> honest forward-pointer, not a silent no-op. WORKING.
|
||||
|
||||
#28 — bad blueprint arg to `aura graph` is a usage fault, not a silent sample (WORKING):
|
||||
$ aura graph does_not_exist.json
|
||||
aura: Usage: aura graph [<blueprint.json>] [build|introspect|register];
|
||||
does_not_exist.json is not a readable .json blueprint [exit 2, 0 bytes stdout]
|
||||
$ aura graph notabp.txt
|
||||
aura: ... notabp.txt is not a readable .json blueprint [exit 2, 0 bytes stdout]
|
||||
$ aura graph (no arg)
|
||||
-> renders the built-in sample HTML [exit 0, ~1.43MB]
|
||||
|
||||
#28-adjacent friction — op-script (valid JSON, wrong schema) rendered:
|
||||
$ aura graph triage_1_ganged_midchannel.ops.json
|
||||
aura: ...ops.json: blueprint document is not valid JSON:
|
||||
invalid type: map, expected u32 at line 2 column 2 [exit 2, 0 bytes stdout]
|
||||
-> exit 2 correct (no silent sample), BUT the file IS valid JSON; the message
|
||||
"not valid JSON" misdescribes a schema mismatch. And the render positional does
|
||||
NOT shape-discriminate an op-script array the way sibling introspect verbs do
|
||||
(--params/--content-id/--identity-id/register accept EITHER shape; render does not).
|
||||
@@ -0,0 +1,3 @@
|
||||
/target
|
||||
Cargo.lock
|
||||
/runs
|
||||
@@ -0,0 +1,4 @@
|
||||
# Static project context only (C17); paths only.
|
||||
[paths]
|
||||
runs = "runs"
|
||||
# data = "/path/to/archive" # the recorded-data root; defaults to the built-in path
|
||||
@@ -0,0 +1,10 @@
|
||||
# throwaway_proj — an aura research project
|
||||
|
||||
This crate is an aura project: a cdylib of node/strategy blueprints the
|
||||
`aura` host loads during research (see the engine's docs/project-layout.md).
|
||||
|
||||
- Build: `cargo build` (the next `aura` invocation loads the fresh dylib)
|
||||
- Run: `aura run blueprints/signal.json` (from anywhere inside this dir)
|
||||
- Nodes live in `src/`; each is registered in `vocabulary()`/`type_ids()`
|
||||
under the `throwaway_proj::` namespace prefix.
|
||||
- Topology is data (`blueprints/*.json`), node logic is Rust.
|
||||
@@ -0,0 +1,14 @@
|
||||
[package]
|
||||
name = "throwaway_proj"
|
||||
version = "0.1.0"
|
||||
edition = "2024"
|
||||
publish = false
|
||||
|
||||
[lib]
|
||||
crate-type = ["cdylib"]
|
||||
|
||||
[dependencies]
|
||||
aura-core = { path = "/home/brummel/dev/aura/crates/aura-core" }
|
||||
|
||||
# Standalone workspace root: never a member of an enclosing workspace.
|
||||
[workspace]
|
||||
@@ -0,0 +1 @@
|
||||
{"format_version":1,"blueprint":{"name":"graph","nodes":[{"primitive":{"type":"RollingMax","name":"channel_hi"}},{"primitive":{"type":"RollingMin","name":"channel_lo"}},{"primitive":{"type":"Add","name":"sum"}},{"primitive":{"type":"Scale","name":"mid","bound":[{"pos":0,"name":"factor","kind":"F64","value":{"F64":0.5}}]}},{"primitive":{"type":"Sub","name":"dev"}},{"primitive":{"type":"Bias","name":"bias","bound":[{"pos":0,"name":"scale","kind":"F64","value":{"F64":1.0}}]}}],"edges":[{"from":0,"to":2,"slot":0,"from_field":0},{"from":1,"to":2,"slot":1,"from_field":0},{"from":2,"to":3,"slot":0,"from_field":0},{"from":3,"to":4,"slot":1,"from_field":0},{"from":4,"to":5,"slot":0,"from_field":0}],"input_roles":[{"name":"price","targets":[{"node":0,"slot":0},{"node":1,"slot":0},{"node":4,"slot":0}],"source":"F64"}],"output":[{"node":5,"field":0,"name":"bias"}],"gangs":[{"name":"channel_length","kind":"I64","members":[{"node":0,"pos":0,"name":"length"},{"node":1,"pos":0,"name":"length"}]}]}}
|
||||
@@ -0,0 +1 @@
|
||||
{"format_version":1,"blueprint":{"name":"throwaway_proj_signal","nodes":[{"primitive":{"type":"SMA","name":"fast","bound":[{"pos":0,"name":"length","kind":"I64","value":{"I64":2}}]}},{"primitive":{"type":"SMA","name":"slow","bound":[{"pos":0,"name":"length","kind":"I64","value":{"I64":4}}]}},{"primitive":{"type":"Sub"}},{"primitive":{"type":"Bias","name":"bias","bound":[{"pos":0,"name":"scale","kind":"F64","value":{"F64":0.5}}]}},{"primitive":{"type":"throwaway_proj::Scale","bound":[{"pos":0,"name":"factor","kind":"F64","value":{"F64":1.0}}]}}],"edges":[{"from":0,"to":2,"slot":0,"from_field":0},{"from":1,"to":2,"slot":1,"from_field":0},{"from":2,"to":3,"slot":0,"from_field":0},{"from":3,"to":4,"slot":0,"from_field":0}],"input_roles":[{"name":"price","targets":[{"node":0,"slot":0},{"node":1,"slot":0}],"source":"F64"}],"output":[{"node":4,"field":0,"name":"bias"}]}}
|
||||
@@ -0,0 +1,72 @@
|
||||
//! throwaway_proj — an aura research project: node logic + the exported vocabulary.
|
||||
//!
|
||||
//! Every node type this crate provides is registered in `vocabulary()` /
|
||||
//! `type_ids()` under the `throwaway_proj::` namespace prefix; the `aura` host loads
|
||||
//! this cdylib per invocation and merges it with the std vocabulary.
|
||||
|
||||
use aura_core::{
|
||||
Cell, Ctx, FieldSpec, Firing, Node, NodeSchema, ParamSpec, PortSpec, PrimitiveBuilder,
|
||||
ScalarKind,
|
||||
};
|
||||
|
||||
/// One-input scalar gain: emits `input * factor`. Emits `None` until its
|
||||
/// input has a value (warm-up filter, C8).
|
||||
pub struct Scale {
|
||||
factor: f64,
|
||||
out: [Cell; 1],
|
||||
}
|
||||
|
||||
impl Scale {
|
||||
pub fn new(factor: f64) -> Self {
|
||||
Self { factor, out: [Cell::from_f64(0.0)] }
|
||||
}
|
||||
pub fn builder() -> PrimitiveBuilder {
|
||||
PrimitiveBuilder::new(
|
||||
"throwaway_proj::Scale",
|
||||
NodeSchema {
|
||||
inputs: vec![PortSpec {
|
||||
kind: ScalarKind::F64,
|
||||
firing: Firing::Any,
|
||||
name: "value".into(),
|
||||
}],
|
||||
output: vec![FieldSpec { name: "value".into(), kind: ScalarKind::F64 }],
|
||||
params: vec![ParamSpec { name: "factor".into(), kind: ScalarKind::F64 }],
|
||||
},
|
||||
|p| Box::new(Scale::new(p[0].f64())),
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
impl Node for Scale {
|
||||
fn lookbacks(&self) -> Vec<usize> {
|
||||
vec![1]
|
||||
}
|
||||
fn eval(&mut self, ctx: Ctx<'_>) -> Option<&[Cell]> {
|
||||
let w = ctx.f64_in(0);
|
||||
if w.is_empty() {
|
||||
return None;
|
||||
}
|
||||
self.out[0] = Cell::from_f64(w[0] * self.factor);
|
||||
Some(&self.out)
|
||||
}
|
||||
fn label(&self) -> String {
|
||||
format!("throwaway_proj::Scale({})", self.factor)
|
||||
}
|
||||
}
|
||||
|
||||
fn vocabulary(type_id: &str) -> Option<PrimitiveBuilder> {
|
||||
match type_id {
|
||||
"throwaway_proj::Scale" => Some(Scale::builder()),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
fn type_ids() -> &'static [&'static str] {
|
||||
&["throwaway_proj::Scale"]
|
||||
}
|
||||
|
||||
aura_core::aura_project! {
|
||||
namespace: "throwaway_proj",
|
||||
vocabulary: vocabulary,
|
||||
type_ids: type_ids,
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
{"format_version":1,"blueprint":{"name":"graph","nodes":[{"primitive":{"type":"RollingMax","name":"channel_hi"}},{"primitive":{"type":"RollingMin","name":"channel_lo"}},{"primitive":{"type":"Add","name":"sum"}},{"primitive":{"type":"Scale","name":"mid","bound":[{"pos":0,"name":"factor","kind":"F64","value":{"F64":0.5}}]}},{"primitive":{"type":"Sub","name":"dev"}},{"primitive":{"type":"Bias","name":"bias","bound":[{"pos":0,"name":"scale","kind":"F64","value":{"F64":1.0}}]}}],"edges":[{"from":0,"to":2,"slot":0,"from_field":0},{"from":1,"to":2,"slot":1,"from_field":0},{"from":2,"to":3,"slot":0,"from_field":0},{"from":3,"to":4,"slot":1,"from_field":0},{"from":4,"to":5,"slot":0,"from_field":0}],"input_roles":[{"name":"price","targets":[{"node":0,"slot":0},{"node":1,"slot":0},{"node":4,"slot":0}],"source":"F64"}],"output":[{"node":5,"field":0,"name":"bias"}],"gangs":[{"name":"channel_length","kind":"I64","members":[{"node":0,"pos":0,"name":"length"},{"node":1,"pos":0,"name":"length"}]}]}}
|
||||
@@ -0,0 +1,17 @@
|
||||
[
|
||||
{"op": "source", "role": "price", "kind": "F64"},
|
||||
{"op": "add", "type": "RollingMax", "name": "channel_hi"},
|
||||
{"op": "add", "type": "RollingMin", "name": "channel_lo"},
|
||||
{"op": "add", "type": "Add", "name": "sum"},
|
||||
{"op": "add", "type": "Scale", "name": "mid", "bind": {"factor": {"F64": 0.5}}},
|
||||
{"op": "add", "type": "Sub", "name": "dev"},
|
||||
{"op": "add", "type": "Bias", "name": "bias", "bind": {"scale": {"F64": 1.0}}},
|
||||
{"op": "feed", "role": "price", "into": ["channel_hi.series", "channel_lo.series", "dev.lhs"]},
|
||||
{"op": "connect", "from": "channel_hi.value", "to": "sum.lhs"},
|
||||
{"op": "connect", "from": "channel_lo.value", "to": "sum.rhs"},
|
||||
{"op": "connect", "from": "sum.value", "to": "mid.signal"},
|
||||
{"op": "connect", "from": "mid.value", "to": "dev.rhs"},
|
||||
{"op": "connect", "from": "dev.value", "to": "bias.signal"},
|
||||
{"op": "expose", "from": "bias.bias", "as": "bias"},
|
||||
{"op": "gang", "as": "channel_length", "into": ["channel_hi.length", "channel_lo.length"]}
|
||||
]
|
||||
Reference in New Issue
Block a user