fieldtest: document-first — 4 examples, 7 findings
Downstream-author fieldtest of the #300 surface, public interface only, real GER40 archive. All four axes hold: the closed generate -> show -> hand-extend -> re-register -> run loop (2-cell extended campaign), the typed stop grammar's clap rejections, bare plateau's content-id identity with plateau:mean, and the show read-back (byte-identical, no framing) with a clean unknown-id refusal. Findings routed: 1 bug (show outside a project blames the id instead of the missing Aura.toml — RED-first fix follows), 2 friction items filed to the tracker (opaque process-document select refusal; no prefix-id hint on show), 4 working (carry-on). The fieldtest spec survives git-ignored as the next planning cycle's reference input. refs #300
This commit is contained in:
@@ -0,0 +1,67 @@
|
||||
# df — example 1 — the document-first closed loop (Axis 1)
|
||||
|
||||
Task: as a headless author, generate a campaign via the walkforward
|
||||
sugar over real GER40, read it back with `campaign show`, hand-extend
|
||||
it (add a second data window), re-register, and run the extended doc.
|
||||
Binary: `target/debug/aura` @ a12617bb (fresh HEAD build).
|
||||
|
||||
## generate (walkforward sugar registers a process + campaign, runs it)
|
||||
|
||||
$ aura walkforward blueprints/r_sma.json --real GER40 \
|
||||
--from 1704067200000 --to 1719792000000 \
|
||||
--axis fast.length=2,4 --name df-wf
|
||||
aura: axis "fast.length" is not one of this blueprint's sweepable axes
|
||||
— run 'aura sweep <bp> --list-axes' to see them [exit 2]
|
||||
|
||||
The raw op-script param name is rejected by the CLI `--axis`; the error
|
||||
points at `--list-axes`, which prints the wrapped form:
|
||||
|
||||
$ aura sweep blueprints/r_sma.json --list-axes
|
||||
sma_signal.fast.length:I64 default=2
|
||||
sma_signal.slow.length:I64 default=4
|
||||
sma_signal.bias.scale:F64 default=0.5
|
||||
|
||||
$ aura walkforward blueprints/r_sma.json --real GER40 \
|
||||
--from 1704067200000 --to 1719792000000 \
|
||||
--axis sma_signal.fast.length=2,4 --name df-wf [exit 0]
|
||||
{"family_id":"248c2d71-0-GER40-w0-r0-s1-0","report":{...}}
|
||||
... (all-zero R metrics: SMA crossover + vol stop takes 0 trades in
|
||||
this window — a strategy/data artifact, not the cycle surface)
|
||||
|
||||
## read back with `campaign show`
|
||||
|
||||
$ aura campaign runs
|
||||
248c2d71199bc83ea6d40cf5ba13c7c91915d6e07c14464dd4103c065ac4a4a2 run 0
|
||||
— 1 cell(s), std::grid -> std::walk_forward
|
||||
|
||||
$ aura campaign show 248c2d71...a4a2
|
||||
{"format_version":1,"kind":"campaign","name":"df-wf","data":{"instruments":
|
||||
["GER40"],"windows":[{"from_ms":1704154499999,"to_ms":1719604380000}]},
|
||||
"risk":[{"vol":{"length":3,"k":2.0}}],"strategies":[{"ref":{"content_id":
|
||||
"884b1ba0..."},"axes":{"fast.length":{"kind":"I64","values":[2,4]}}}],
|
||||
"process":{"ref":{"content_id":"752d6cca..."}},"seed":0,"presentation":
|
||||
{"persist_taps":[],"emit":["family_table"]}} [exit 0]
|
||||
|
||||
- byte-identical to runs/campaigns/248c2d71...a4a2.json (diff clean)
|
||||
- no trailing newline / no framing (print!, #164)
|
||||
- axes stored in RAW form `fast.length` (not wrapped) — confirms #210
|
||||
|
||||
## hand-extend (add a second window) -> campaigns/df_wf_extended.json
|
||||
|
||||
$ aura campaign validate campaigns/df_wf_extended.json
|
||||
... valid (intrinsic): 1 strategy(ies), 1 axes (2 points), 1 instrument(s),
|
||||
2 window(s), 1 regime(s) — 2 cell(s)
|
||||
... valid (referential): all references resolve, axes are in the param space
|
||||
... valid (executable): pipeline shape and static guards pass [exit 0]
|
||||
|
||||
$ aura campaign register campaigns/df_wf_extended.json
|
||||
registered campaign a72028e8b18ab...f4a1 (...) [exit 0]
|
||||
|
||||
## run the extended document
|
||||
|
||||
$ aura campaign run a72028e8b18ab...f4a1 [exit 0]
|
||||
cell window_ms=[1704154499999, 1719604380000] stages=[std::grid, std::walk_forward]
|
||||
cell window_ms=[1719792000000, 1735689600000] stages=[std::grid, std::walk_forward]
|
||||
|
||||
Outcome: full loop generate -> show -> extend -> re-register -> run
|
||||
completed, exit 0, both windows exercised. WORKING.
|
||||
@@ -0,0 +1,35 @@
|
||||
# df — example 2 — the typed stop grammar (Axis 2)
|
||||
|
||||
Task: probe `--stop-length` (i64) / `--stop-k` (f64) on `walkforward`
|
||||
over real GER40 — valid single values, the retired multi-value CSV form,
|
||||
non-numeric input, and a float into the i64 slot. Binary @ a12617bb.
|
||||
|
||||
$ aura walkforward blueprints/r_sma.json --real GER40 \
|
||||
--from 1704067200000 --to 1719792000000 \
|
||||
--axis sma_signal.fast.length=2,4 --stop-length 3,5 [exit 2]
|
||||
error: invalid value '3,5' for '--stop-length <STOP_LENGTH>':
|
||||
invalid digit found in string
|
||||
For more information, try '--help'.
|
||||
|
||||
$ ... --stop-k abc [exit 2]
|
||||
error: invalid value 'abc' for '--stop-k <STOP_K>':
|
||||
invalid float literal
|
||||
|
||||
$ ... --stop-length 2.5 [exit 2]
|
||||
error: invalid value '2.5' for '--stop-length <STOP_LENGTH>':
|
||||
invalid digit found in string
|
||||
|
||||
$ ... --stop-length 5 --stop-k 2 (bare int into f64) [exit 0]
|
||||
{"walkforward":{...}} # accepted; "2" parses as f64
|
||||
|
||||
Outcome: invalid / multi-value / mistyped input is a standard clap
|
||||
exit-2 argv rejection that names the flag, the offending value, and a
|
||||
reason. A bare integer coerces into the f64 slot. WORKING.
|
||||
|
||||
Note (discoverability): the stdlib reason ("invalid digit found in
|
||||
string") is terse — for the specific migration case of the retired
|
||||
multi-value CSV grid (`3,5`, which USED to be a stop grid before this
|
||||
cycle) it does not hint that stops are now single-valued-by-design or
|
||||
point at the campaign `risk` regime list as the replacement grid axis.
|
||||
The standard argv error IS the ratified design, so this is a rough edge,
|
||||
not a defect.
|
||||
@@ -0,0 +1,47 @@
|
||||
# df — example 3 — bare `plateau` vs explicit spellings (Axis 3)
|
||||
|
||||
Task: confirm the CLI `--select plateau` sugar is content-id-identical
|
||||
to `--select plateau:mean`, distinct from the other spellings, refuses
|
||||
garbage cleanly, and check whether the alias carries into a hand-authored
|
||||
process *document*. Binary @ a12617bb.
|
||||
|
||||
## CLI --select: content-id identity (walkforward over GER40, same --name)
|
||||
|
||||
campaign8 = first 8 hex of the generated campaign content id:
|
||||
|
||||
--select plateau -> campaign8 = e35e493e
|
||||
--select plateau:mean -> campaign8 = e35e493e <- IDENTICAL
|
||||
--select argmax -> campaign8 = eeede651
|
||||
--select plateau:worst -> campaign8 = ea1f4e3a
|
||||
|
||||
`campaign runs` confirms plateau and plateau:mean collapse to ONE stored
|
||||
id (e35e493ea4a9...d13acc, run 0 + run 1), argmax and plateau:worst are
|
||||
separate ids. Confirms the cycle claim: the two spellings generate a
|
||||
content-id-identical campaign.
|
||||
|
||||
## CLI refuses an unknown spelling (usage lists the accepted set)
|
||||
|
||||
$ aura walkforward ... --select plateau:bogus
|
||||
aura: Usage: aura walkforward ... [--select <argmax|plateau|plateau:mean|
|
||||
plateau:worst>] ...
|
||||
$ aura walkforward ... --select garbage (same usage refusal)
|
||||
|
||||
The `--select` help string also documents it:
|
||||
`argmax | plateau (alias for plateau:mean) | plateau:mean | plateau:worst`.
|
||||
|
||||
## the alias does NOT carry into a process document (df_3_..._REFUSED.json)
|
||||
|
||||
$ aura process validate df_3_select_plateau_process_REFUSED.json [exit 1]
|
||||
aura: process document: block std::sweep: unknown select rule "plateau"
|
||||
|
||||
$ aura process validate <same doc with "select":"plateau:mean"> [exit 0]
|
||||
process document valid (intrinsic): 1 pipeline blocks, 0 gate predicates
|
||||
|
||||
Outcome: the CLI-flag behaviour and identity claim are WORKING and
|
||||
documented (help + usage). The FRICTION is the document-side refusal:
|
||||
`unknown select rule "plateau"` neither enumerates the accepted rules
|
||||
(argmax | plateau:mean | plateau:worst — as `aura process introspect
|
||||
--block std::sweep` would) nor bridges bare `plateau` -> `plateau:mean`.
|
||||
In the document-first hand-extend workflow this cycle is about, an author
|
||||
who learned bare `plateau` at the CLI will paste it into a document and
|
||||
hit an opaque refusal.
|
||||
@@ -0,0 +1 @@
|
||||
{"format_version":1,"kind":"process","name":"sel-plateau-doc","pipeline":[{"block":"std::sweep","metric":"sqn","select":"plateau"}]}
|
||||
@@ -0,0 +1,56 @@
|
||||
# df — example 4 — process show read-back + the show refusal surface (Axis 4)
|
||||
|
||||
Task: exercise `process show` as the campaign-side twin (register a
|
||||
process, read it back, hand-extend with a gate stage, re-register), then
|
||||
probe the `show` refusal surface a headless author will actually hit:
|
||||
unknown id, a truncated/prefix id, and invocation outside a project.
|
||||
Binary @ a12617bb.
|
||||
|
||||
## process show read-back + closed loop (processes/df_proc_*.json)
|
||||
|
||||
$ aura process register processes/df_proc_base.json
|
||||
registered process a2149b7c65467797...85cef (...) [exit 0]
|
||||
|
||||
$ aura process show a2149b7c65467797...85cef [exit 0]
|
||||
{"format_version":1,"kind":"process","name":"df-proc-base",...,"pipeline":
|
||||
[{"block":"std::sweep",...},{"block":"std::walk_forward",...}]}
|
||||
# byte-identical to stored file; ends "}]}" — no trailing newline (#164)
|
||||
|
||||
$ aura process register processes/df_proc_gated.json # + a std::gate stage
|
||||
registered process a4ad90f4d1380d69...c27 (...) [exit 0]
|
||||
|
||||
## refusal surface — INSIDE the project
|
||||
|
||||
$ aura campaign show 0000...0000 (unknown 64-hex) [exit 1]
|
||||
aura: campaign 0000...0000 not found in the project store
|
||||
$ aura process show 0000...0000 (unknown 64-hex) [exit 1]
|
||||
aura: process 0000...0000 not found in the project store
|
||||
|
||||
$ aura campaign show a72028e8 (a REAL registered id, truncated to 8) [exit 1]
|
||||
aura: campaign a72028e8 not found in the project store
|
||||
$ aura process show a2149b7c (truncated) [exit 1]
|
||||
aura: process a2149b7c not found in the project store
|
||||
|
||||
The unknown-id refusal is clean and names id + store + exit 1 (WORKING).
|
||||
The truncated-id case is FRICTION: `a72028e8` is a real prefix of a
|
||||
registered campaign, yet the refusal is the flat "not found" — no hint
|
||||
that the full 64-hex content id is required, no "did you mean", and no
|
||||
pointer to `aura campaign runs` (which lists valid ids). Every family_id
|
||||
this surface prints is an 8-hex prefix, so an author will predictably
|
||||
paste one. (Contrast: `reproduce` gained prefix/name resolution in #298.)
|
||||
|
||||
## refusal surface — OUTSIDE any project (no Aura.toml up-tree)
|
||||
|
||||
$ cd /tmp/no-project
|
||||
$ aura campaign show a72028e8b18ab...f4a1 (a valid id) [exit 1]
|
||||
aura: campaign a72028e8b18ab...f4a1 not found in the project store
|
||||
$ aura process show a2149b7c65467797...85cef [exit 1]
|
||||
aura: process a2149b7c... not found in the project store
|
||||
|
||||
BUG (misleading diagnostic): outside a project there IS no project store
|
||||
to look in, yet the prose asserts "not found in the project store",
|
||||
blaming the id. It should degrade honestly the way `campaign validate`
|
||||
already does ("referential checks skipped (no Aura.toml found up from
|
||||
<cwd>)"). A headless author in the wrong directory is told their id is
|
||||
unregistered when the real cause is "you are not in a project". Behaviour
|
||||
(exit 1 refusal) is correct — only the prose names a wrong cause.
|
||||
@@ -0,0 +1 @@
|
||||
/runs
|
||||
@@ -0,0 +1,4 @@
|
||||
# Static project context only (C17); paths only.
|
||||
[paths]
|
||||
runs = "runs"
|
||||
# data = "/path/to/archive" # the recorded-data root; defaults to the built-in path
|
||||
@@ -0,0 +1,14 @@
|
||||
# lab — an aura research project (data-only)
|
||||
|
||||
This directory is an aura project: blueprints + research documents over the
|
||||
std vocabulary, anchored by `Aura.toml`. There is no crate and no build step.
|
||||
|
||||
- Run: `aura run blueprints/signal.json` (the starter is closed — all
|
||||
params bound; bound values are defaults — any `--axis` may override them
|
||||
(#246))
|
||||
- Sweep: `aura sweep blueprints/signal.json --axis lab_signal.fast.length=2,4,8`
|
||||
(`aura sweep <bp> --list-axes` lists the open + bound-overridable axes)
|
||||
- Native nodes: when the project needs its first project-specific node,
|
||||
`aura nodes new <name>` scaffolds a node crate beside this project and
|
||||
attaches it via `[nodes]` in `Aura.toml` (build it with `cargo build`).
|
||||
- Topology is data (`blueprints/*.json`); results land in `runs/`.
|
||||
@@ -0,0 +1 @@
|
||||
{"format_version":1,"blueprint":{"name":"sma_signal","nodes":[{"primitive":{"type":"SMA","name":"fast","bound":[{"pos":0,"name":"length","kind":"I64","value":{"I64":2}}]}},{"primitive":{"type":"SMA","name":"slow","bound":[{"pos":0,"name":"length","kind":"I64","value":{"I64":4}}]}},{"primitive":{"type":"Sub"}},{"primitive":{"type":"Bias","name":"bias","bound":[{"pos":0,"name":"scale","kind":"F64","value":{"F64":0.5}}]}}],"edges":[{"from":0,"to":2,"slot":0,"from_field":0},{"from":1,"to":2,"slot":1,"from_field":0},{"from":2,"to":3,"slot":0,"from_field":0}],"input_roles":[{"name":"price","targets":[{"node":0,"slot":0},{"node":1,"slot":0}],"source":"F64"}],"output":[{"node":3,"field":0,"name":"bias"}]}}
|
||||
@@ -0,0 +1 @@
|
||||
{"format_version":1,"blueprint":{"name":"lab_signal","nodes":[{"primitive":{"type":"SMA","name":"fast","bound":[{"pos":0,"name":"length","kind":"I64","value":{"I64":2}}]}},{"primitive":{"type":"SMA","name":"slow","bound":[{"pos":0,"name":"length","kind":"I64","value":{"I64":4}}]}},{"primitive":{"type":"Sub"}},{"primitive":{"type":"Bias","name":"bias","bound":[{"pos":0,"name":"scale","kind":"F64","value":{"F64":0.5}}]}}],"edges":[{"from":0,"to":2,"slot":0,"from_field":0},{"from":1,"to":2,"slot":1,"from_field":0},{"from":2,"to":3,"slot":0,"from_field":0}],"input_roles":[{"name":"price","targets":[{"node":0,"slot":0},{"node":1,"slot":0}],"source":"F64"}],"output":[{"node":3,"field":0,"name":"bias"}]}}
|
||||
@@ -0,0 +1 @@
|
||||
{"format_version":1,"kind":"campaign","name":"df-wf-extended","data":{"instruments":["GER40"],"windows":[{"from_ms":1704154499999,"to_ms":1719604380000},{"from_ms":1719792000000,"to_ms":1735689600000}]},"risk":[{"vol":{"length":3,"k":2.0}}],"strategies":[{"ref":{"content_id":"884b1ba0640b2f5bb661c8d47e169db85ac2bb729ff5aa1ff3f9c930085da6d9"},"axes":{"fast.length":{"kind":"I64","values":[2,4]}}}],"process":{"ref":{"content_id":"752d6cca3e4d00b54e99c8c8cf24e47f44a27830ea8ba63427fac2a4f7a44719"}},"seed":0,"presentation":{"persist_taps":[],"emit":["family_table"]}}
|
||||
@@ -0,0 +1 @@
|
||||
{"format_version":1,"kind":"process","name":"df-proc-base","description":"Base: sweep then walk-forward.","pipeline":[{"block":"std::sweep","metric":"sqn","select":"argmax"},{"block":"std::walk_forward","in_sample_ms":1209600000,"out_of_sample_ms":604800000,"step_ms":604800000,"mode":"rolling","metric":"sqn","select":"argmax"}]}
|
||||
@@ -0,0 +1 @@
|
||||
{"format_version":1,"kind":"process","name":"df-proc-gated","description":"Base: sweep then walk-forward.","pipeline":[{"block":"std::sweep","metric":"sqn","select":"argmax"},{"block":"std::gate","all":[{"metric":"expectancy_r","cmp":"gt","value":0.0}]},{"block":"std::walk_forward","in_sample_ms":1209600000,"out_of_sample_ms":604800000,"step_ms":604800000,"mode":"rolling","metric":"sqn","select":"argmax"}]}
|
||||
Reference in New Issue
Block a user