fieldtest: document-first — 4 examples, 7 findings

Downstream-author fieldtest of the #300 surface, public interface only,
real GER40 archive. All four axes hold: the closed
generate -> show -> hand-extend -> re-register -> run loop (2-cell
extended campaign), the typed stop grammar's clap rejections, bare
plateau's content-id identity with plateau:mean, and the show read-back
(byte-identical, no framing) with a clean unknown-id refusal.

Findings routed: 1 bug (show outside a project blames the id instead of
the missing Aura.toml — RED-first fix follows), 2 friction items filed
to the tracker (opaque process-document select refusal; no prefix-id
hint on show), 4 working (carry-on). The fieldtest spec survives
git-ignored as the next planning cycle's reference input.

refs #300
This commit is contained in:
2026-07-21 13:16:59 +02:00
parent a12617bb80
commit 63711a1d3a
13 changed files with 230 additions and 0 deletions
@@ -0,0 +1,67 @@
# df — example 1 — the document-first closed loop (Axis 1)
Task: as a headless author, generate a campaign via the walkforward
sugar over real GER40, read it back with `campaign show`, hand-extend
it (add a second data window), re-register, and run the extended doc.
Binary: `target/debug/aura` @ a12617bb (fresh HEAD build).
## generate (walkforward sugar registers a process + campaign, runs it)
$ aura walkforward blueprints/r_sma.json --real GER40 \
--from 1704067200000 --to 1719792000000 \
--axis fast.length=2,4 --name df-wf
aura: axis "fast.length" is not one of this blueprint's sweepable axes
— run 'aura sweep <bp> --list-axes' to see them [exit 2]
The raw op-script param name is rejected by the CLI `--axis`; the error
points at `--list-axes`, which prints the wrapped form:
$ aura sweep blueprints/r_sma.json --list-axes
sma_signal.fast.length:I64 default=2
sma_signal.slow.length:I64 default=4
sma_signal.bias.scale:F64 default=0.5
$ aura walkforward blueprints/r_sma.json --real GER40 \
--from 1704067200000 --to 1719792000000 \
--axis sma_signal.fast.length=2,4 --name df-wf [exit 0]
{"family_id":"248c2d71-0-GER40-w0-r0-s1-0","report":{...}}
... (all-zero R metrics: SMA crossover + vol stop takes 0 trades in
this window — a strategy/data artifact, not the cycle surface)
## read back with `campaign show`
$ aura campaign runs
248c2d71199bc83ea6d40cf5ba13c7c91915d6e07c14464dd4103c065ac4a4a2 run 0
— 1 cell(s), std::grid -> std::walk_forward
$ aura campaign show 248c2d71...a4a2
{"format_version":1,"kind":"campaign","name":"df-wf","data":{"instruments":
["GER40"],"windows":[{"from_ms":1704154499999,"to_ms":1719604380000}]},
"risk":[{"vol":{"length":3,"k":2.0}}],"strategies":[{"ref":{"content_id":
"884b1ba0..."},"axes":{"fast.length":{"kind":"I64","values":[2,4]}}}],
"process":{"ref":{"content_id":"752d6cca..."}},"seed":0,"presentation":
{"persist_taps":[],"emit":["family_table"]}} [exit 0]
- byte-identical to runs/campaigns/248c2d71...a4a2.json (diff clean)
- no trailing newline / no framing (print!, #164)
- axes stored in RAW form `fast.length` (not wrapped) — confirms #210
## hand-extend (add a second window) -> campaigns/df_wf_extended.json
$ aura campaign validate campaigns/df_wf_extended.json
... valid (intrinsic): 1 strategy(ies), 1 axes (2 points), 1 instrument(s),
2 window(s), 1 regime(s) — 2 cell(s)
... valid (referential): all references resolve, axes are in the param space
... valid (executable): pipeline shape and static guards pass [exit 0]
$ aura campaign register campaigns/df_wf_extended.json
registered campaign a72028e8b18ab...f4a1 (...) [exit 0]
## run the extended document
$ aura campaign run a72028e8b18ab...f4a1 [exit 0]
cell window_ms=[1704154499999, 1719604380000] stages=[std::grid, std::walk_forward]
cell window_ms=[1719792000000, 1735689600000] stages=[std::grid, std::walk_forward]
Outcome: full loop generate -> show -> extend -> re-register -> run
completed, exit 0, both windows exercised. WORKING.
@@ -0,0 +1,35 @@
# df — example 2 — the typed stop grammar (Axis 2)
Task: probe `--stop-length` (i64) / `--stop-k` (f64) on `walkforward`
over real GER40 — valid single values, the retired multi-value CSV form,
non-numeric input, and a float into the i64 slot. Binary @ a12617bb.
$ aura walkforward blueprints/r_sma.json --real GER40 \
--from 1704067200000 --to 1719792000000 \
--axis sma_signal.fast.length=2,4 --stop-length 3,5 [exit 2]
error: invalid value '3,5' for '--stop-length <STOP_LENGTH>':
invalid digit found in string
For more information, try '--help'.
$ ... --stop-k abc [exit 2]
error: invalid value 'abc' for '--stop-k <STOP_K>':
invalid float literal
$ ... --stop-length 2.5 [exit 2]
error: invalid value '2.5' for '--stop-length <STOP_LENGTH>':
invalid digit found in string
$ ... --stop-length 5 --stop-k 2 (bare int into f64) [exit 0]
{"walkforward":{...}} # accepted; "2" parses as f64
Outcome: invalid / multi-value / mistyped input is a standard clap
exit-2 argv rejection that names the flag, the offending value, and a
reason. A bare integer coerces into the f64 slot. WORKING.
Note (discoverability): the stdlib reason ("invalid digit found in
string") is terse — for the specific migration case of the retired
multi-value CSV grid (`3,5`, which USED to be a stop grid before this
cycle) it does not hint that stops are now single-valued-by-design or
point at the campaign `risk` regime list as the replacement grid axis.
The standard argv error IS the ratified design, so this is a rough edge,
not a defect.
@@ -0,0 +1,47 @@
# df — example 3 — bare `plateau` vs explicit spellings (Axis 3)
Task: confirm the CLI `--select plateau` sugar is content-id-identical
to `--select plateau:mean`, distinct from the other spellings, refuses
garbage cleanly, and check whether the alias carries into a hand-authored
process *document*. Binary @ a12617bb.
## CLI --select: content-id identity (walkforward over GER40, same --name)
campaign8 = first 8 hex of the generated campaign content id:
--select plateau -> campaign8 = e35e493e
--select plateau:mean -> campaign8 = e35e493e <- IDENTICAL
--select argmax -> campaign8 = eeede651
--select plateau:worst -> campaign8 = ea1f4e3a
`campaign runs` confirms plateau and plateau:mean collapse to ONE stored
id (e35e493ea4a9...d13acc, run 0 + run 1), argmax and plateau:worst are
separate ids. Confirms the cycle claim: the two spellings generate a
content-id-identical campaign.
## CLI refuses an unknown spelling (usage lists the accepted set)
$ aura walkforward ... --select plateau:bogus
aura: Usage: aura walkforward ... [--select <argmax|plateau|plateau:mean|
plateau:worst>] ...
$ aura walkforward ... --select garbage (same usage refusal)
The `--select` help string also documents it:
`argmax | plateau (alias for plateau:mean) | plateau:mean | plateau:worst`.
## the alias does NOT carry into a process document (df_3_..._REFUSED.json)
$ aura process validate df_3_select_plateau_process_REFUSED.json [exit 1]
aura: process document: block std::sweep: unknown select rule "plateau"
$ aura process validate <same doc with "select":"plateau:mean"> [exit 0]
process document valid (intrinsic): 1 pipeline blocks, 0 gate predicates
Outcome: the CLI-flag behaviour and identity claim are WORKING and
documented (help + usage). The FRICTION is the document-side refusal:
`unknown select rule "plateau"` neither enumerates the accepted rules
(argmax | plateau:mean | plateau:worst — as `aura process introspect
--block std::sweep` would) nor bridges bare `plateau` -> `plateau:mean`.
In the document-first hand-extend workflow this cycle is about, an author
who learned bare `plateau` at the CLI will paste it into a document and
hit an opaque refusal.
@@ -0,0 +1 @@
{"format_version":1,"kind":"process","name":"sel-plateau-doc","pipeline":[{"block":"std::sweep","metric":"sqn","select":"plateau"}]}
@@ -0,0 +1,56 @@
# df — example 4 — process show read-back + the show refusal surface (Axis 4)
Task: exercise `process show` as the campaign-side twin (register a
process, read it back, hand-extend with a gate stage, re-register), then
probe the `show` refusal surface a headless author will actually hit:
unknown id, a truncated/prefix id, and invocation outside a project.
Binary @ a12617bb.
## process show read-back + closed loop (processes/df_proc_*.json)
$ aura process register processes/df_proc_base.json
registered process a2149b7c65467797...85cef (...) [exit 0]
$ aura process show a2149b7c65467797...85cef [exit 0]
{"format_version":1,"kind":"process","name":"df-proc-base",...,"pipeline":
[{"block":"std::sweep",...},{"block":"std::walk_forward",...}]}
# byte-identical to stored file; ends "}]}" — no trailing newline (#164)
$ aura process register processes/df_proc_gated.json # + a std::gate stage
registered process a4ad90f4d1380d69...c27 (...) [exit 0]
## refusal surface — INSIDE the project
$ aura campaign show 0000...0000 (unknown 64-hex) [exit 1]
aura: campaign 0000...0000 not found in the project store
$ aura process show 0000...0000 (unknown 64-hex) [exit 1]
aura: process 0000...0000 not found in the project store
$ aura campaign show a72028e8 (a REAL registered id, truncated to 8) [exit 1]
aura: campaign a72028e8 not found in the project store
$ aura process show a2149b7c (truncated) [exit 1]
aura: process a2149b7c not found in the project store
The unknown-id refusal is clean and names id + store + exit 1 (WORKING).
The truncated-id case is FRICTION: `a72028e8` is a real prefix of a
registered campaign, yet the refusal is the flat "not found" — no hint
that the full 64-hex content id is required, no "did you mean", and no
pointer to `aura campaign runs` (which lists valid ids). Every family_id
this surface prints is an 8-hex prefix, so an author will predictably
paste one. (Contrast: `reproduce` gained prefix/name resolution in #298.)
## refusal surface — OUTSIDE any project (no Aura.toml up-tree)
$ cd /tmp/no-project
$ aura campaign show a72028e8b18ab...f4a1 (a valid id) [exit 1]
aura: campaign a72028e8b18ab...f4a1 not found in the project store
$ aura process show a2149b7c65467797...85cef [exit 1]
aura: process a2149b7c... not found in the project store
BUG (misleading diagnostic): outside a project there IS no project store
to look in, yet the prose asserts "not found in the project store",
blaming the id. It should degrade honestly the way `campaign validate`
already does ("referential checks skipped (no Aura.toml found up from
<cwd>)"). A headless author in the wrong directory is told their id is
unregistered when the real cause is "you are not in a project". Behaviour
(exit 1 refusal) is correct — only the prose names a wrong cause.
@@ -0,0 +1 @@
/runs
@@ -0,0 +1,4 @@
# Static project context only (C17); paths only.
[paths]
runs = "runs"
# data = "/path/to/archive" # the recorded-data root; defaults to the built-in path
@@ -0,0 +1,14 @@
# lab — an aura research project (data-only)
This directory is an aura project: blueprints + research documents over the
std vocabulary, anchored by `Aura.toml`. There is no crate and no build step.
- Run: `aura run blueprints/signal.json` (the starter is closed — all
params bound; bound values are defaults — any `--axis` may override them
(#246))
- Sweep: `aura sweep blueprints/signal.json --axis lab_signal.fast.length=2,4,8`
(`aura sweep <bp> --list-axes` lists the open + bound-overridable axes)
- Native nodes: when the project needs its first project-specific node,
`aura nodes new <name>` scaffolds a node crate beside this project and
attaches it via `[nodes]` in `Aura.toml` (build it with `cargo build`).
- Topology is data (`blueprints/*.json`); results land in `runs/`.
@@ -0,0 +1 @@
{"format_version":1,"blueprint":{"name":"sma_signal","nodes":[{"primitive":{"type":"SMA","name":"fast","bound":[{"pos":0,"name":"length","kind":"I64","value":{"I64":2}}]}},{"primitive":{"type":"SMA","name":"slow","bound":[{"pos":0,"name":"length","kind":"I64","value":{"I64":4}}]}},{"primitive":{"type":"Sub"}},{"primitive":{"type":"Bias","name":"bias","bound":[{"pos":0,"name":"scale","kind":"F64","value":{"F64":0.5}}]}}],"edges":[{"from":0,"to":2,"slot":0,"from_field":0},{"from":1,"to":2,"slot":1,"from_field":0},{"from":2,"to":3,"slot":0,"from_field":0}],"input_roles":[{"name":"price","targets":[{"node":0,"slot":0},{"node":1,"slot":0}],"source":"F64"}],"output":[{"node":3,"field":0,"name":"bias"}]}}
@@ -0,0 +1 @@
{"format_version":1,"blueprint":{"name":"lab_signal","nodes":[{"primitive":{"type":"SMA","name":"fast","bound":[{"pos":0,"name":"length","kind":"I64","value":{"I64":2}}]}},{"primitive":{"type":"SMA","name":"slow","bound":[{"pos":0,"name":"length","kind":"I64","value":{"I64":4}}]}},{"primitive":{"type":"Sub"}},{"primitive":{"type":"Bias","name":"bias","bound":[{"pos":0,"name":"scale","kind":"F64","value":{"F64":0.5}}]}}],"edges":[{"from":0,"to":2,"slot":0,"from_field":0},{"from":1,"to":2,"slot":1,"from_field":0},{"from":2,"to":3,"slot":0,"from_field":0}],"input_roles":[{"name":"price","targets":[{"node":0,"slot":0},{"node":1,"slot":0}],"source":"F64"}],"output":[{"node":3,"field":0,"name":"bias"}]}}
@@ -0,0 +1 @@
{"format_version":1,"kind":"campaign","name":"df-wf-extended","data":{"instruments":["GER40"],"windows":[{"from_ms":1704154499999,"to_ms":1719604380000},{"from_ms":1719792000000,"to_ms":1735689600000}]},"risk":[{"vol":{"length":3,"k":2.0}}],"strategies":[{"ref":{"content_id":"884b1ba0640b2f5bb661c8d47e169db85ac2bb729ff5aa1ff3f9c930085da6d9"},"axes":{"fast.length":{"kind":"I64","values":[2,4]}}}],"process":{"ref":{"content_id":"752d6cca3e4d00b54e99c8c8cf24e47f44a27830ea8ba63427fac2a4f7a44719"}},"seed":0,"presentation":{"persist_taps":[],"emit":["family_table"]}}
@@ -0,0 +1 @@
{"format_version":1,"kind":"process","name":"df-proc-base","description":"Base: sweep then walk-forward.","pipeline":[{"block":"std::sweep","metric":"sqn","select":"argmax"},{"block":"std::walk_forward","in_sample_ms":1209600000,"out_of_sample_ms":604800000,"step_ms":604800000,"mode":"rolling","metric":"sqn","select":"argmax"}]}
@@ -0,0 +1 @@
{"format_version":1,"kind":"process","name":"df-proc-gated","description":"Base: sweep then walk-forward.","pipeline":[{"block":"std::sweep","metric":"sqn","select":"argmax"},{"block":"std::gate","all":[{"metric":"expectancy_r","cmp":"gt","value":0.0}]},{"block":"std::walk_forward","in_sample_ms":1209600000,"out_of_sample_ms":604800000,"step_ms":604800000,"mode":"rolling","metric":"sqn","select":"argmax"}]}