Brummel
36fbf492ee
Consumer scenario against the public surface only: selection-free process doc round-trip, the dissolved real-data sweep (doc dedupe + realization accumulation + C1 bit-identity), refusal edges, rank-later over the persisted family. 0 bugs; the cycle's own axes all came back working. 4 friction + 2 spec-gap findings on the verb entry ergonomics and pre-existing guide drift — transcribed in docs/specs/fieldtest-sweep-dissolution.md for triage. refs #210
246 lines
15 KiB
Markdown
246 lines
15 KiB
Markdown
# Fieldtest — cycle 0110 (sweep dissolution) — 2026-07-04
|
||
|
||
**Status:** Draft — awaiting orchestrator triage
|
||
**Author:** fieldtester (dispatched by fieldtest skill)
|
||
**Binary exercised:** `target/debug/aura` (workspace built from HEAD
|
||
`2c72996`; project cdylib `liblab.so` built from the scaffolded `lab/` crate).
|
||
|
||
## Scope
|
||
Cycle 0110 is cycle 1 of the "verb dissolution" milestone (#210). Two
|
||
user-visible changes: (1) the `std::sweep` process block's selection triple
|
||
(`metric`+`select`+`deflate`) became an optional, all-or-nothing selection
|
||
group — omit it for a **selection-free sweep**, permitted only as the
|
||
pipeline's terminal stage; (2) `aura sweep <blueprint.json> --real SYM --from
|
||
--to --axis k=v,…` is now thin sugar over the campaign engine: it
|
||
auto-registers a generated selection-free process document + a campaign
|
||
document (content-addressed) and appends a campaign-run record, printing the
|
||
same member lines as before (member manifests now carry an `instrument`
|
||
stamp). This fieldtest exercises both, plus the two consumer paths the
|
||
selection-free sweep exists for (refusal edges, rank-later).
|
||
|
||
## Examples
|
||
All fixtures under `fieldtests/cycle-0110-sweep-dissolution/`; the scaffolded
|
||
project is `lab/` (its `runs/` store is kept as evidence, `target/` removed).
|
||
Real data: GER40, Sep 2024 (`--from 1725148800000 --to 1727740799999`).
|
||
|
||
### c0110_1_explore_sweep.json — selection-free single-stage process document
|
||
- A bare `{"block":"std::sweep"}` pipeline authored straight from
|
||
authoring-guide §2.
|
||
- Fits scope: exercises the new optional-selection-group surface at its
|
||
simplest (the omitted group).
|
||
- Outcome: `process validate` → `valid (intrinsic): 1 pipeline blocks`;
|
||
`process register` → id `78f2d8d7…`, re-register dedupes (one file);
|
||
stored canonical form is byte-exactly `…"pipeline":[{"block":"std::sweep"}]`.
|
||
`process introspect --block std::sweep` matches the guide verbatim
|
||
(metric/select/deflate all "optional"). **Ran, matched expected.**
|
||
|
||
### c0110_2_smacross_open_op.json (+ built blueprint) — the dissolved verb
|
||
- Unbound SMA-crossover op-script → `graph build` → blueprint with three open
|
||
axes; `aura sweep <bp> --real GER40 --from … --to … --axis
|
||
graph.fast.length=2,4 --axis graph.slow.length=8,16 --axis
|
||
graph.bias.scale=0.5 --name c0110-explore`.
|
||
- Fits scope: this *is* the dissolved verb; a realistic ad-hoc explore-sweep.
|
||
- Outcome: prints 4 GER40-stamped member lines; auto-registers a generated
|
||
process (`9227c300`, name "sweep", selection-free) + campaign (`46e4f7ef`,
|
||
raw axis names, GER40, ms sub-range window) and appends one
|
||
`campaign_runs.jsonl` record; identical re-run **dedupes both documents**
|
||
(no new files, same ids) while appending a second realization (family
|
||
suffix `s0-0`→`s0-1`, metrics bit-identical — C1); `campaign runs` lists
|
||
both. **Ran, matched expected.** (`c0110_2_smacross_op.json`, the *bound*
|
||
variant, is kept as evidence for finding F3.)
|
||
|
||
### c0110_3a_metric_only.json / c0110_3a_select_only.json / c0110_3b_* — refusal edges
|
||
- Half-populated selection group (metric-only, select-only); a selection-free
|
||
sweep followed by a `std::gate` stage, driven to the executor via
|
||
`c0110_3b_campaign_nonterminal.json`.
|
||
- Fits scope: the two new refusals (all-or-nothing group; terminal-only).
|
||
- Outcome: half-group → clean intrinsic refusal naming the missing slot;
|
||
non-terminal free sweep → passes intrinsic, refused at the executable tier
|
||
and at `campaign run` with clear prose, no realization appended. **Ran,
|
||
matched expected.**
|
||
|
||
### (Task 4) rank-later over the persisted family — `aura runs family … rank …`
|
||
- No new fixture; consumes the `46e4f7ef-0-GER40-w0-s0-0` family persisted by
|
||
example 2.
|
||
- Fits scope: the rank-later pattern the selection-free sweep exists for.
|
||
- Outcome: `rank sqn` / `rank expectancy_r` return the 4 members best-first;
|
||
ranking respects metric directionality (max_drawdown ascending, sqn
|
||
descending); the rank roster equals the `rankable` metric roster
|
||
(`win_rate`/nonexistent refused with the known list). **Ran, matched
|
||
expected.**
|
||
|
||
## Findings
|
||
|
||
### [working] Selection-free process document round-trips cleanly
|
||
- Example 1.
|
||
- `process validate` → `process document valid (intrinsic): 1 pipeline
|
||
blocks, 0 gate predicates`; register id `78f2d8d7…`, idempotent on
|
||
re-register; stored form is the bare block; `introspect --block std::sweep`
|
||
output is byte-for-byte the guide's transcription.
|
||
- Working: the new optional group is reachable, authorable from the doc alone,
|
||
and the introspection self-description matches the ledger/guide.
|
||
- Action: carry-on.
|
||
|
||
### [working] The dissolved sweep verb generates + dedupes a selection-free campaign
|
||
- Example 2.
|
||
- One invocation produced the generated process `9227c300`
|
||
(`{"name":"sweep","pipeline":[{"block":"std::sweep"}]}`), campaign
|
||
`46e4f7ef` (raw axes `fast.length`/`slow.length`/`bias.scale`, `instrument:
|
||
GER40`, window `[1725236099999,1727726220000]` — a ms sub-range of the
|
||
request), and one `campaign_runs.jsonl` line. Identical re-run: **no new
|
||
documents** (same content ids), a second realization (`run 1`, family suffix
|
||
`s0-1`), member metrics bit-identical. `campaign runs` lists both runs.
|
||
- Working: the sugar→campaign parity, content-addressed dedup, deterministic
|
||
accumulation (C1), and the additive `instrument` stamp all hold end to end.
|
||
- Action: carry-on.
|
||
|
||
### [working] All-or-nothing + terminal-only refusals are clear and guide-matching
|
||
- Examples 3a/3b.
|
||
- metric-only → `block std::sweep: slot "metric" without "select" — the
|
||
selection group is all-or-nothing`; select-only symmetric; non-terminal free
|
||
sweep → `campaign is not executable: process stage 0: a sweep without a
|
||
selection group (metric + select) must be the last stage of its process`
|
||
(same prose at `campaign run`, no realization appended).
|
||
- Working: both new refusals fire at the right tier with comprehensible prose
|
||
that matches authoring-guide §2.
|
||
- Action: carry-on.
|
||
|
||
### [working] Rank-later over the persisted family, best-first, roster-enforced
|
||
- Task 4.
|
||
- `rank sqn` orders 1.054 > 0.849 > 0.052 > −0.545; `rank max_drawdown` orders
|
||
4.22M < 6.16M < 6.57M < 9.14M (ascending — lower drawdown ranked better);
|
||
`rank win_rate` → `unknown metric 'win_rate' (known: total_pips,
|
||
max_drawdown, bias_sign_flips, sqn, sqn_normalized, expectancy_r,
|
||
net_expectancy_r)`.
|
||
- Working: the selection-free sweep's reason to exist (defer selection to a
|
||
rank-later step) works, and the rank roster equals the documented `rankable`
|
||
set. (Direction convention noted as F-min below.)
|
||
- Action: carry-on.
|
||
|
||
### [spec_gap] `aura sweep --axis` requires the `graph.`-wrapped name; the docs teach the raw form
|
||
- Example 2 (works with `graph.fast.length`) vs probe P2 (fails with
|
||
`fast.length`).
|
||
- `graph introspect --params` prints `fast.length` and the authoring guide
|
||
(§1) calls that "the raw param-space namespace (what a campaign's axes bind
|
||
against)". The stored campaign doc also uses the raw form. But the dissolved
|
||
`aura sweep --axis` accepts **only** the wrapped `graph.<param>` form —
|
||
discoverable solely via `aura sweep <bp> --list-axes`, which the authoring
|
||
guide never mentions (the guide does not document `aura sweep` at all; the
|
||
glossary and `--help` say only `--axis name=<csv>`).
|
||
- Why spec_gap: the public interface documents the raw namespace as canonical
|
||
and is silent on the wrapper; a consumer following it picks the wrong form.
|
||
The wrapper leaks the root-composite instance name (`graph`) into the user's
|
||
axis vocabulary.
|
||
- Recommended action: document (glossary `sweep` + a note in authoring-guide)
|
||
that `aura sweep --axis` uses the `--list-axes` (wrapped) names, **or** make
|
||
the dispatch accept the raw `--params` names too. → tighten the design
|
||
ledger / docs (+ plan if the surface is to accept raw).
|
||
|
||
### [friction] The wrapped-name refusal mangles the axis the user typed
|
||
- Probe P2: `aura sweep … --axis fast.length=2,4 --axis slow.length=8,16
|
||
--axis bias.scale=0.5` →
|
||
`strategy 597d…: axis "length" is not in the param space` /
|
||
`axis "scale" is not in the param space` /
|
||
`open param "fast.length" is bound by no campaign axis` …
|
||
- The `#203` first-segment strip consumes the leading `fast`/`slow`/`bias`
|
||
segment, so the diagnostic names `length`/`scale` — fragments the user never
|
||
typed as a whole axis — while separately reporting the real params as
|
||
"bound by no axis". The two same-stripped legs (`fast.length`,`slow.length`)
|
||
even collapse to a single `length` key. The message never says "a leading
|
||
segment was stripped as the graph name".
|
||
- Why friction: the task can be completed (use the wrapped name), but the
|
||
refusal actively misdirects — it names a mangled fragment instead of echoing
|
||
the input and explaining the wrapper convention. (Same root cause as the
|
||
spec_gap above; split per the one-class-per-finding rule.)
|
||
- Recommended action: echo the typed axis and/or name the expected wrapper in
|
||
the refusal. → plan (tidy iteration).
|
||
|
||
### [spec_gap] Authoring-guide §1 inline SMA-crossover binds `length`, contradicting every id / `--params` / §3 axis it prints
|
||
- Encountered building the sweep blueprint (Task 2); evidence
|
||
`c0110_2_smacross_op.json`.
|
||
- The guide's §1 worked op-script binds `fast.length=2` and `slow.length=4`
|
||
and claims (line 72) it matches the on-disk corpus
|
||
`mra_1_strategy_smacross.json`. It does not: the on-disk file leaves both
|
||
**unbound**. The bound inline form hashes to `e2be81b2…` with `--params` =
|
||
`bias.scale` only; the unbound corpus form hashes to `597d719b…` (the id the
|
||
guide later prints at §1/§3 and the id every campaign example references)
|
||
with `--params` = `fast.length`,`slow.length`,`bias.scale`. A reader copying
|
||
the inline example gets a blueprint whose id matches nothing in the guide and
|
||
whose `fast.length`/`slow.length` are closed — so authoring-guide §3's
|
||
campaign (axes `fast.length`,`slow.length`) would refuse against it.
|
||
- Why spec_gap: the guide is internally inconsistent, and the underlying rule —
|
||
that an op-script `bind` *pins a param out of the sweepable space* (not a
|
||
sweep-overridable default) — is nowhere stated. Note the 0110 audit commit
|
||
claimed "worked example verified against the live binary"; this §1 example
|
||
(authored earlier, #197/#208) was not.
|
||
- Recommended action: drop the `bind` from the §1 inline example (or state
|
||
that binding closes the axis) and document `bind`-vs-axis interaction. →
|
||
tighten docs.
|
||
|
||
### [friction] `aura sweep <op-script>` rejects the op-script array with a raw serde error
|
||
- Probe: `aura sweep c0110_2_smacross_open_op.json --list-axes` →
|
||
`aura: …_op.json: Json(Error("invalid type: map, expected u32", line: 2,
|
||
column: 2))`.
|
||
- `graph introspect --content-id/--params/register` auto-discriminate the
|
||
op-script array vs the built envelope (guide §1), but `aura sweep`'s
|
||
blueprint slot accepts only the built envelope, and the mismatch surfaces as
|
||
a leaked serde `Json(Error(...))` rather than a house-style
|
||
"expected a built blueprint; run `aura graph build` first". The guide never
|
||
says `aura sweep` needs a pre-built blueprint.
|
||
- Why friction: the path completes once you build first, but the diagnostic is
|
||
an internal-type leak (contrast #162's Display-free-error convention seen
|
||
elsewhere) and the required build step is undocumented for `aura sweep`.
|
||
- Recommended action: shape-discriminate (as `graph introspect` does) or emit
|
||
a house-style hint. → plan.
|
||
|
||
### [friction] Refused dissolved sweeps still register their generated campaign document (store litter, some malformed)
|
||
- Probes P2 (raw names) and P3 (subset axes) both **refused** at the
|
||
referential tier, yet each left a generated campaign doc in
|
||
`runs/campaigns/` (`92c66b18…` "c0110-rawname", `b0e3f90b…` "c0110-subset").
|
||
`campaign_runs.jsonl` correctly stayed at 2 (no realization). The rawname
|
||
doc is *malformed*: its axes are the stripped/collapsed `{"length":[8,16],
|
||
"scale":[0.5]}` — a document the tool itself immediately rejects, now
|
||
permanent in the content-addressed store.
|
||
- Why friction: the sugar registers the generated process+campaign documents
|
||
*before* referential/executable validation, so a fat-fingered axis pollutes
|
||
the store with dead (occasionally malformed) documents. Content-addressing
|
||
bounds the growth (identical bad invocations dedupe), and no realization is
|
||
recorded, so nothing downstream breaks — hence friction, not bug.
|
||
(`campaign run <file>` registering before an executable refusal — the Task-3
|
||
`d10ed054…` doc — is by contrast *documented* register-then-run behaviour and
|
||
is not flagged.)
|
||
- Recommended action: in the sweep sugar, validate the generated documents
|
||
(referential/executable) before `register_generated`, registering only on a
|
||
clean preflight. → plan.
|
||
|
||
### [friction] `process`/`campaign` intrinsic validate needs the built project cdylib when inside a project
|
||
- Task 1 setup: inside the fresh `lab/` project, `process validate
|
||
c0110_1_explore_sweep.json` (and register/introspect) first failed with
|
||
`aura: cargo metadata failed: no cdylib target …`, then after `cargo build`
|
||
with `aura: project dylib not found … run cargo build`. The very same
|
||
`process validate` runs **dylib-free outside any project** (probe P1 →
|
||
`valid (intrinsic)`).
|
||
- Why friction: a std-only process document references no project nodes, so
|
||
the intrinsic (and referential-over-std) check needs no compiled dylib; the
|
||
CLI nonetheless loads the project env eagerly for any `process`/`campaign`
|
||
subcommand once an `Aura.toml` is found. A newcomer following the guide
|
||
inside a fresh project hits a build wall on a pure-shape check. (Likely
|
||
pre-existing, not introduced by 0110; the error is actionable.)
|
||
- Recommended action: let the intrinsic tier run before/without the dylib load
|
||
inside a project, or document that any in-project `process`/`campaign` call
|
||
requires a built cdylib. → plan or ratify (decide the intended coupling).
|
||
|
||
## Recommendation summary
|
||
| Finding | Class | Action |
|
||
|---|---|---|
|
||
| Selection-free process doc round-trips | working | carry-on |
|
||
| Dissolved sweep → deduped selection-free campaign | working | carry-on |
|
||
| All-or-nothing + terminal-only refusals clear | working | carry-on |
|
||
| Rank-later best-first, roster-enforced | working | carry-on |
|
||
| `aura sweep --axis` needs wrapped name, docs teach raw | spec_gap | tighten docs (+ plan if raw to be accepted) |
|
||
| Wrapped-name refusal mangles the typed axis | friction | plan |
|
||
| Guide §1 inline binds `length`, inconsistent with its ids/axes | spec_gap | tighten docs |
|
||
| `aura sweep <op-script>` leaks a serde error | friction | plan |
|
||
| Refused sweeps still register their generated campaign doc | friction | plan |
|
||
| In-project intrinsic validate needs the built cdylib | friction | plan / ratify |
|