Brummel
78806108d3
Downstream-consumer run (roles 5/6b, public interface only) over the document-authoring loop: canonicalization byte-exact and stable across reformatting (F8), intrinsic validation thorough and path-addressed (F9), referential tier correct across the project boundary with progressive resolution (F10). Findings: one bug (F2: introspect --block advertises an identity_id process ref the validator refuses), four frictions (F1 envelope not derivable from introspection incl. the block-id/field-name mismatch and the un-blocked seed; F3 content:-prefixed id vs bare-hash refs breaking referential resolution silently; F4 --unwired only reports omitted refs, which validate then fails; F5 strategy refs unresolvable from the public surface — no blueprint register/file-content-id/param-space verbs), one doc-gap (F6 no worked examples, metric vocabulary unlisted), and the spec-gap F7: a valid, registered campaign has no run path — the intent-persistence re-test the executor decision waited on. refs #189, #188
112 lines
6.6 KiB
Markdown
112 lines
6.6 KiB
Markdown
# Fieldtest — cycle 0106 — research-artifact document surface — 2026-07-03
|
|
|
|
**Role:** downstream consumer (methodology designer / campaign designer), public
|
|
interface only (ledger C18/C20/C22/C25, glossary, CLI help + introspection, the
|
|
fieldtests corpus shape). No `crates/` source read. Binary built from HEAD;
|
|
referential tier exercised after `cargo build` in the scaffolded `demo_project/`.
|
|
|
|
## Fixtures (this directory)
|
|
|
|
- `c0106_1_attempt1..3.json`, `c0106_1_partial.json`, `c0106_1_reformatted.json`
|
|
— process doc: 3-round envelope discovery + canonicalization + empty-gate probe
|
|
- `c0106_2_strategy.json` — SMA-cross op-script (content id `573262…`)
|
|
- `c0106_2_campaign_attempt1..2.json`, `c0106_2_campaign_valid.json`,
|
|
`c0106_2_campaign_partial.json`, `c0106_2_probe_{barestring,refnull,noref,prefixed}.json`
|
|
- `c0106_4_{emptyaxis,floatinint,noinstr,backwardwin,badmetric,badcmp,process_identity}.json`
|
|
- `demo_project/` — `aura new` scaffold + campaign/process docs + in-project store
|
|
- `runs/{processes,campaigns}/…` — content-addressed store written by `register`
|
|
|
|
## Findings
|
|
|
|
### F2 [bug] `introspect --block std::process_ref` advertises `identity_id`, which the validator refuses
|
|
The slot line prints `ref required, one of { content_id } | { identity_id }`
|
|
while validation refuses an identity_id process ref (`process.ref: a process is
|
|
referenced by content id in this version`, exit 1); the block header itself says
|
|
"(content id)". The 0106 audit fixed the parallel `--unwired` hint but not this
|
|
`--block` describe. Repro: `aura campaign introspect --block std::process_ref`.
|
|
→ debug (RED-pin content-id-only).
|
|
|
|
### F1 [friction] Document envelope not derivable from the introspection contract
|
|
Process doc took 3 blind rounds (`stages` → wrong-kind refusal that does not
|
|
name the `kind` key → `unknown field stages, expected one of … pipeline`).
|
|
Campaign worse: block ids `std::data/std::strategy/std::process_ref/std::presentation`
|
|
map to fields `data`/`strategies` (plural)/`process`/`presentation`, and the
|
|
required `seed` appears in NO block (`missing field seed`). Introspection
|
|
describes block slots but never the envelope composing them.
|
|
→ plan (envelope/skeleton introspection; reconcile block ids with field names).
|
|
|
|
### F3 [friction] content-id printed `content:`-prefixed, refs need the bare hash — silent referential break
|
|
`process|campaign introspect --content-id` print `content:<hash>`; `graph
|
|
introspect --content-id` prints bare. Pasting the prefixed form into a ref
|
|
PASSES intrinsic validation (different id!) then fails referential resolution
|
|
without the error naming the prefix as the cause.
|
|
→ plan (one display form across verbs; accept/strip the prefix in refs).
|
|
|
|
### F4 [friction] `--unwired` open-slot hints fire only when a ref is omitted, which `validate` fails
|
|
The ref open-slot hints appear only with the `ref` key removed (then validate
|
|
says `missing field ref`); placeholders `{}` / `null` give cryptic serde errors
|
|
(`expected map with a single key`; `invalid type: null, expected string or map`)
|
|
and `--unwired` reports `no open slots`. One draft file cannot both
|
|
validate-so-far and report its open slots.
|
|
→ plan (treat `ref: {}`/`null` as the reported open slot).
|
|
|
|
### F5 [friction] A campaign's strategy ref cannot be driven to resolve from the public surface
|
|
No verb computes a blueprint FILE's content id (`graph introspect --content-id`
|
|
is op-list-only), no verb registers a standalone blueprint into the store (only
|
|
sweep/mc/walkforward populate it), no verb lists a strategy's `param_space`
|
|
(axis names/kinds undiscoverable). The ref stays unresolved unless a sweep
|
|
already ran — but the sweep is what the campaign describes (see F7).
|
|
→ planner (blueprint register + file content-id + param-space listing; scope
|
|
with the executor).
|
|
|
|
### F6 [doc-gap] No worked document example anywhere; `metric_vocabulary` unlisted by any verb
|
|
Zero process/campaign JSON examples in docs/ or the corpus at test time. Block
|
|
hints say "metric name (see metric_vocabulary)" but no verb lists it, and
|
|
`unknown metric` does not enumerate valid names.
|
|
→ tighten docs (worked example per type; metric enumeration via introspect or
|
|
in the refusal).
|
|
|
|
### F7 [spec_gap] No executor — a valid campaign has no run path (the pending executor decision)
|
|
After registering a campaign whose process declares sweep → gate → walk_forward
|
|
over a strategy + 9-point axis, the expected `aura campaign run <id>` does not
|
|
exist; `aura run`/`sweep` take no campaign. The campaign+process pair is a
|
|
complete declarative spec of machinery the engine already runs
|
|
(sweep_family / optimize_deflated / walkforward_family); what exists is
|
|
persisted, content-addressed, legible intent that nothing consumes. The trail
|
|
ends at `runs/campaigns/<hash>.json`.
|
|
→ this run IS the intent-persistence re-test the executor decision waited on;
|
|
scope the executor.
|
|
|
|
### F8 [working] Content-id canonicalization is exact and stable
|
|
Reordered keys + reflowed whitespace + `0` vs `0.0` yield the identical content
|
|
id; stored form is compact, declared-key-order, no trailing newline, and
|
|
re-derives its id. → carry-on.
|
|
|
|
### F9 [working] Intrinsic validation is thorough, path-addressed, house-style
|
|
Empty gate, unknown metric/cmp, empty axis, float-in-I64, empty instruments,
|
|
backwards window, invalid-doc content-id refusal — all precise prose, exit
|
|
codes usage=2 / invalid=1 / ok=0. Minor: the kind-mismatch error omits the axis
|
|
path its siblings carry. → carry-on.
|
|
|
|
### F10 [working] Referential tier correct across the project boundary
|
|
Outside: skip line names the search root. Inside without dylib: actionable
|
|
build hint. Inside built: per-store diagnostics, progressive resolution as
|
|
documents get registered. → carry-on.
|
|
|
|
## Verdict
|
|
|
|
The shipped surface delivers standalone value: methodology and experiment
|
|
intent are nameable, diffable (byte-exact content id surviving reformatting),
|
|
and auditable (content-addressed, referentially checkable, honest
|
|
diagnostics) — with no compile and no run. The shortfall concentrates in the
|
|
authoring on-ramp (envelope invisible to introspection, no worked example, the
|
|
`content:`-vs-bare id trap) and the terminal wall: the campaign+process pair
|
|
fully specifies a sweep→gate→walk-forward the engine already runs, yet nothing
|
|
consumes it. The single most valuable next capability is an executor
|
|
(`aura campaign run <id>` driving the existing family machinery) — the one
|
|
addition that turns persisted intent into a result and retroactively gives
|
|
every id, store, axis, and gate its reason to exist.
|
|
|
|
Counts: bug 1 (F2), friction 4 (F1/F3/F4/F5), doc-gap 1 (F6), spec-gap 1 (F7),
|
|
working 3 (F8/F9/F10).
|