fieldtest: milestone the-world param-space & sweep — green, delivers promise
Milestone-scope fieldtest (closing functional gate) for "The World —
parameter-space & sweep" (issues #30-#36). Three curated end-to-end scenarios
derived top-down from the milestone promise, driven from the public interface
only (ledger + specs + rustdoc; no crates/*/src read), built and run from HEAD
via a downstream consumer crate.
Delivery verdict: GREEN. The milestone delivers its promise.
- mw_1 single run bound by name: .with("sma_cross.fast", 2).bootstrap() binds
and runs (12 equity rows); diagnostics precise and knob-named (UnknownKnob /
AmbiguousKnob / KindMismatch).
- mw_2 named-axis sweep + compare: .axis(..).sweep(run) yields a 4-member,
ordered (odometer), disjoint, non-constant SweepFamily; ranking from public
fields. CLI (aura sweep / aura runs rank) delivers the same with no Rust.
- mw_3 structural-constant negative space (deliberate): SMA2-entry's
definitional `2` has no honest expression today -> grounds #55. Out of
milestone scope; not a blocker.
Findings: 0 bugs, 2 working, 3 friction, 1 spec_gap. Verified independently
(re-ran all three from HEAD; confirmed a plain SMA-cross emits sma_cross.length
twice and the shipped CLI sample carries the fast/slow ParamAlias overlays).
Friction (routed to the forward queue, none blocks the gate):
- the promised sma_cross.fast/.slow names are not free — they require
author-added ParamAlias overlays, doubly mandatory because the canonical
2-SMA cross also will not bootstrap without them (IndistinguishableFanIn);
- the sweep closure drops back to positional bind + manual name re-zip.
spec_gap -> #55: the wished-for structural-constant consumer code is recorded
in mw_3 as #55's acceptance evidence.
This commit is the milestone fieldtest itself; it closes no issue. Closing the
tracker milestone stays a deliberate manual act on this green gate.
This commit is contained in:
@@ -0,0 +1,262 @@
|
||||
# Fieldtest — milestone "The World — parameter-space & sweep" — 2026-06-11
|
||||
|
||||
**Status:** Draft — awaiting orchestrator triage
|
||||
**Author:** fieldtester (dispatched by fieldtest skill, milestone-scope variant)
|
||||
**Scope mode:** MILESTONE (closing functional gate), not per-cycle.
|
||||
|
||||
## Milestone delivery roll-up
|
||||
|
||||
The milestone promises a native-Rust strategy author can **declare** typed knobs,
|
||||
**bind** a value-empty blueprint into one runnable instance **by name**, **spread**
|
||||
it across a **family** over named axes, and **index/compare** the family
|
||||
(issues #30–#36). Per the three curated end-to-end scenarios:
|
||||
|
||||
| Scenario | Promise clause | Delivered? |
|
||||
|---|---|---|
|
||||
| 1 — single run, bound **by name** | declare + bind by name (`.with(...).bootstrap()`) | 🟢 GREEN — **with one caveat**: the promised names `sma_cross.fast/.slow` are not free; they require author-added C23 `ParamAlias` overlays. Once aliased, `.with("sma_cross.fast", 2).bootstrap()` binds and runs; diagnostics are clean. |
|
||||
| 2 — **sweep** over named axes + **compare** the family | spread + index/compare (`.axis(...).sweep(run)`, #32/#33) | 🟢 GREEN — `.axis("sma_cross.fast", [2,3]).sweep(run)` yields a 4-member, ordered, disjoint, non-constant `SweepFamily`; comparison/ranking assembled from the public `SweepPoint`/`RunReport`/`RunMetrics` fields. The `aura sweep` + `aura runs rank` CLI delivers the same payoff with no Rust. |
|
||||
| 3 — structural-constant negative space | (deliberately **out of milestone scope**; grounds #55) | 🔴 RED **by design** — the definitional `2` of SMA2-entry can only be expressed as an open `param_space()` knob (avenue a) or via an escape hatch reachable only for `SimBroker`, not indicators (avenue b). This is the forward-pointer to #55, **not** a milestone-blocking defect. |
|
||||
|
||||
**Verdict (one line):** the milestone's promise is **empirically delivered** —
|
||||
scenarios 1 and 2 are green end-to-end (the by-name caveat is a friction, not a
|
||||
gap in the capability) — with scenario 3 logged as the empirical grounding for
|
||||
#55 (structural-constant binding), which the milestone never promised.
|
||||
|
||||
## Scope
|
||||
|
||||
The milestone spans cycles 0015→0030 (param declaration #30, binding #31, sweep
|
||||
#32, run registry #33, named binding #35). It promises that an author can declare
|
||||
a node's tunable knobs (typed), aggregate them into a flat, path-qualified
|
||||
`param_space()`, bind one runnable instance **by name** (`.with()/.bootstrap()`),
|
||||
spread a family over **named axes** (`.axis()/.sweep()`), and index/compare the
|
||||
resulting `SweepFamily`. The deeper promise (C11/C12/C19): a blueprint is
|
||||
param-generic; binding sizes nodes but never changes topology; family members are
|
||||
C1-disjoint; `param_space()` is the single ground truth both single-run binding
|
||||
and sweep enumeration address.
|
||||
|
||||
This fieldtest derives three end-to-end scenarios top-down from that promise and
|
||||
drives them from the **public interface only**: the design ledger
|
||||
(`docs/design/INDEX.md`), the specs (0015/0028/0030), and the crates' published
|
||||
rustdoc (`cargo doc --workspace --no-deps`). **No `crates/*/src` was read** — the
|
||||
current API (incl. the 0024 deletion of `struct Blueprint`, the root-is-a-Composite
|
||||
model, the `Sma::factory` → `Sma::builder` rename, and the `Role.source` field)
|
||||
was discovered entirely from the ledger + rustdoc + the compiler's own
|
||||
diagnostics. The artefact under test was built from HEAD (`cargo build
|
||||
--workspace`; the consumer crate path-deps the engine crates; fixtures run via
|
||||
`cargo run`, so HEAD is always what executed).
|
||||
|
||||
## Examples
|
||||
|
||||
### fieldtests/milestone-the-world-param-sweep/mw_1_single_named.rs — single concrete run, bound by name
|
||||
- Authors the SMA-cross harness as a root `Composite` (0024 model: a Composite
|
||||
all of whose roles are source-bound), inspects `param_space()`, and tries to
|
||||
bind by the milestone-promise names `sma_cross.fast/.slow`. Records the
|
||||
diagnostic, then the actual-emitted name, then the aliased fix, then five
|
||||
diagnostic probes.
|
||||
- Fits the milestone: this is the headline by-name binding story (#30/#31/#35) —
|
||||
declare, inspect, bind one instance by name, run it.
|
||||
- Outcome: built? **yes.** ran? yes. matched expected? **partially** — the
|
||||
promise-form bind **fails** (`UnknownKnob("sma_cross.fast")`); the real names
|
||||
collide (`AmbiguousKnob("sma_cross.length")`); after adding `ParamAlias`
|
||||
overlays the promise-form bind **works** and runs (12 non-degenerate equity
|
||||
rows). Diagnostics clean.
|
||||
|
||||
### fieldtests/milestone-the-world-param-sweep/mw_2_sweep_compare.rs — sweep over named axes + compare the family
|
||||
- Takes the (aliased) harness, spreads it over `.axis("sma_cross.fast", [2,3])
|
||||
.axis("sma_cross.slow", [4,5]).axis("scale", [0.5]).sweep(run)`, then builds a
|
||||
comparison table + best-by-`total_pips` ranking from the public `SweepFamily`
|
||||
fields, asserts the family is ordered (odometer), disjoint, and non-constant.
|
||||
- Fits the milestone: the "N instead of 1" core (#32) + the index/compare surface
|
||||
(#33).
|
||||
- Outcome: built? yes. ran? yes. matched expected? **yes** — 4 members, 4
|
||||
distinct `total_pips`, odometer order `[2,4]→[2,5]→[3,4]→[3,5]`, ranking works.
|
||||
|
||||
### fieldtests/milestone-the-world-param-sweep/mw_3_structural_constant.rs — negative-space probe (grounds #55)
|
||||
- Authors the "SMA2-entry" strategy and tries to express the definitional `2`
|
||||
honestly via both avenues the carrier names: (a) leave it as an open
|
||||
`param_space()` knob fixed to 2; (b) imitate `SimBroker::builder(constant)`.
|
||||
Records both as wrong, plus the wished-for `.bind()`/`builder_with()` consumer
|
||||
code (commented out — it does not compile).
|
||||
- Fits the milestone: the deliberate negative space — what the param-space layer
|
||||
**cannot** express, the acceptance evidence for #55.
|
||||
- Outcome: built? yes. ran? yes (avenue a runs; avenue b is a documented compile
|
||||
wall). matched expected? **yes** — both avenues are recordably wrong; the `2`
|
||||
is fungible (a) or the escape hatch is unreachable for an indicator (b).
|
||||
|
||||
### Run artefacts
|
||||
- `out_mw_1.txt` / `out_mw_2.txt` / `out_mw_3.txt` — verbatim stdout of each
|
||||
fixture. Plus the `aura sweep` + `aura runs rank total_pips` CLI output
|
||||
(recorded inline in findings), the no-Rust face of the same family payoff.
|
||||
|
||||
## Findings
|
||||
|
||||
### [working] The named-axis sweep delivers a comparable, ordered, disjoint family from the public surface
|
||||
- Example: mw_2; corroborated by `aura sweep` / `aura runs rank`.
|
||||
- What happened: `.axis(name, vals)*.sweep(run)` resolved the named axes to a
|
||||
4-point grid, ran each disjointly, and returned a `SweepFamily` whose
|
||||
`points[i].params` (slot-order coordinate) and `points[i].report.metrics`
|
||||
(`total_pips`/`max_drawdown`/`exposure_sign_flips`) let a consumer build a
|
||||
comparison table and a best-by-metric ranking with **no scaffolding beyond a
|
||||
for-loop**. Order is odometer (last axis fastest), the 4 metrics are distinct
|
||||
(not a constant family), every `total_pips` finite. The `aura sweep` CLI emits
|
||||
the same family as JSON lines and `aura runs rank total_pips` orders them
|
||||
best-first across invocations.
|
||||
- Why working: this is the milestone's core "N instead of 1" + index/compare
|
||||
promise (#32/#33), delivered end-to-end from the public interface.
|
||||
- Recommended action: carry-on.
|
||||
|
||||
### [working] Single-run binding by name + clean, knob-named diagnostics (once knobs have distinct names)
|
||||
- Example: mw_1 (aliased path) + probes 3/4/5/6.
|
||||
- What happened: `.with("sma_cross.fast", 2).with("sma_cross.slow", 4)
|
||||
.with("scale", 0.5).bootstrap()` binds and runs (12-row trace). The diagnostics
|
||||
are precise and name the offending knob: a typo →
|
||||
`UnknownKnob("sma_cross.slo")`; an unqualified short name →
|
||||
`UnknownKnob("fast")`; a wrong kind → `KindMismatch { knob: "scale", expected:
|
||||
F64, got: I64 }`; the un-aliased collision → `AmbiguousKnob("sma_cross.length")`.
|
||||
Raw literals (`2`, `0.5`) lower without suffixes.
|
||||
- Why working: spec 0030's by-name binding + the total error order land exactly
|
||||
as designed; the error vocabulary is a strict win over a positional vector.
|
||||
- Recommended action: carry-on.
|
||||
|
||||
### [friction] The promised knob names `sma_cross.fast/.slow` do NOT exist out of the box — they require hand-added `ParamAlias` overlays
|
||||
- Examples: mw_1 [2a]/[2b]/[2c]; the engine's own `aura sweep` sample (whose
|
||||
manifest shows `sma_cross.fast`/`.slow`, so even the shipped sample had to add
|
||||
the aliases).
|
||||
- What happened: the milestone promise and the spec 0030 worked example both
|
||||
bind `.with("sma_cross.fast", 2)`. Run verbatim against a plain SMA-cross
|
||||
composite, `param_space()` emits **`sma_cross.length` twice** (the raw `Sma`
|
||||
factory param name, path-qualified only by the composite name), so
|
||||
`.with("sma_cross.fast", 2)` → `UnknownKnob("sma_cross.fast")` and binding the
|
||||
real name `.with("sma_cross.length", 2)` → `AmbiguousKnob("sma_cross.length")`
|
||||
(two identical names). To get the promised distinct names the author must
|
||||
author two `ParamAlias { name: "fast"/"slow", node, slot }` overlays on the
|
||||
composite. Only then does `param_space()` emit `sma_cross.fast`/`sma_cross.slow`
|
||||
and the promise-form bind succeed.
|
||||
- Why friction (not bug): the binding layer is correct — it faithfully matches
|
||||
the exact `param_space()` name, and spec 0030 §Architecture explicitly says
|
||||
short/distinct names are an authoring choice resolved by `ParamAlias`. But the
|
||||
milestone's headline example reads as if `fast`/`slow` are free, and a new
|
||||
author following it verbatim hits two consecutive errors before discovering the
|
||||
required alias step. The acceptance evidence (the code a user writes) is
|
||||
longer than advertised. The task **completes**, but only after the author
|
||||
supplies the naming overlay the example silently assumed.
|
||||
- Recommended action: plan (tidy / docs) — either (i) make the worked example in
|
||||
the ledger/spec show the `ParamAlias` step explicitly (so `sma_cross.fast` is
|
||||
visibly *earned*), or (ii) give `Sma::builder()` a name argument / a
|
||||
per-instance recipe label so two sibling SMAs carry distinct factory names
|
||||
without a separate alias. Low-to-medium; it is the first thing a by-name author
|
||||
trips on.
|
||||
|
||||
### [friction] A fan-in of two same-signature un-aliased knobs cannot bootstrap — aliasing is mandatory, not optional, for the canonical SMA-cross
|
||||
- Examples: mw_3 (initial bootstrap → `IndistinguishableFanIn { node: 2 }`);
|
||||
mw_1's plain harness is only ever *inspected*, never bootstrapped, for the same
|
||||
reason.
|
||||
- What happened: `Sub` fanning in two `Sma`s with equal recursive signatures —
|
||||
each carrying an unaliased param slot — is a compile-time
|
||||
`CompileError`/`BootstrapError::...IndistinguishableFanIn` (the C9 fan-in
|
||||
distinguishability refinement). So the **most canonical** strategy in the whole
|
||||
project (a 2-leg SMA cross) does not bootstrap until the author aliases its two
|
||||
length knobs. The same `ParamAlias` overlays demanded by the friction above are
|
||||
thus doubly mandatory: once to bind by distinct names, once to clear the fan-in
|
||||
check.
|
||||
- Why friction (not bug): the `IndistinguishableFanIn` rejection is a designed,
|
||||
documented C9 guarantee (it catches a hidden unnamed configuration axis), and
|
||||
the diagnostic names the node. But it converts "author a 2-SMA cross and bind
|
||||
it" — the project's running example — into a four-step dance (build, alias to
|
||||
satisfy fan-in, alias to get distinct names — the same two aliases, then bind).
|
||||
The friction is that the milestone's own canonical example cannot be run in its
|
||||
minimal form.
|
||||
- Recommended action: plan — pair with the friction above; both are dissolved by
|
||||
per-instance SMA naming at construction (a recipe label), which would satisfy
|
||||
fan-in distinguishability AND yield distinct param-space names in one step.
|
||||
|
||||
### [friction] The sweep closure must re-bind positionally and re-zip param names the author already declared
|
||||
- Example: mw_2 (the `.sweep(|point| { ... bootstrap_with_params(point.to_vec())
|
||||
... })` closure; the empty `manifest.params`).
|
||||
- What happened: `.axis(...).sweep(run)` resolves named axes to a positional
|
||||
point, but the per-point closure receives a bare `&[Scalar]` and must call
|
||||
`bootstrap_with_params(point.to_vec())` (positional) and rebuild the blueprint
|
||||
by hand. To put readable knob names into the per-point `RunManifest.params`,
|
||||
the author must re-zip `param_space()` names onto the point themselves — the
|
||||
named layer that resolved the axes does not thread the names through to the
|
||||
closure. (The `aura sweep` CLI does this zip by hand internally.)
|
||||
- Why friction: the named-binding ergonomics stop at the grid boundary; inside
|
||||
the sweep the author is back to positional vectors and manual name-zipping,
|
||||
re-deriving information `param_space()` already holds. The comparison in mw_2
|
||||
works, but only because the consumer re-zips names onto `pt.params` for display.
|
||||
- Recommended action: plan — consider passing the closure a named view of the
|
||||
point (e.g. `&[(/* name */ &str, Scalar)]` or a small `Point` with `get(name)`),
|
||||
or have `SweepPoint` carry the param names so a consumer need not re-zip from
|
||||
`param_space()`. Low-to-medium.
|
||||
|
||||
### [spec_gap] The definitional structural constant (SMA2-entry's `2`) has no honest expression — grounds #55
|
||||
- Example: mw_3 avenues (a) and (b).
|
||||
- What happened: the SMA2-entry strategy's `2` is bound to the two-candle
|
||||
construction (sweeping it deforms the strategy). Today only two avenues exist,
|
||||
both wrong:
|
||||
- **(a)** leave the length open: `param_space()` emits `sma2_entry.bias : I64`
|
||||
(forced to alias it just to clear `IndistinguishableFanIn`), **indistinguishable
|
||||
from the genuinely-tunable `sma2_entry.slow`**. A sweep
|
||||
`.axis("sma2_entry.bias", [2,5,10])` would enumerate **deformed** strategies
|
||||
as valid family members. `param_space()` declares a degree of freedom the
|
||||
strategy's definition forbids (C20 says structural axes are not sweep params —
|
||||
but nothing here distinguishes the two).
|
||||
- **(b)** imitate `SimBroker::builder(1e-4)` (a constant captured **outside**
|
||||
`param_space`). Verbatim wall: `Sma::builder()` takes **no** length arg
|
||||
(length is always an injected param), and `Sma::new(2)` returns a **built**
|
||||
`Sma`, not a `PrimitiveBuilder`/`BlueprintNode` — and the only value-conversion
|
||||
is `From<PrimitiveBuilder> for BlueprintNode` (confirmed from rustdoc; there
|
||||
is **no** `From<Sma>`). A blueprint holds value-empty recipes, never built
|
||||
nodes. The construction-constant escape hatch exists for `SimBroker` only
|
||||
because its `builder(pip_size)` **takes** the constant; there is no
|
||||
`Sma::builder_with(2)`. So the escape hatch is **structurally unreachable**
|
||||
for a stock indicator node.
|
||||
- Why spec_gap (not a milestone bug): the milestone **never promised**
|
||||
structural-constant binding — the spec 0030 §Goal explicitly puts it out of
|
||||
scope ("structural blueprint constants … is the opposite operation and lives in
|
||||
#55"). This finding is the **empirical grounding** for #55: the wished-for
|
||||
consumer code (a `Sma::builder().bind("length", 2)` that removes the knob from
|
||||
`param_space`, or a `Sma::builder_with(2)` recipe constructor) is recorded
|
||||
verbatim in mw_3 as the acceptance evidence a future #55 spec needs.
|
||||
- Recommended action: ratify the deferral as correct (the milestone gate is not
|
||||
blocked) **and** route the recorded wished-for code to #55 as its acceptance
|
||||
evidence. The `.bind()` name is already reserved for exactly this (spec 0030).
|
||||
|
||||
## Recommendation summary
|
||||
|
||||
| Finding | Class | Action | Routing |
|
||||
|---|---|---|---|
|
||||
| Named-axis sweep yields a comparable/ordered/disjoint family | working | carry-on | — |
|
||||
| Single-run by-name binding + clean knob-named diagnostics | working | carry-on | — |
|
||||
| Promised `sma_cross.fast/.slow` need hand-added `ParamAlias` | friction | plan (tidy/docs) | #30/#35 follow-up or doc-fix |
|
||||
| Canonical SMA-cross can't bootstrap without alias (`IndistinguishableFanIn`) | friction | plan | pairs with the alias-naming friction |
|
||||
| Sweep closure re-binds positionally + re-zips names | friction | plan | #32/#33 ergonomics follow-up |
|
||||
| SMA2-entry's definitional `2` has no honest expression | spec_gap | ratify deferral + route evidence | **#55** |
|
||||
|
||||
## Milestone roll-up
|
||||
|
||||
**The milestone "The World — parameter-space & sweep" DELIVERS its promise.** A
|
||||
native-Rust author can declare typed knobs, inspect them via a flat
|
||||
path-qualified `param_space()`, bind one runnable instance **by name**
|
||||
(`.with(...).bootstrap()`) with a precise, knob-named error vocabulary, spread the
|
||||
blueprint across a **named-axis family** (`.axis(...).sweep(run)`), and
|
||||
index/compare the disjoint, ordered, non-constant `SweepFamily` — all from the
|
||||
public interface, with the `aura sweep` + `aura runs rank` CLI as the no-Rust
|
||||
face of the same payoff. Scenarios 1 and 2 (the actual close-gate) are green
|
||||
end-to-end.
|
||||
|
||||
The frictions are about **the naming ergonomics on the way in**, not the
|
||||
capability: the milestone's headline by-name example (`sma_cross.fast/.slow`) is
|
||||
not free — those distinct names require author-added `ParamAlias` overlays, which
|
||||
are **doubly mandatory** because the canonical 2-SMA cross also will not bootstrap
|
||||
without them (`IndistinguishableFanIn`). A by-name author following the worked
|
||||
example verbatim hits two consecutive errors before discovering the alias step.
|
||||
All are tidy follow-ups, none block the gate.
|
||||
|
||||
Scenario 3 is the deliberate negative space: the definitional structural constant
|
||||
of SMA2-entry has no honest expression today (it leaks into `param_space()` as a
|
||||
fungible knob, and the `SimBroker` construction-constant escape hatch is
|
||||
unreachable for an indicator). This is **exactly** the gap #55 is filed for — the
|
||||
milestone never promised it — and the wished-for consumer code is now recorded as
|
||||
#55's empirical acceptance evidence. **No bugs found.** Two `working`, three
|
||||
`friction`, one `spec_gap` (→ #55) — none blocking the milestone gate.
|
||||
Reference in New Issue
Block a user