af0191884d
Public-API field test of the random param-sweep surface, from a standalone downstream-consumer crate (path-deps only; the public interface = ledger + glossary + spec 0049 + cargo doc rustdoc; no crates/*/src read). Four bins, each built from HEAD and run: continuous tuning (200-point random tune ranked by total_pips), the typed validation gate (all five reachable SweepError variants pre-run), reproducibility + seed-sensitivity + the full i64::MIN..=MAX sampler edge, and Space-trait interchangeability (one tune_and_rank<S: Space> over both GridSpace and RandomSpace). Findings: 0 bugs, 4 working, 2 spec_gap, 1 friction. The four working findings confirm the cycle's acceptance criterion empirically — the headline tune reads as the code a researcher would write, the gate is precise and fires before any run, the C1 reproducibility promise is checkable in one line, and the Space trait delivers one-consumer/both-enumerations. Triage of the actionable findings: - friction (no named-axis builder for RandomSpace — positional Vec<ParamRange> must align with param_space() by hand, and a same-kind transposition passes validation silently): filed as a feature for a future cycle, refs #79 (a RandomBinder sibling to the grid's SweepBinder). - spec_gap (SweepError rustdoc summary named only GridSpace): fixed inline in a follow-up doc commit. - spec_gap (NonNumericRange / Bool-slot ranges unreachable with the shipped aura-std node roster — no node declares a Bool/Timestamp knob): RATIFIED as intentional. The variant is a forward-looking structural guard for the C16 "author your own node" path (a Bool/Timestamp param-slot a custom node may declare); its current untriggerability with the standard roster is expected, not drift. refs #79
184 lines
11 KiB
Markdown
184 lines
11 KiB
Markdown
# Fieldtest — cycle-0049 (random param-sweep) — 2026-06-17
|
|
|
|
**Status:** Draft — awaiting orchestrator triage
|
|
**Author:** fieldtester (dispatched by fieldtest skill)
|
|
|
|
## Scope
|
|
|
|
Cycle 0049 shipped the **random** half of the C12.1 param-sweep axis in
|
|
`aura_engine`. New public surface: `RandomSpace` (draws `N` seeded uniform
|
|
points over per-slot continuous ranges, validated against a blueprint's
|
|
param-space before any run), a typed `ParamRange` (`{lo, hi}` with
|
|
`ParamRange::i64` / `ParamRange::f64` constructors and `kind()`), a `Space`
|
|
trait (`points()` / `param_specs()`) that both `GridSpace` and `RandomSpace`
|
|
implement so `sweep` / `sweep_with_threads` are generic over `impl Space`, and
|
|
three new `SweepError` variants (`NonNumericRange`, `RangeKindMismatch`,
|
|
`EmptyRange`). The per-point run closure is unchanged (`Fn(&[Cell]) ->
|
|
RunReport`). This field test exercised the surface from a standalone downstream
|
|
consumer crate (path-deps only; public interface = ledger + glossary + spec
|
|
0049 + `cargo doc` rustdoc; no `crates/*/src` read). Binary exercised: each bin
|
|
rebuilt from HEAD via `cargo run --manifest-path …` after a clean
|
|
`cargo build --workspace`.
|
|
|
|
## Examples
|
|
|
|
### fieldtests/cycle-0049-random-sweep/c0049_1_continuous_tune.rs — headline continuous tuning
|
|
- Builds a 3-param SMA-cross harness (`sma_cross.fast.length: I64`,
|
|
`sma_cross.slow.length: I64`, `exposure.scale: F64`), declares one
|
|
`ParamRange` per slot, draws 200 seeded points via `RandomSpace::new`, runs
|
|
`sweep`, and picks the best point by `total_pips`.
|
|
- Fits the cycle scope: this is the named "audience reaches for it" task — a
|
|
grid over continuous ranges explodes; random draw is the standard tool.
|
|
- Outcome: built (first try after a self-inflicted port-name slip), ran,
|
|
matched expected. 200 in-range points, 92 distinct metrics, best #121 =
|
|
6.3216 pips at fast=8/slow=12/scale=1.3182.
|
|
|
|
### fieldtests/cycle-0049-random-sweep/c0049_2_validation_gate.rs — typed validation gate
|
|
- Declares invalid ranges and records the typed `SweepError` from
|
|
`RandomSpace::new` before any run: `Arity`, `RangeKindMismatch` (F64 range on
|
|
I64 slot), `EmptyRange` (I64 `lo>hi`), `EmptyRange` (F64 `lo>=hi`), the valid
|
|
I64 `lo==hi` single point (accepted), and `NonNumericRange` (range on a Bool
|
|
slot). The Bool slot required authoring a tiny custom node (`GateNode`),
|
|
because no shipped `aura-std` node declares a Bool/Timestamp param.
|
|
- Fits the cycle scope: directly probes axis-hint 2 (the typed gate).
|
|
- Outcome: built, ran, matched expected. Every fault caught pre-run with exact
|
|
slot + kind; `NonNumericRange` wins over `RangeKindMismatch` on the Bool slot.
|
|
|
|
### fieldtests/cycle-0049-random-sweep/c0049_3_reproducibility.rs — reproducibility + seed-sensitivity + wide-range edge
|
|
- Compares whole `SweepFamily`s (`derive(PartialEq)`): same `(ranges, count,
|
|
seed)` reproduces an identical family; a different seed changes it. Then
|
|
probes the full `ParamRange::i64(i64::MIN, i64::MAX)` range through
|
|
`Space::points()` to check the commit-body hardening claim (the literal
|
|
sampler would `% 0`).
|
|
- Fits the cycle scope: axis-hint 3 (the C1 promise) plus the documented
|
|
sampler deviation.
|
|
- Outcome: built, ran, matched expected. Same-seed family identical;
|
|
different-seed family differs; the full i64-domain range was accepted and
|
|
`points()` produced 8 values spanning the signed domain with no panic.
|
|
|
|
### fieldtests/cycle-0049-random-sweep/c0049_4_space_interchange.rs — Space-trait interchangeability
|
|
- One downstream `tune_and_rank<S: Space>(space: &S)` consumer, written once and
|
|
called with both a `GridSpace` and a `RandomSpace` over the same param-space,
|
|
same `run_one` closure, asserting both produce comparable, non-degenerate
|
|
families carrying the same schema.
|
|
- Fits the cycle scope: axis-hint 4 (the abstraction's payoff).
|
|
- Outcome: built, ran, matched expected. Both 12-point families ranked through
|
|
the identical generic consumer; shared `family.space`; named view identical.
|
|
|
|
## Findings
|
|
|
|
### [working] Headline continuous-range tuning is the code a researcher would write
|
|
- Example: c0049_1.
|
|
- The whole flow — `param_space()` → one `ParamRange` per slot →
|
|
`RandomSpace::new(&space, ranges, count, seed)?` → `sweep(&rand_space,
|
|
run_one)` → rank by metric → `family.named_params(best)` for a readable
|
|
coordinate — read naturally and compiled/ran on the first real attempt. All
|
|
200 draws landed inside their declared ranges (I64 inclusive, F64 half-open),
|
|
and `bootstrap_with_cells(point)` paired cleanly with the `Fn(&[Cell]) ->
|
|
RunReport` closure (no `Scalar`/`Cell` conversion dance in the hot loop). The
|
|
family spread (92 distinct metrics over 200 points) and a single argmax pass
|
|
picked the best. This is the cycle's primary acceptance criterion met
|
|
empirically.
|
|
- Recommended downstream action: carry-on.
|
|
|
|
### [working] The typed validation gate is precise and fires before any run
|
|
- Example: c0049_2.
|
|
- Every reachable fault surfaced from `RandomSpace::new` with an exact,
|
|
self-describing variant: `Arity { expected: 3, got: 2 }`, `RangeKindMismatch
|
|
{ slot: 0, expected: I64, got: F64 }`, `EmptyRange { slot: 0 }` (I64 lo>hi)
|
|
and `{ slot: 2 }` (F64 lo>=hi), and `NonNumericRange { slot: 0, kind: Bool }`.
|
|
The valid I64 `lo==hi` single point was accepted (7 points). The ordering is
|
|
sensible: a non-numeric slot is reported as `NonNumericRange` even when the
|
|
supplied range's kind also mismatches, so the author sees the structural cause
|
|
(wrong slot kind) rather than a downstream symptom.
|
|
- Recommended downstream action: carry-on.
|
|
|
|
### [working] Reproducibility, seed-sensitivity, and the wide-range i64 hardening all hold
|
|
- Example: c0049_3.
|
|
- `SweepFamily`'s `derive(PartialEq)` made the C1 promise checkable in one line:
|
|
`family(seed) == family(seed)` held end to end (manifest + metrics included),
|
|
and `family(seed_a) != family(seed_b)`. The full `[i64::MIN, i64::MAX]` range
|
|
— the case the commit body (e17d78f) flags as a spec deviation hardened
|
|
against `% 0` / signed-overflow — was accepted by `new` and produced 8 finite
|
|
values across the signed domain through the public `Space::points()`, no
|
|
panic. The documented deviation is observably upheld at the public surface.
|
|
- Recommended downstream action: carry-on.
|
|
|
|
### [working] The Space trait delivers its payoff: one consumer, both enumerations
|
|
- Example: c0049_4.
|
|
- A single generic `tune_and_rank<S: Space>` drove a `GridSpace` and a
|
|
`RandomSpace` with zero per-enumeration branching, the same `run_one`, and a
|
|
shared `family.space` schema for the named view. The grid path stayed
|
|
behaviour-preserving (the existing `GridSpace::new` / `points()` /
|
|
`param_specs()` inherent methods are still there alongside the trait impl).
|
|
- Recommended downstream action: carry-on.
|
|
|
|
### [spec_gap] `NonNumericRange` / Bool-slot ranges are unreachable with the shipped node roster
|
|
- Example: c0049_2 (the GateNode workaround).
|
|
- Every shipped `aura-std` node declares only `I64` (`length`) or `F64`
|
|
(`scale`) params; none has a `Bool` or `Timestamp` knob. So a downstream
|
|
consumer using only the standard roster can never hit the `NonNumericRange`
|
|
gate (nor a Bool `RangeKindMismatch`) — I had to author a custom
|
|
`PrimitiveBuilder`-based node with a Bool param purely to construct a Bool
|
|
slot. The variant is correct and the C16 "author your own node" path makes it
|
|
reachable, but the cycle ships a validation arm that no shipped surface can
|
|
trigger. The ledger/spec do not say whether a Bool knob is an expected
|
|
near-term authoring case or a purely defensive guard. Plausible alternate
|
|
reading: `NonNumericRange` is dead-until-a-Bool-knob-node-exists and could
|
|
have been deferred with the rest of the random axis. I picked the reading "it
|
|
is a defensive structural guard for future Bool/Timestamp knobs" and the
|
|
example demonstrates it works under that reading.
|
|
- Recommended downstream action: ratify (record in the ledger that the
|
|
Bool/Timestamp param-slot is a forward-looking authoring case the gate guards,
|
|
so the otherwise-untriggerable variant is intentional, not drift).
|
|
|
|
### [friction] `RandomSpace` has no named-axis builder; positional `ranges` must line up with `param_space()` by hand
|
|
- Examples: c0049_1, c0049_3, c0049_4.
|
|
- `GridSpace` has a fluent, by-name sibling: `Composite::axis("sma_cross.fast",
|
|
…).sweep(run)` (the `SweepBinder`), which resolves axes against
|
|
`param_space()` by name so a mis-ordered or mis-counted axis is impossible.
|
|
`RandomSpace` has no counterpart: the author must build `Vec<ParamRange>` in
|
|
exactly `param_space()` slot order and pass it positionally to
|
|
`RandomSpace::new(&space, ranges, …)`. The `Arity` / `RangeKindMismatch`
|
|
checks catch a wrong count or a kind-swapped slot, but a same-kind transposition
|
|
(e.g. swapping the two I64 ranges for `fast.length` and `slow.length`) passes
|
|
validation silently and tunes the wrong knob over the wrong interval — the
|
|
exact failure class the named `SweepBinder` was built to remove for the grid
|
|
axis (cf. node memory: the mw_1 ParamAlias pain, fixed by naming). The task
|
|
completes, but the author re-takes on the positional-alignment burden that the
|
|
grid path already retired.
|
|
- Recommended downstream action: plan (a `RandomSpace` named-axis builder — a
|
|
`RandomBinder` sibling to `SweepBinder` mapping `name -> ParamRange` against
|
|
`param_space()`, so the by-name resolution that protects the grid sweep also
|
|
protects the random sweep).
|
|
|
|
### [spec_gap] `SweepError`'s rustdoc summary names only GridSpace though it now gates both spaces
|
|
- Examples: c0049_2 (the error type the consumer reads).
|
|
- The `SweepError` enum's top-level rustdoc (the public doc a consumer reads to
|
|
understand the type) still reads: "A structural fault constructing a
|
|
**`GridSpace`** — the typed gate before any run." The enum now also carries
|
|
the three RandomSpace-only variants (`NonNumericRange`, `RangeKindMismatch`,
|
|
`EmptyRange`), each documented individually as RandomSpace faults, but the
|
|
type-level summary is stale: a consumer scanning the type doc would not learn
|
|
that this same error type gates `RandomSpace::new`. The audit commit (3fca781)
|
|
noted refreshing the *module* doc to name `RandomSpace` + the `Space` trait,
|
|
but the `SweepError` type doc itself was not updated. Another equally-plausible
|
|
reading is that `SweepError` is deliberately the single shared sweep-error type
|
|
and the summary should say so; I cannot tell which from the public surface.
|
|
- Recommended downstream action: tighten the design ledger / docs (one-line
|
|
doc-debt fix: broaden the `SweepError` summary to "constructing a `GridSpace`
|
|
*or* `RandomSpace`" — doc-only, behaviour-preserving, the same low-grade
|
|
doc-debt class the audit already fixed inline for the module doc).
|
|
|
|
## Recommendation summary
|
|
|
|
| Finding | Class | Action |
|
|
|---|---|---|
|
|
| Headline continuous tuning works | working | carry-on |
|
|
| Typed validation gate is precise | working | carry-on |
|
|
| Reproducibility + seed-sensitivity + wide-i64 hold | working | carry-on |
|
|
| Space-trait interchangeability payoff | working | carry-on |
|
|
| NonNumericRange unreachable with shipped roster | spec_gap | ratify |
|
|
| No named-axis builder for RandomSpace | friction | plan |
|
|
| SweepError doc summary names only GridSpace | spec_gap | tighten the design ledger / docs |
|