Files
Aura/docs/specs/fieldtest-0049-random-sweep.md
T
Brummel af0191884d fieldtest: cycle-0049 — 4 examples, 7 findings
Public-API field test of the random param-sweep surface, from a standalone
downstream-consumer crate (path-deps only; the public interface = ledger +
glossary + spec 0049 + cargo doc rustdoc; no crates/*/src read). Four bins,
each built from HEAD and run: continuous tuning (200-point random tune ranked
by total_pips), the typed validation gate (all five reachable SweepError
variants pre-run), reproducibility + seed-sensitivity + the full i64::MIN..=MAX
sampler edge, and Space-trait interchangeability (one tune_and_rank<S: Space>
over both GridSpace and RandomSpace).

Findings: 0 bugs, 4 working, 2 spec_gap, 1 friction. The four working findings
confirm the cycle's acceptance criterion empirically — the headline tune reads
as the code a researcher would write, the gate is precise and fires before any
run, the C1 reproducibility promise is checkable in one line, and the Space
trait delivers one-consumer/both-enumerations.

Triage of the actionable findings:
- friction (no named-axis builder for RandomSpace — positional Vec<ParamRange>
  must align with param_space() by hand, and a same-kind transposition passes
  validation silently): filed as a feature for a future cycle, refs #79
  (a RandomBinder sibling to the grid's SweepBinder).
- spec_gap (SweepError rustdoc summary named only GridSpace): fixed inline in a
  follow-up doc commit.
- spec_gap (NonNumericRange / Bool-slot ranges unreachable with the shipped
  aura-std node roster — no node declares a Bool/Timestamp knob): RATIFIED as
  intentional. The variant is a forward-looking structural guard for the C16
  "author your own node" path (a Bool/Timestamp param-slot a custom node may
  declare); its current untriggerability with the standard roster is expected,
  not drift.

refs #79
2026-06-17 13:42:39 +02:00

184 lines
11 KiB
Markdown

# Fieldtest — cycle-0049 (random param-sweep) — 2026-06-17
**Status:** Draft — awaiting orchestrator triage
**Author:** fieldtester (dispatched by fieldtest skill)
## Scope
Cycle 0049 shipped the **random** half of the C12.1 param-sweep axis in
`aura_engine`. New public surface: `RandomSpace` (draws `N` seeded uniform
points over per-slot continuous ranges, validated against a blueprint's
param-space before any run), a typed `ParamRange` (`{lo, hi}` with
`ParamRange::i64` / `ParamRange::f64` constructors and `kind()`), a `Space`
trait (`points()` / `param_specs()`) that both `GridSpace` and `RandomSpace`
implement so `sweep` / `sweep_with_threads` are generic over `impl Space`, and
three new `SweepError` variants (`NonNumericRange`, `RangeKindMismatch`,
`EmptyRange`). The per-point run closure is unchanged (`Fn(&[Cell]) ->
RunReport`). This field test exercised the surface from a standalone downstream
consumer crate (path-deps only; public interface = ledger + glossary + spec
0049 + `cargo doc` rustdoc; no `crates/*/src` read). Binary exercised: each bin
rebuilt from HEAD via `cargo run --manifest-path …` after a clean
`cargo build --workspace`.
## Examples
### fieldtests/cycle-0049-random-sweep/c0049_1_continuous_tune.rs — headline continuous tuning
- Builds a 3-param SMA-cross harness (`sma_cross.fast.length: I64`,
`sma_cross.slow.length: I64`, `exposure.scale: F64`), declares one
`ParamRange` per slot, draws 200 seeded points via `RandomSpace::new`, runs
`sweep`, and picks the best point by `total_pips`.
- Fits the cycle scope: this is the named "audience reaches for it" task — a
grid over continuous ranges explodes; random draw is the standard tool.
- Outcome: built (first try after a self-inflicted port-name slip), ran,
matched expected. 200 in-range points, 92 distinct metrics, best #121 =
6.3216 pips at fast=8/slow=12/scale=1.3182.
### fieldtests/cycle-0049-random-sweep/c0049_2_validation_gate.rs — typed validation gate
- Declares invalid ranges and records the typed `SweepError` from
`RandomSpace::new` before any run: `Arity`, `RangeKindMismatch` (F64 range on
I64 slot), `EmptyRange` (I64 `lo>hi`), `EmptyRange` (F64 `lo>=hi`), the valid
I64 `lo==hi` single point (accepted), and `NonNumericRange` (range on a Bool
slot). The Bool slot required authoring a tiny custom node (`GateNode`),
because no shipped `aura-std` node declares a Bool/Timestamp param.
- Fits the cycle scope: directly probes axis-hint 2 (the typed gate).
- Outcome: built, ran, matched expected. Every fault caught pre-run with exact
slot + kind; `NonNumericRange` wins over `RangeKindMismatch` on the Bool slot.
### fieldtests/cycle-0049-random-sweep/c0049_3_reproducibility.rs — reproducibility + seed-sensitivity + wide-range edge
- Compares whole `SweepFamily`s (`derive(PartialEq)`): same `(ranges, count,
seed)` reproduces an identical family; a different seed changes it. Then
probes the full `ParamRange::i64(i64::MIN, i64::MAX)` range through
`Space::points()` to check the commit-body hardening claim (the literal
sampler would `% 0`).
- Fits the cycle scope: axis-hint 3 (the C1 promise) plus the documented
sampler deviation.
- Outcome: built, ran, matched expected. Same-seed family identical;
different-seed family differs; the full i64-domain range was accepted and
`points()` produced 8 values spanning the signed domain with no panic.
### fieldtests/cycle-0049-random-sweep/c0049_4_space_interchange.rs — Space-trait interchangeability
- One downstream `tune_and_rank<S: Space>(space: &S)` consumer, written once and
called with both a `GridSpace` and a `RandomSpace` over the same param-space,
same `run_one` closure, asserting both produce comparable, non-degenerate
families carrying the same schema.
- Fits the cycle scope: axis-hint 4 (the abstraction's payoff).
- Outcome: built, ran, matched expected. Both 12-point families ranked through
the identical generic consumer; shared `family.space`; named view identical.
## Findings
### [working] Headline continuous-range tuning is the code a researcher would write
- Example: c0049_1.
- The whole flow — `param_space()` → one `ParamRange` per slot →
`RandomSpace::new(&space, ranges, count, seed)?` → `sweep(&rand_space,
run_one)` → rank by metric → `family.named_params(best)` for a readable
coordinate — read naturally and compiled/ran on the first real attempt. All
200 draws landed inside their declared ranges (I64 inclusive, F64 half-open),
and `bootstrap_with_cells(point)` paired cleanly with the `Fn(&[Cell]) ->
RunReport` closure (no `Scalar`/`Cell` conversion dance in the hot loop). The
family spread (92 distinct metrics over 200 points) and a single argmax pass
picked the best. This is the cycle's primary acceptance criterion met
empirically.
- Recommended downstream action: carry-on.
### [working] The typed validation gate is precise and fires before any run
- Example: c0049_2.
- Every reachable fault surfaced from `RandomSpace::new` with an exact,
self-describing variant: `Arity { expected: 3, got: 2 }`, `RangeKindMismatch
{ slot: 0, expected: I64, got: F64 }`, `EmptyRange { slot: 0 }` (I64 lo>hi)
and `{ slot: 2 }` (F64 lo>=hi), and `NonNumericRange { slot: 0, kind: Bool }`.
The valid I64 `lo==hi` single point was accepted (7 points). The ordering is
sensible: a non-numeric slot is reported as `NonNumericRange` even when the
supplied range's kind also mismatches, so the author sees the structural cause
(wrong slot kind) rather than a downstream symptom.
- Recommended downstream action: carry-on.
### [working] Reproducibility, seed-sensitivity, and the wide-range i64 hardening all hold
- Example: c0049_3.
- `SweepFamily`'s `derive(PartialEq)` made the C1 promise checkable in one line:
`family(seed) == family(seed)` held end to end (manifest + metrics included),
and `family(seed_a) != family(seed_b)`. The full `[i64::MIN, i64::MAX]` range
— the case the commit body (e17d78f) flags as a spec deviation hardened
against `% 0` / signed-overflow — was accepted by `new` and produced 8 finite
values across the signed domain through the public `Space::points()`, no
panic. The documented deviation is observably upheld at the public surface.
- Recommended downstream action: carry-on.
### [working] The Space trait delivers its payoff: one consumer, both enumerations
- Example: c0049_4.
- A single generic `tune_and_rank<S: Space>` drove a `GridSpace` and a
`RandomSpace` with zero per-enumeration branching, the same `run_one`, and a
shared `family.space` schema for the named view. The grid path stayed
behaviour-preserving (the existing `GridSpace::new` / `points()` /
`param_specs()` inherent methods are still there alongside the trait impl).
- Recommended downstream action: carry-on.
### [spec_gap] `NonNumericRange` / Bool-slot ranges are unreachable with the shipped node roster
- Example: c0049_2 (the GateNode workaround).
- Every shipped `aura-std` node declares only `I64` (`length`) or `F64`
(`scale`) params; none has a `Bool` or `Timestamp` knob. So a downstream
consumer using only the standard roster can never hit the `NonNumericRange`
gate (nor a Bool `RangeKindMismatch`) — I had to author a custom
`PrimitiveBuilder`-based node with a Bool param purely to construct a Bool
slot. The variant is correct and the C16 "author your own node" path makes it
reachable, but the cycle ships a validation arm that no shipped surface can
trigger. The ledger/spec do not say whether a Bool knob is an expected
near-term authoring case or a purely defensive guard. Plausible alternate
reading: `NonNumericRange` is dead-until-a-Bool-knob-node-exists and could
have been deferred with the rest of the random axis. I picked the reading "it
is a defensive structural guard for future Bool/Timestamp knobs" and the
example demonstrates it works under that reading.
- Recommended downstream action: ratify (record in the ledger that the
Bool/Timestamp param-slot is a forward-looking authoring case the gate guards,
so the otherwise-untriggerable variant is intentional, not drift).
### [friction] `RandomSpace` has no named-axis builder; positional `ranges` must line up with `param_space()` by hand
- Examples: c0049_1, c0049_3, c0049_4.
- `GridSpace` has a fluent, by-name sibling: `Composite::axis("sma_cross.fast",
…).sweep(run)` (the `SweepBinder`), which resolves axes against
`param_space()` by name so a mis-ordered or mis-counted axis is impossible.
`RandomSpace` has no counterpart: the author must build `Vec<ParamRange>` in
exactly `param_space()` slot order and pass it positionally to
`RandomSpace::new(&space, ranges, …)`. The `Arity` / `RangeKindMismatch`
checks catch a wrong count or a kind-swapped slot, but a same-kind transposition
(e.g. swapping the two I64 ranges for `fast.length` and `slow.length`) passes
validation silently and tunes the wrong knob over the wrong interval — the
exact failure class the named `SweepBinder` was built to remove for the grid
axis (cf. node memory: the mw_1 ParamAlias pain, fixed by naming). The task
completes, but the author re-takes on the positional-alignment burden that the
grid path already retired.
- Recommended downstream action: plan (a `RandomSpace` named-axis builder — a
`RandomBinder` sibling to `SweepBinder` mapping `name -> ParamRange` against
`param_space()`, so the by-name resolution that protects the grid sweep also
protects the random sweep).
### [spec_gap] `SweepError`'s rustdoc summary names only GridSpace though it now gates both spaces
- Examples: c0049_2 (the error type the consumer reads).
- The `SweepError` enum's top-level rustdoc (the public doc a consumer reads to
understand the type) still reads: "A structural fault constructing a
**`GridSpace`** — the typed gate before any run." The enum now also carries
the three RandomSpace-only variants (`NonNumericRange`, `RangeKindMismatch`,
`EmptyRange`), each documented individually as RandomSpace faults, but the
type-level summary is stale: a consumer scanning the type doc would not learn
that this same error type gates `RandomSpace::new`. The audit commit (3fca781)
noted refreshing the *module* doc to name `RandomSpace` + the `Space` trait,
but the `SweepError` type doc itself was not updated. Another equally-plausible
reading is that `SweepError` is deliberately the single shared sweep-error type
and the summary should say so; I cannot tell which from the public surface.
- Recommended downstream action: tighten the design ledger / docs (one-line
doc-debt fix: broaden the `SweepError` summary to "constructing a `GridSpace`
*or* `RandomSpace`" — doc-only, behaviour-preserving, the same low-grade
doc-debt class the audit already fixed inline for the module doc).
## Recommendation summary
| Finding | Class | Action |
|---|---|---|
| Headline continuous tuning works | working | carry-on |
| Typed validation gate is precise | working | carry-on |
| Reproducibility + seed-sensitivity + wide-i64 hold | working | carry-on |
| Space-trait interchangeability payoff | working | carry-on |
| NonNumericRange unreachable with shipped roster | spec_gap | ratify |
| No named-axis builder for RandomSpace | friction | plan |
| SweepError doc summary names only GridSpace | spec_gap | tighten the design ledger / docs |