fieldtest: cycle-0108 annotator-stages corpus (GREEN)
Verdict: honest, recorded, reproducible inference on top of the 0107 loop — pooled_oos provably pools the wf OOS windows (n_trades 948+895 = 1843), per_survivor flags a razor-thin gate-passer at prob_le_zero 0.525 (the inference layer earning its keep), the two-instrument std::generalize records worst_case/sign_agreement over last-wf-window nominees, byte-identical across three runs modulo the run counter, and all eight boundary probes refuse loudly at the right tier. 0 bugs. Triage: F1-F5 carry-on; F6 (campaign validate blesses executor-unrunnable shapes — the third tier invisible) and F11 (no CLI read-back for recorded realizations) filed as issues; F7/F9/F10 doc-tightens follow this commit inline; F8 (generalize R-metric set undiscoverable + misleading refusal prose for R-denominated-but- unranked metrics) filed with the --metrics tag half. The fieldtest spec rides in docs/specs/ as next-cycle planner input per convention. refs #200
This commit is contained in:
@@ -0,0 +1,133 @@
|
||||
# Fieldtest — cycle 0108 (annotator stages, #200) — 2026-07-04
|
||||
|
||||
**Status:** Draft — awaiting orchestrator triage
|
||||
**Author:** fieldtester (dispatched by fieldtest skill)
|
||||
|
||||
## Scope
|
||||
Cycle 0108 makes the two terminal annotator stages execute on top of the 0107
|
||||
campaign loop. The v2 pipeline shape is
|
||||
`std::sweep [std::gate]* [std::walk_forward]? [std::monte_carlo]? [std::generalize]?`.
|
||||
`std::monte_carlo` bootstraps the stage's incoming R-evidence with
|
||||
position-dependent semantics (PooledOos over the walk_forward OOS series, else
|
||||
PerSurvivor over each gate survivor's in-sample series); `std::generalize` runs
|
||||
at campaign scope over per-cell nominees per (strategy, window), scoring the
|
||||
cross-instrument worst-case R floor. Both are terminal (nothing flows out). The
|
||||
realization records widen serde-default-sparse (`StageRealization.bootstrap`,
|
||||
`CampaignRunRecord.generalizations`). A static preflight admits the shape and
|
||||
refuses the eight off-shape/param/metric/instrument faults.
|
||||
|
||||
## Examples
|
||||
### fieldtests/cycle-0108-annotator-stages/c0108_campaign_full_pooled.json — full-suffix pooled campaign
|
||||
- sweep(deflate) → gate → walk_forward → monte_carlo over GER40 2024-09.
|
||||
- Fits scope: exercises the PooledOos mc path — the headline new surface.
|
||||
- Outcome: built ✓, ran ✓ (exit 0), matched expected ✓ (`pooled_oos` bootstrap,
|
||||
`n_trades:1843` = the two wf OOS windows 948+895 pooled).
|
||||
|
||||
### fieldtests/cycle-0108-annotator-stages/c0108_campaign_mc_persurvivor.json — no-wf per-survivor campaign
|
||||
- sweep → gate → monte_carlo over GER40 2024-09.
|
||||
- Fits scope: exercises the PerSurvivor mc path (the other input shape).
|
||||
- Outcome: built ✓, ran ✓ (exit 0), matched expected ✓ (`per_survivor` list of
|
||||
`[ordinal, bootstrap]` over survivors [0,1,2]; survivor 1 flagged
|
||||
`prob_le_zero:0.525`).
|
||||
|
||||
### fieldtests/cycle-0108-annotator-stages/c0108_campaign_2instr_generalize.json — end-to-end generalize
|
||||
- sweep → gate → wf → mc → generalize over GER40 + FRA40, 2024-09.
|
||||
- Fits scope: exercises campaign-scope `std::generalize` (needs ≥2 instruments).
|
||||
- Outcome: built ✓, ran ✓ (exit 0), matched expected ✓ (`generalizations` entry,
|
||||
`worst_case:0.0437`, `sign_agreement:2`, nominees = each cell's last wf-OOS
|
||||
winner; sparse widening present only on this line in the store).
|
||||
|
||||
### fieldtests/cycle-0108-annotator-stages/c0108_probe_*.json — annotator boundary probes
|
||||
- gate-after-mc, wf-after-mc, dup-mc, gen-not-last, zero-resamples, zero-blocklen,
|
||||
non-R-generalize, single-instrument-generalize (+ their campaign runners).
|
||||
- Fits scope: probes both refusal tiers (intrinsic vs executor preflight).
|
||||
- Outcome: built ✓, ran ✓, matched expected ✓ (all refuse; exit 1; one intrinsic,
|
||||
seven executor-tier — see F5).
|
||||
|
||||
## Findings
|
||||
### [working] F1 — full annotator suffix closes; pooled_oos over the wf OOS series
|
||||
- `c0108_campaign_full_pooled.json`. mc records `pooled_oos` with `n_trades:1843`
|
||||
provably = 948+895 (the two wf OOS windows pooled in roll order). Honest,
|
||||
recorded, seeded. Recommended action: carry-on.
|
||||
|
||||
### [working] F2 — no-wf campaign records one bootstrap per survivor; honesty seam bites
|
||||
- `c0108_campaign_mc_persurvivor.json`. `per_survivor` over gate survivors
|
||||
[0,1,2] each over the member's in-sample R series; survivor 1 passes the gate at
|
||||
`expectancy_r:0.0019` but the bootstrap flags `prob_le_zero:0.525`. Carry-on.
|
||||
|
||||
### [working] F3 — determinism is byte-identical modulo the run counter
|
||||
- Three runs of `c0108_campaign_full_pooled.json`; every bootstrap percentile /
|
||||
`prob_le_zero` / survivor set / selection reproduces. C1 intact. Carry-on.
|
||||
|
||||
### [working] F4 — campaign-scope generalize closes end-to-end over two instruments
|
||||
- `c0108_campaign_2instr_generalize.json`. `worst_case:0.0437`,
|
||||
`sign_agreement:2`, nominees = last wf-OOS winner per cell, divergent winners
|
||||
exposed via params; sparse `generalizations` key present only on this line.
|
||||
Carry-on.
|
||||
|
||||
### [working] F5 — the annotator tier boundary is honest and precise on both sides
|
||||
- All `c0108_probe_*` runners. Intrinsic (`process validate`) catches only
|
||||
gate-after-terminal; the executor preflight catches wf-after-mc, dup-mc,
|
||||
gen-not-last, zero resamples/block_len, non-R metric, single-instrument
|
||||
generalize — each with per-pair prose, exit 1. Carry-on.
|
||||
|
||||
### [friction] F6 — `campaign validate` blesses executor-refused process shapes
|
||||
- `c0108_probe_campaign_{wf_after_mc,dup_mc,gen_not_last}.json`: `campaign
|
||||
validate` returns exit 0, only `campaign run` refuses. wf-after-mc is a pure
|
||||
process-shape fault detectable without data. The three-tier model is invisible
|
||||
from validate. Recommended action: plan (run the executor shape/param preflight
|
||||
at validate time, or name the third tier in the validate output).
|
||||
|
||||
### [doc-gap] F7 — pooled-vs-per-survivor duality invisible to the author surface
|
||||
- `--block std::monte_carlo` says only "R-bootstrap over realised R". Nothing on
|
||||
the CLI/glossary surface tells the author the output shape+semantics flip on
|
||||
whether wf precedes mc (documented only in ledger C18 0108). Recommended
|
||||
action: tighten docs (fold position-dependent semantics into `--block` /
|
||||
glossary).
|
||||
|
||||
### [doc-gap] F8 — generalize R-metric set unpredictable; refusal prose misleads
|
||||
- Accepted: expectancy_r, net_expectancy_r, sqn, sqn_normalized. Refused:
|
||||
total_pips + max_drawdown (both rankable), win_rate, and `max_r_drawdown` —
|
||||
R-denominated yet refused as "not an R metric (… pip metrics do not)".
|
||||
`--metrics` has no R/generalizable tag. Fixture:
|
||||
`c0108_probe_generalize_maxrdrawdown.json`. Recommended action: tighten (add
|
||||
applicability tag to `--metrics`; correct refusal prose for R-but-unranked
|
||||
metrics).
|
||||
|
||||
### [doc-gap] F9 — mc slot hint says "non-negative" but the executor refuses 0
|
||||
- `--block std::monte_carlo` labels resamples/block_len "non-negative integer
|
||||
(u32)" (the type), but 0 is refused (`… must be > 0`). Fixture:
|
||||
`c0108_probe_campaign_zero_resamples.json`. Recommended action: tighten the
|
||||
slot hint to "positive integer".
|
||||
|
||||
### [doc-gap] F10 — the recorded realization output schema is undocumented
|
||||
- `prob_le_zero`, `per_survivor`, `pooled_oos`, `worst_case`, `sign_agreement`,
|
||||
`e_r` percentiles appear in no public prose (only internal enum names in the
|
||||
ledger). Stdout wraps as `{"campaign_run":…}` but the persisted JSONL is bare.
|
||||
Recommended action: tighten docs (define the realization fields; a
|
||||
glossary/introspect schema).
|
||||
|
||||
### [friction] F11 — annotator realizations are write-only (no CLI read-back)
|
||||
- `aura runs` exposes only families; no verb reads back a `campaign_run`. The
|
||||
bootstrap/generalization records live only in run stdout + a gitignored JSONL,
|
||||
one unbounded line per run. Recommended action: plan (an `aura campaign runs` /
|
||||
`run <id>` read-back verb).
|
||||
|
||||
## Recommendation summary
|
||||
| finding | class | action |
|
||||
|---|---|---|
|
||||
| F1 pooled_oos over wf OOS | working | carry-on |
|
||||
| F2 per_survivor + honesty seam | working | carry-on |
|
||||
| F3 determinism byte-identical | working | carry-on |
|
||||
| F4 campaign-scope generalize | working | carry-on |
|
||||
| F5 tier boundary precise | working | carry-on |
|
||||
| F6 validate blesses unrunnable shapes | friction | plan |
|
||||
| F7 pooled-vs-per-survivor undiscoverable | doc-gap | tighten the design ledger / introspection |
|
||||
| F8 generalize R-metric set + prose | doc-gap | tighten docs / prose |
|
||||
| F9 "non-negative" hint vs >0 guard | doc-gap | tighten introspection hint |
|
||||
| F10 realization output schema undocumented | doc-gap | tighten docs |
|
||||
| F11 no realization read-back | friction | plan |
|
||||
|
||||
**Verdict:** Yes — the annotator stages deliver honest, recorded, reproducible
|
||||
inference on the 0107 loop. No bugs. Residue is discoverability at the author's
|
||||
first contact with the two new stages (F6–F11).
|
||||
Reference in New Issue
Block a user