Files
Aura/fieldtests/milestone-cost-model-graph/FINDINGS.md
T
Brummel dff99c93e4 fieldtest(milestone): cost-model graph (#148) — capability delivered green
Milestone-close gate for "Cost-model graph (in R)" (#148). The fieldtester exercised
the milestone capability end-to-end from the public CLI only (no implementation
source), on the stage1-r harness, against ledger C10 + the glossary.

Roll-up: friction_found — 0 capability bugs; the milestone promise is empirically
delivered:
- bare gross-R run is clean (net == gross, no cost forced);
- per-trade cost: gross pinned, net drops strictly/monotonically, net_r_equity tap
  persists;
- the capstone per-held-cycle carry BLEEDS the net-R curve continuously over a hold
  (the gross-net gap grows 0.367 -> 0.734 -> 1.101 -> 1.468 across one hold, vs the
  per-trade run's flat 0.367) — the holding-cost shape change, observable end-to-end;
- composition is exact-additive (the composed drag equals the sum of the single-cost
  drags to the last ULP) and `aura chart` renders the real net_r_equity series.

Evidence under fieldtests/milestone-cost-model-graph/ (run_scenarios.sh +
captured_outputs.txt + traces/ + the charted bleed + FINDINGS.md).

Non-blocking findings filed forward (none gates the close):
- cost-flag CLI ergonomics — units hint, negative-rate diagnostic, non-R-harness
  no-op + a C10 spec_gap -> #153;
- docs/project-layout.md describes the retired `aura backtest --broker` path and omits
  the cost flags -> folded into #151.

refs #148
2026-06-29 02:35:24 +02:00

211 lines
12 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Fieldtest — milestone #148 "Cost-model graph (in R)" — 2026-06-29
**Status:** Draft — awaiting orchestrator triage
**Author:** fieldtester (dispatched by fieldtest skill, milestone-close gate)
## Scope
Milestone #148 promises a composable **cost-model graph in R**: a strategy's
gross-R signal is turned into a **net-R** curve by composing cost nodes, each a
labelled stress parameter, on the `stage1-r` harness. The capability must span
**both** charge modes — per-trade costs (deduct at close) and per-held-cycle
costs (accrue over the hold) — so a research user can measure how much friction
costs their edge and see whether a holding cost changes the **shape** of the
equity curve. Shipped across cycles 00810085: `ConstantCost`, the `net_r_equity`
tap + `summarize_r` cost fold, `VolSlippageCost` + `CostSum`, the `CostNode`
trait, the `cost_graph` composite-builder, and per-held-cycle accrual
(`CarryCost` + `ChargeMode`). Public surface exercised: the `aura` CLI
(`run --harness stage1-r` with `--cost-per-trade` / `--slip-vol-mult` /
`--carry-per-cycle`, `--trace`, `chart --tap`), against the design ledger C10 and
the glossary. Binary exercised: `target/debug/aura` freshly built from HEAD
(`cargo build --workspace`, commit 84ec2ab).
The milestone promise is **delivered**: all four end-to-end scenarios behave as
the contract states, including the capstone holding-cost shape change and exact
cost composition. No capability bug. The findings are doc/diagnostic friction at
the public surface and one undefined-behaviour spec_gap; none blocks the close.
## Examples
All under `fieldtests/milestone-cost-model-graph/`. `run_scenarios.sh` reproduces
every command; `captured_outputs.txt` is its recorded output; `traces/` holds the
`r_equity` / `net_r_equity` series the runs persisted (the `runs/` store is
gitignored); `chart_carry_net_r_equity.html` is the charted capstone bleed.
### run_scenarios.sh §1 — bare gross-R, no cost model
- `aura run --harness stage1-r` with no cost flag.
- Fits scope: the zero-cost floor — C10's "a bare gross-R run with no cost model
attached is valid", the additive-on baseline every cost composes onto.
- Outcome: ran; `expectancy_r` = `net_expectancy_r` = `1.2710005136982836`
exactly (net == gross when no cost is attached). Clean.
### run_scenarios.sh §2 — flat per-trade cost (ConstantCost, deduct at close)
- `aura run --harness stage1-r --cost-per-trade {0.0001,0.001,0.01}`, plus a
`--trace pertrade_cost` run.
- Fits scope: the per-trade charge mode (cycle 0081), "how much does friction
cost my edge?".
- Outcome: ran; gross `expectancy_r` stays pinned at `1.2710005136982836`
while `net_expectancy_r` drops strictly and monotonically with cost
(`1.2402``0.9632``-1.8069`); the `net_r_equity` tap persists under
`--trace`. Matched expected.
### run_scenarios.sh §3 — per-held-cycle carry (CarryCost, accrue over hold) — capstone
- `aura run --harness stage1-r --carry-per-cycle 0.001 --trace carry_cost`.
- Fits scope: the milestone capstone (cycle 0085) — the per-held-cycle charge
mode; "does a holding cost change the SHAPE of the curve?".
- Outcome: ran; `net_expectancy_r` = `-0.2679604926492904`. The `net_r_equity`
gap (gross net) **grows continuously over a single hold** — ts4..ts7:
`0.367 → 0.734 → 1.101 → 1.468` — versus the per-trade run's **flat** `0.367`
over the same hold. The holding cost bleeds the curve continuously rather than
stepping at close. Matched expected exactly.
### run_scenarios.sh §4 — compose all three costs + chart
- `aura run --harness stage1-r --cost-per-trade 0.001 --slip-vol-mult 0.5
--carry-per-cycle 0.001 --trace composed_cost`, each single cost alone for
contrast, then `aura chart composed/carry --tap net_r_equity`.
- Fits scope: the cost graph **composes** (cycles 0082/0084) — N cost nodes sum
into one net-R curve via `CostSum`/`cost_graph`.
- Outcome: ran; composed `net_expectancy_r` = `-1.2927895081922123`, strictly
below every single cost alone (`0.5540` slip, `0.9632` per-trade, `-0.2680`
carry). The composed net-R drag (`gross composed = 2.5637900218904959`)
equals the **exact** sum of the three single-cost drags
(`0.30779220 + 0.71703681 + 1.53896101 = 2.56379002`, agreeing to the last
ULP). The chart HTML bakes the real `net_r_equity` points array in (uPlot), so
the bleed renders — not a shell. Matched expected.
## Findings
### [working] Gross-only run is clean: net == gross with no cost
- run_scenarios.sh §1.
- `net_expectancy_r` is byte-equal to `expectancy_r` (`1.2710005136982836`) and
no `net_r_equity` is forced; the cost layer is genuinely optional and additive
(C10 "the zero-cost baseline is the default-simple floor").
- Why working: the optional-cost contract holds exactly; the win to protect is
that attaching no cost changes nothing.
- Recommended action: carry-on.
### [working] Per-trade cost: gross pinned, net drops monotonically, tap persists
- run_scenarios.sh §2.
- Across three magnitudes gross `expectancy_r` never moves; `net_expectancy_r`
is strictly below it and monotone decreasing; `runs/traces/pertrade_cost/`
contains a `net_r_equity.json` sibling of `r_equity.json`.
- Why working: the per-trade charge mode + the gross/net split + the named tap
all behave as C10 / the glossary describe, first try.
- Recommended action: carry-on.
### [working] Capstone — holding cost bleeds the curve continuously over the hold
- run_scenarios.sh §3 (+ `traces/carry_cost_net_r_equity.json`).
- The grossnet gap during the cycles-4..7 hold grows linearly
(`0.367, 0.734, 1.101, 1.468`), where the per-trade run's gap over the same
hold is flat (`0.367, 0.367, 0.367, 0.367`). The per-held-cycle accrual makes
the net-R equity curve bleed continuously across a hold instead of stepping at
close — the milestone's headline "shape change", directly observable in the
persisted series.
- Why working: this is the milestone capstone (cycle 0085, approach B) and it is
empirically true end-to-end from the CLI, not merely asserted in a unit test.
- Recommended action: carry-on.
### [working] Composition is exact-additive and the chart renders the bleed
- run_scenarios.sh §4 (+ `chart_carry_net_r_equity.html`,
`traces/composed_cost_*.json`).
- Three heterogeneous cost factors (per-trade + state-dependent slippage +
per-held-cycle carry) sum into one net-R curve; the composed drag equals the
exact sum of the single-cost drags (to the last ULP) and composed net is
strictly below every single. `aura chart` emits a self-contained HTML page with
the actual `net_r_equity` points array embedded.
- Why working: `CostSum` / `cost_graph` "one home for cost, no double-count"
(C10) is exact, the flags compose as the usage string implies, and the headline
artifact charts as a real result render (matching the "render results, not a
shell" rule).
- Recommended action: carry-on.
### [friction] Negative cost-rate guard prints the bare usage string, names no cause
- `aura run --harness stage1-r --carry-per-cycle -0.001` (and `--cost-per-trade
-0.5`).
- Verbatim: `aura: usage: aura run [--harness <sma|macd|stage1-r>] ...` then
`exit=2`. The guard correctly refuses a negative rate (capability-safe, per the
commit-noted "negative-rate guard"), but the diagnostic is the entire 1-line
usage dump — it never says *why* (e.g. "cost rate must be ≥ 0"), and the usage
string itself never states the non-negativity constraint, so a user who
fat-fingers a sign cannot tell a rejected value from a mistyped flag name.
- Why friction: the task is refused correctly but the surface gives an
uninformative diagnostic that names no cause.
- Recommended action: plan (a tidy iteration: emit `cost rate must be
non-negative, got <v>` instead of the generic usage dump).
### [friction] Cost flags silently no-op on a non-R harness (sma / macd)
- `aura run --harness sma --cost-per-trade 0.001` → `exit=0`, output
byte-identical to the no-cost `sma` run (`"total_pips":-0.13000000000001358`,
no `r` block, no `net_*`, no warning).
- Why friction: a cost flag attached to a harness that has no R-evaluator /
executor chain is silently swallowed; a user who believes they measured a
cost-adjusted result got an unmodified one with no signal that the flag did
nothing. (Distinct from the spec_gap below: this entry is the missing-feedback
foot-gun, regardless of which behaviour is chosen.)
- Recommended action: plan (warn-on-ignored-cost-flag, or reject the
combination).
### [spec_gap] Cost-flag-on-non-R-harness behaviour is undefined by the ledger
- Same repro as the friction above (`--cost-per-trade` on `--harness sma`).
- C10 / the glossary define cost only against the R chain (gross R → net R) and
say a bare gross-R run with no cost is valid; neither constrains what a cost
flag should do on a harness (`sma` / `macd`) that produces *no* R at all. The
project picked the "silently accept and ignore" reading; "reject: cost requires
an R-evaluator harness" was equally plausible. I did not have to guess to
complete the milestone scenarios (they all use `stage1-r`), but a downstream
user mixing flags hits an unspecified corner.
- Why spec_gap: the ledger is silent on the case and the project chose one of two
equally-plausible readings.
- Recommended action: ratify (pick error-vs-noop and add a one-line C10 note;
pairs with the friction fix above).
### [friction] Cost-flag units / scale are undiscoverable at the CLI
- All scenarios. The usage string offers only `[--cost-per-trade <f64>]
[--slip-vol-mult <f64>] [--carry-per-cycle <f64>]` — no unit, no
per-close-vs-per-held-cycle note, no scale guidance. I had to trial
`0.0001 / 0.001 / 0.01` to find a magnitude that moved net-R sensibly, because
the R-normalizer (the stop distance `|entry stop|`, the denominator of
`cost-in-R = cost_per_trade / |entry stop|`) is not surfaced anywhere a CLI
user can see, and `--carry-per-cycle`'s unit (price units, **per engine cycle**
— not per day / not an overnight swap) is invisible. The semantics *are* in
C10 and the glossary "cost model" entry, so the capability is discoverable in
the ledger — just not at the point of use.
- Why friction: the surface forced a trial-and-error dance to choose a sensible
stress-parameter value that a one-line units hint would have removed.
- Recommended action: plan / docwriter (a per-flag units note in the usage
string: "price units, charged in R as cost/|entrystop|; --carry-per-cycle is
per engine cycle").
### [friction] The public "how aura is used" doc describes the retired pre-reframe path
- `docs/project-layout.md` (the only public usage doc) — its "day in the life"
still says `aura backtest … --broker pepperstone` yields a "realistic currency
curve" alongside an "exposure stream" from a "realistic broker". All three —
`aura backtest`, the `--broker` realistic broker, the exposure framing — are
**retired** by the 2026-06-28 C10 reframe; the actual surface is
`aura run --harness stage1-r --cost-per-trade …` producing net-R, which the doc
never mentions. A downstream user reading it to learn how to charge cost is
pointed at a path that no longer exists and never told about the cost flags.
- Why friction: a public-surface document that actively misleads about the
shipped capability (doc rot, not a ledger gap — C10 itself is current).
- Recommended action: plan / docwriter (refresh `project-layout.md` to the
post-reframe `aura run` + cost-flag surface).
## Recommendation summary
| Finding | Class | Action |
|---|---|---|
| Gross-only run clean (net == gross) | working | carry-on |
| Per-trade: gross pinned, net monotone, tap persists | working | carry-on |
| Capstone: holding cost bleeds over the hold | working | carry-on |
| Composition exact-additive + chart renders | working | carry-on |
| Negative-rate guard prints bare usage, no cause | friction | plan |
| Cost flags silently no-op on sma/macd | friction | plan |
| Cost-flag-on-non-R-harness undefined by ledger | spec_gap | ratify |
| Cost-flag units/scale undiscoverable at CLI | friction | plan / docwriter |
| project-layout.md describes the retired path | friction | plan / docwriter |
**Milestone-close roll-up:** `friction_found`. The capability is delivered
end-to-end (both charge modes, composition, the holding-cost shape change); no
capability bug blocks the close. All findings are doc / diagnostic friction or an
undefined-corner spec_gap — non-gating for the milestone close.