Brummel
13941a24d5
Milestone-close fieldtest for "Runway — real-data ergonomics & honesty hardening". Four curated end-to-end scenarios derived top-down from the milestone promise, run as a downstream consumer against real GER40/EURUSD M1 bars (public interface only; no crates/*/src read): 1. per-instrument pip honesty (#22) — GER40 index pip 1.0 AND EURUSD forex pip 0.0001 both honest end-to-end; AAPL.US/FOOBAR un-vetted refused exit 2, no stdout leak 2. real GER40 M1 open via open_ohlc from aura-ingest ALONE (#80/#81/#92) — no data-server dep, 53173 real bars, C4 O/H/L/C order, ts-window inverse 3. multi-tap ts-join (#93) — two taps at different cadences, 27744 rows, 0 by-ts mismatches; warm-up None rows where a positional zip would corrupt 4. run-registry coherence (#73/#82) — list/rank retired exit 2, families/ family live, two-store Registry read-back + rustdoc Findings: 4 working, 1 friction, 1 spec_gap. - friction: Cell exposes only from_i64/from_f64/... while the side-by-side Scalar is a tagged enum with public I64(..)/F64(..) variants. The natural param-authoring entry point is Scalar (Cell is the hot-path buffer carrier), so the wrong reach is a self-correcting compile snag — ergonomic only. Filed as a follow-up idea. - spec_gap: unknown family id exits 0 empty — ratified into the C18 note in the preceding commit (treat-as-empty contract). Gate verdict: GREEN. All seven axes mapped to a covering scenario and exercised on real bars; both finding classifications and the one material coverage gap were adversarially re-verified. That gap — #22's FX-vetted pip face, untested by the original scenario set — was probed (EURUSD -> sim-optimal(pip_size=0.0001), FX-scaled, not index-mis-scaled) and folded into scenario 1, so the pip promise is demonstrated on both positive faces. Every scenario re-run independently on HEAD before commit.
34 lines
2.4 KiB
Plaintext
34 lines
2.4 KiB
Plaintext
# Runway milestone fieldtest — scenario 1: per-instrument pip honesty (#22)
|
|
#
|
|
# The task: a downstream researcher runs a real backtest on a VETTED index and
|
|
# on an UN-VETTED instrument, and must get an honest pip on the first, a clean
|
|
# REFUSAL (non-zero exit, no silent mis-scaled output) on the second.
|
|
#
|
|
# Consumer form: the canonical CLI invocation itself (aura run --real).
|
|
# Built/run via 'cargo run' so HEAD is what runs.
|
|
|
|
$ aura run --real GER40 --from 1420070400000 --to 1422748800000
|
|
{"manifest":{"commit":"0d185fb4c3d2be3002760dbb9d9a8a2ceed795e0","params":[["sma_fast",{"I64":2}],["sma_slow",{"I64":4}],["exposure_scale",{"F64":0.5}]],"window":[1420156800000000000,1422648780000000000],"seed":0,"broker":"sim-optimal(pip_size=1)"},"metrics":{"total_pips":977.3799999994587,"max_drawdown":654.9300000000931,"exposure_sign_flips":8155}}
|
|
exit=0
|
|
|
|
# ^ VETTED INDEX: broker label reports sim-optimal(pip_size=1) — the correct
|
|
# index point, NOT the FX default 0.0001. Honest PnL.
|
|
|
|
$ aura run --real EURUSD --from 1420070400000 --to 1422748800000
|
|
{"manifest":{"commit":"0d185fb4c3d2be3002760dbb9d9a8a2ceed795e0","params":[["sma_fast",{"I64":2}],["sma_slow",{"I64":4}],["exposure_scale",{"F64":0.5}]],"window":[1420156800000000000,1422654780000000000],"seed":0,"broker":"sim-optimal(pip_size=0.0001)"},"metrics":{"total_pips":-0.8768415000001664,"max_drawdown":0.9156020000001746,"exposure_sign_flips":8503}}
|
|
exit=0
|
|
|
|
# ^ VETTED FX MAJOR: broker label reports sim-optimal(pip_size=0.0001) — the
|
|
# correct forex pip, NOT the index point 1. total_pips=-0.8768 is FX-scaled
|
|
# (sub-pip on this window), NOT a mis-scaled index number (~977 on GER40 above
|
|
# over the same window). The pip promise's SECOND positive face holds: index
|
|
# point AND forex pip both demonstrated end-to-end, each at its own scale.
|
|
|
|
$ aura run --real AAPL.US --from 1420070400000 --to 1422748800000 # un-vetted (instrument_spec -> None)
|
|
aura: no vetted pip/instrument spec for symbol 'AAPL.US' — refusing to run a real instrument with a guessed pip (add it to the instrument table)
|
|
exit=2 (stdout above is the stderr diagnostic only — no mis-scaled stdout leak)
|
|
|
|
$ aura run --real FOOBAR --from 1420070400000 --to 1422748800000 # un-vetted AND no archive
|
|
aura: no vetted pip/instrument spec for symbol 'FOOBAR' — refusing to run a real instrument with a guessed pip (add it to the instrument table)
|
|
exit=2 (pip-refusal fires at the lookup, BEFORE any data access)
|