Brummel
90c1de841d
Public-API research deep-dive driving the just-shipped GER40 real-data
session-breakout demo (5b5f034) through the escalating sequence a serious
researcher attempts next: scale to multi-year, walk-forward, structural-param
sweep, multi-instrument compare. Standalone consumer crate (path-deps only;
public interface = ledger + glossary + example corpus + cargo doc; no
crates/*/src read). All runs against the real /mnt/tickdata/Pepperstone archive.
Findings: 3 bugs, 1 friction, 3 spec_gap, 1 working. Triaged to the tracker:
- spec_gap: the flagship breakout ships as a raw FlatGraph the World API cannot
consume (sweep / walk_forward / compare all forced a hand re-author to a
Composite, which reproduced the FlatGraph numbers EXACTLY) — refs #94.
- bug: process residency is O(window) (6->173 MiB across 1mo->10y) despite the
documented O(one-chunk) claim; the aura ring stays <=1024 but data-server
retention scales, and streaming_seam.rs only measures the ring — refs #95.
- bug: the bar-period is one knob split across Resample (tunable param) and
Session (baked) and desyncs silently to 0.0 pips; Delay.lag leaks into the
default param_space (the C34 deform-vs-tune discriminator) — refs #96.
- spec_gap: the walk_forward empty-space idiom for param-free strategies is
undefined — refs #97.
- spec_gap (no per-instrument pip registry) and friction (one-directional
ns->ms conversion) re-confirm the known #22 and #80.
- working: C1 determinism over 7y / 148k bars, the lazy streaming source, and
the sweep/WFO cores held under real multi-year load; the Composite re-author
was behaviour-preserving (C23).
Scratch consumer crate + run transcripts committed under fieldtests/ per the
fieldtest convention (matching cycle-0049). The crate path-deps the engine and
reads the real archive; its bins are the repros cited in the issues above.
79 lines
2.8 KiB
Rust
79 lines
2.8 KiB
Rust
// Task 1 (deep dive 2) — the smoking gun: does PROCESS residency scale with
|
|
// window length, contradicting the docs' "a 20-year window streams in the same
|
|
// memory as a one-day one" (C12 cycle-0041 realization)?
|
|
//
|
|
// Drives ONE M1FieldSource (Close) to exhaustion over increasingly long windows,
|
|
// reading VmHWM after each. The aura-level resident_records stays <= CHUNK_SIZE
|
|
// (the docs' asserted predicate) the whole time; the question is whether VmHWM
|
|
// (the OS peak RSS) is flat or grows with the window. Fresh DataServer per window
|
|
// so a cache inside it cannot mask the effect. PUBLIC SURFACE ONLY.
|
|
|
|
use std::sync::Arc;
|
|
|
|
use chrono::TimeZone;
|
|
use data_server::{DataServer, DEFAULT_DATA_PATH};
|
|
|
|
use aura_engine::Source;
|
|
use aura_ingest::{M1Field, M1FieldSource};
|
|
|
|
const SYMBOL: &str = "GER40";
|
|
|
|
fn vmhwm_mib() -> u64 {
|
|
let status = std::fs::read_to_string("/proc/self/status").unwrap_or_default();
|
|
for line in status.lines() {
|
|
if let Some(rest) = line.strip_prefix("VmHWM:") {
|
|
let kib: u64 = rest.split_whitespace().next().and_then(|s| s.parse().ok()).unwrap_or(0);
|
|
return kib / 1024;
|
|
}
|
|
}
|
|
0
|
|
}
|
|
|
|
fn ms(y0: i32, m0: u32, y1: i32, m1: u32) -> (i64, i64) {
|
|
let from = chrono::Utc.with_ymd_and_hms(y0, m0, 1, 0, 0, 0).unwrap().timestamp_millis();
|
|
let (ny, nm) = if m1 == 12 { (y1 + 1, 1) } else { (y1, m1 + 1) };
|
|
let next = chrono::Utc.with_ymd_and_hms(ny, nm, 1, 0, 0, 0).unwrap().timestamp_millis();
|
|
(from, next - 1)
|
|
}
|
|
|
|
fn drive(label: &str, from_ms: i64, to_ms: i64) {
|
|
// Fresh server each time: no shared chunk cache carrying over.
|
|
let server = Arc::new(DataServer::new(DEFAULT_DATA_PATH));
|
|
let mut s = M1FieldSource::open(&server, SYMBOL, Some(from_ms), Some(to_ms), M1Field::Close)
|
|
.expect("window overlaps data");
|
|
let mut total = 0u64;
|
|
let mut peak_resident = 0usize;
|
|
while s.peek().is_some() {
|
|
peak_resident = peak_resident.max(s.resident_records());
|
|
Source::next(&mut s);
|
|
total += 1;
|
|
}
|
|
println!(
|
|
"{label:<16} records={:<9} aura-resident<=1024? {:<5} | VmHWM={} MiB",
|
|
total,
|
|
peak_resident <= 1024,
|
|
vmhwm_mib(),
|
|
);
|
|
}
|
|
|
|
fn main() {
|
|
if !Arc::new(DataServer::new(DEFAULT_DATA_PATH)).has_symbol(SYMBOL) {
|
|
println!("skip: no {SYMBOL} data");
|
|
return;
|
|
}
|
|
println!("=== Task 1 deep dive: does VmHWM scale with window length? ===");
|
|
println!("(the docs promise it does NOT — O(one chunk) residency)\n");
|
|
println!("baseline VmHWM = {} MiB\n", vmhwm_mib());
|
|
|
|
let (f, t) = ms(2024, 9, 2024, 9);
|
|
drive("1 month", f, t);
|
|
let (f, t) = ms(2024, 1, 2024, 6);
|
|
drive("6 months", f, t);
|
|
let (f, t) = ms(2024, 1, 2024, 12);
|
|
drive("1 year", f, t);
|
|
let (f, t) = ms(2021, 1, 2024, 12);
|
|
drive("4 years", f, t);
|
|
let (f, t) = ms(2014, 8, 2024, 12);
|
|
drive("~10 years", f, t);
|
|
}
|