From 0900f3f4133b05bcbb4a326047d890cbd02ad4f1 Mon Sep 17 00:00:00 2001 From: Brummel Date: Tue, 19 May 2026 02:40:16 +0200 Subject: [PATCH] iter embedding-abi-m5.3 (DONE 3/3): time-shard boundary-invisibility proof + friction harvest MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Final functional M5 iteration (spec ae905de, plan 67027ab; on green m5.1 204c171 + m5.2 b724cd1 whose leak-proof resolved via dbd76e5). Proves the chunk/window-boundary-invisibility claim the M4 retirement rests on: - adapter::fold_window — purely additive windowed sibling of fold_symbol (same MidPriceStream+Kernel path; stream_tick_windowed). fold_symbol/MidPriceStream/Kernel byte-untouched. - chrono as a DEV-dep only — the time-shard test derives (y,m)->Unix-ms bounds (data-server's own precedent) and passes raw ms to the bin; the bin is date-math-free; Invariant 1 unaffected. - new single-responsibility [[bin]] timeshard_runner — one thread per disjoint window, each owning its Kernel (Ctx:!Send => compile-time per-thread-ctx). - tests/timeshard.rs — spec §3 strength-ordered: (a) per-shard (acc,n) BIT-EXACT vs single-thread host fold of that exact window in stream order (THE boundary-invisibility proof); (b) Σ partials bit-exact by construction; (c) whole-window within REL_TOL=1e-6 only (cross-shard f64 reassociation is host arithmetic, never a kernel property — bit-exact there would be the bug; tolerance derived from the recursive-summation error bound). Plus the now-deterministic leak-Σ + a journal-only friction timing capture (no bench gate, no timing assertion). Concrete pin: EURUSD 2017-03/04/05 (recon-verified present, contiguous, no gaps; data-server has no month-listing API). RED was a genuine deterministic compile failure (declared-but-absent bin source; Cargo missing-source error vs the plan's predicted env! error — same cause, no false-green; benign plan-text imprecision in Concerns). Boss-verified independently: timeshard determinism 5/5 (no jitter); full ail-embed suite 0 failed AND 0 ignored across all binaries (m5.2 symbol_fan_swarm_leak_free stays un-ignored & green — the saga's resolution intact); isolated embed_rc_global_stats_race still green; AILang workspace data-server count 0; compiler-surface diff empty (zero diff to crates/ailang-*, crates/ail/, runtime/, examples/*.ail, root Cargo.toml, DESIGN.md). Friction-harvest deliverable (P2 flat-array-decision input): host-per-tick-FFI ~= 658 ms / 3,192,562 ticks ~= ~206 ns/tick at real EURUSD volume. M5 functional work complete; milestone closes next via the mandatory audit (which also handles the pre-existing DESIGN.md:2358-2360 "additive M4 concern" drift, out of m5.3 scope). Includes the per-iter journal, stats, and the INDEX.md line. --- ail-embed/Cargo.lock | 1 + ail-embed/Cargo.toml | 10 + ail-embed/src/adapter.rs | 19 ++ ail-embed/src/bin/timeshard_runner.rs | 73 ++++++ ail-embed/tests/timeshard.rs | 225 ++++++++++++++++++ .../2026-05-19-iter-embedding-abi-m5.3.json | 21 ++ .../2026-05-19-iter-embedding-abi-m5.3.md | 101 ++++++++ docs/journals/INDEX.md | 1 + 8 files changed, 451 insertions(+) create mode 100644 ail-embed/src/bin/timeshard_runner.rs create mode 100644 ail-embed/tests/timeshard.rs create mode 100644 bench/orchestrator-stats/2026-05-19-iter-embedding-abi-m5.3.json create mode 100644 docs/journals/2026-05-19-iter-embedding-abi-m5.3.md diff --git a/ail-embed/Cargo.lock b/ail-embed/Cargo.lock index b7981d8..a2a89fd 100644 --- a/ail-embed/Cargo.lock +++ b/ail-embed/Cargo.lock @@ -32,6 +32,7 @@ dependencies = [ name = "ail-embed" version = "0.0.1" dependencies = [ + "chrono", "data-server", "tempfile", "zip", diff --git a/ail-embed/Cargo.toml b/ail-embed/Cargo.toml index 7bc408a..6f9b264 100644 --- a/ail-embed/Cargo.toml +++ b/ail-embed/Cargo.toml @@ -30,7 +30,17 @@ data-server = { path = "../../libs/data-server" } [dev-dependencies] zip = "2" tempfile = "3" +# Test-only: the time-shard test derives (year,month) -> Unix-ms +# window bounds via chrono (data-server's own bound precedent, +# data-server/src/lib.rs:696-699) and passes raw ms to the bin, so +# the bin itself stays date-math-free. chrono is already transitively +# present via data-server; this is the test-direct declaration. +chrono = "0.4" [[bin]] name = "swarm_runner" path = "src/bin/swarm_runner.rs" + +[[bin]] +name = "timeshard_runner" +path = "src/bin/timeshard_runner.rs" diff --git a/ail-embed/src/adapter.rs b/ail-embed/src/adapter.rs index 4539359..fc9403a 100644 --- a/ail-embed/src/adapter.rs +++ b/ail-embed/src/adapter.rs @@ -68,6 +68,25 @@ pub fn fold_symbol( Some(Kernel::new().run(MidPriceStream::new(it).take(max_ticks))) } +/// Fold the mid-price stream of `symbol` restricted to the inclusive +/// Unix-millisecond window `[from_ms, to_ms]` through the M3 kernel. +/// `None` if the symbol has no tick stream in that window. The window +/// is the bound (no tick cap): the result is `(Σ mid, count)` over +/// exactly the records data-server yields for that window, in stream +/// order — identical f64-add sequence to a single-thread host fold of +/// the same window, hence bit-exact against it. The windowed sibling +/// of `fold_symbol`; same `MidPriceStream` + `Kernel` path, only the +/// data-server entry differs (`stream_tick_windowed`). +pub fn fold_window( + server: &Arc, + symbol: &str, + from_ms: i64, + to_ms: i64, +) -> Option<(f64, i64)> { + let it = server.stream_tick_windowed(symbol, Some(from_ms), Some(to_ms))?; + Some(Kernel::new().run(MidPriceStream::new(it))) +} + #[cfg(test)] mod tests { use super::tick_to_px; diff --git a/ail-embed/src/bin/timeshard_runner.rs b/ail-embed/src/bin/timeshard_runner.rs new file mode 100644 index 0000000..1d03ab4 --- /dev/null +++ b/ail-embed/src/bin/timeshard_runner.rs @@ -0,0 +1,73 @@ +//! Time-shard swarm harness (spec §3). Run as a subprocess by +//! `tests/timeshard.rs` under `AILANG_RC_STATS=1`. +//! +//! argv: ` [ ...]` — one +//! inclusive Unix-ms window per shard. Date math lives in the test +//! (chrono); this bin is date-math-free. Spawns one thread per +//! window, each building its OWN `Kernel` (`Ctx: !Send` forces +//! one-ctx-per-thread — a shared-ctx design is `error[E0277]`). +//! Prints, in shard order, `RESULT +//! `, then one `TIMING total_ticks=<Σn> elapsed_ms=` line +//! (the friction-harvest observation; the test records it in the +//! close-out journal, it is not asserted). + +use std::sync::Arc; +use std::time::Instant; + +use ail_embed::adapter::fold_window; +use data_server::DataServer; + +fn main() { + let mut args = std::env::args().skip(1); + let data_dir = args.next().expect("argv[1] = data dir"); + let symbol = args.next().expect("argv[2] = symbol"); + let windows: Vec<(i64, i64)> = args + .map(|a| { + let (f, t) = a.split_once(':').expect("window arg is from:to"); + ( + f.parse::().expect("from_ms i64"), + t.parse::().expect("to_ms i64"), + ) + }) + .collect(); + assert!( + windows.len() >= 2, + "need >= 2 disjoint windows for a time-shard, got {}", + windows.len() + ); + + let server = Arc::new(DataServer::new(&data_dir)); + + let started = Instant::now(); + let handles: Vec<_> = windows + .iter() + .copied() + .enumerate() + .map(|(idx, (from, to))| { + let srv = Arc::clone(&server); + let sym = symbol.clone(); + // Each thread builds its own Kernel inside the closure — + // Kernel/Ctx are !Send, so this is the ONLY shape that + // compiles (one ctx per thread, type-enforced). + std::thread::spawn(move || { + let (acc, n) = fold_window(&srv, &sym, from, to) + .expect("window has a tick stream"); + (idx, from, to, acc, n) + }) + }) + .collect(); + + let mut results: Vec<(usize, i64, i64, f64, i64)> = handles + .into_iter() + .map(|h| h.join().expect("shard thread panicked")) + .collect(); + let elapsed_ms = started.elapsed().as_millis(); + results.sort_by_key(|r| r.0); + + let mut total_ticks: i64 = 0; + for (idx, from, to, acc, n) in &results { + total_ticks += *n; + println!("RESULT {idx} {from} {to} {:016x} {n}", acc.to_bits()); + } + println!("TIMING total_ticks={total_ticks} elapsed_ms={elapsed_ms}"); +} diff --git a/ail-embed/tests/timeshard.rs b/ail-embed/tests/timeshard.rs new file mode 100644 index 0000000..71fdbd4 --- /dev/null +++ b/ail-embed/tests/timeshard.rs @@ -0,0 +1,225 @@ +//! Time-shard boundary-invisibility E2E (spec Architecture §3 +//! second assertion / Testing strategy §3). One symbol, N disjoint +//! single-month windows, one thread per window (each owning its +//! `Kernel` — `Ctx: !Send`). Asserts, in spec strength order: +//! (a) each shard's `(acc,n)` == a single-thread host fold of THAT +//! EXACT window in stream order, BIT-EXACT — the +//! boundary-invisibility proof (the kernel result for a window +//! is independent of how data-server chunked it); +//! (b) the host-summed `(Σacc,Σn)` == the identically-summed +//! reference partials, bit-exact by construction; +//! (c) the whole-window single-thread total within an +//! f64-reassociation tolerance only (cross-shard re-association +//! is host arithmetic, NOT a kernel property — asserting it +//! bit-exact would be WRONG, per the spec). +//! Plus the global leak-Σ (deterministic since dbd76e5) and a +//! journal-only friction timing capture (NO bench gate). +//! +//! Concrete pin: EURUSD, months 2017-03 / 2017-04 / 2017-05 (all +//! present on /mnt, contiguous, no gaps — recon-verified). data-server +//! has no public month-listing API, so the months are hard-pinned. +//! Skips when /mnt absent — data-server's own precedent. + +use std::process::Command; +use std::sync::Arc; + +use chrono::{NaiveDate, TimeZone, Utc}; +use data_server::{DataServer, DEFAULT_DATA_PATH}; + +const SYMBOL: &str = "EURUSD"; +/// Three disjoint single-month windows. Contiguous so the whole-window +/// union is exactly 2017-03-01 .. 2017-05-31 23:59:59.999. +const SHARDS: [(i32, u32); 3] = [(2017, 3), (2017, 4), (2017, 5)]; +/// Relative tolerance for the (c) whole-window cross-check. Derivation: +/// recursive f64 summation of n terms has error <= (n-1)*eps*Σ|x_i|. +/// One EURUSD month ~250k ticks => 3 shards ~750k adds of O(1) mids; +/// Σ|x| ~ 8.3e5, (n-1)*eps ~ 8.3e-11 => abs err ~ 6.8e-5, rel ~ 8e-11. +/// 1e-6 is ~4 orders of safety yet still catches any real bug +/// (a genuine miscount is off by >> 1 ppm). +const REL_TOL: f64 = 1e-6; + +fn skip_if_no_data() -> bool { + !std::path::Path::new(DEFAULT_DATA_PATH).exists() +} + +/// Inclusive Unix-ms bounds for the whole of calendar month `(y, m)`. +/// `filter_files` windows at month granularity via +/// `unix_ms_to_year_month`, and `filter_chunk` is record-level +/// inclusive `[from, to]`; first-instant .. last-instant of the month +/// selects exactly that month's file and all its records. +fn month_window(y: i32, m: u32) -> (i64, i64) { + let from = Utc + .from_utc_datetime( + &NaiveDate::from_ymd_opt(y, m, 1) + .unwrap() + .and_hms_opt(0, 0, 0) + .unwrap(), + ) + .timestamp_millis(); + let (ny, nm) = if m == 12 { (y + 1, 1) } else { (y, m + 1) }; + let next_from = Utc + .from_utc_datetime( + &NaiveDate::from_ymd_opt(ny, nm, 1) + .unwrap() + .and_hms_opt(0, 0, 0) + .unwrap(), + ) + .timestamp_millis(); + (from, next_from - 1) +} + +/// Single-thread host reference: fold the mid stream over EXACTLY the +/// inclusive `[from_ms, to_ms]` window in data-server stream order +/// (`acc += mid; n += 1`). Bit-exact with `adapter::fold_window` +/// because the f64-add order is identical. +fn reference_window(server: &Arc, from_ms: i64, to_ms: i64) -> (f64, i64) { + let mut it = server + .stream_tick_windowed(SYMBOL, Some(from_ms), Some(to_ms)) + .expect("EURUSD has a tick stream in the pinned window"); + let (mut acc, mut n) = (0.0_f64, 0_i64); + while let Some(chunk) = it.next_chunk() { + for r in chunk.iter() { + acc += (r.ask + r.bid) / 2.0; + n += 1; + } + } + (acc, n) +} + +#[test] +fn timeshard_boundary_invisibility_and_leak_free() { + if skip_if_no_data() { + eprintln!("skipping: {DEFAULT_DATA_PATH} absent (mirrors data-server)"); + return; + } + + let windows: Vec<(i64, i64)> = + SHARDS.iter().map(|&(y, m)| month_window(y, m)).collect(); + + // Spawn the time-shard runner: then one + // "from:to" arg per shard. It folds each window on its own thread + // (own Kernel) and prints `RESULT ` + // + a final `TIMING total_ticks= elapsed_ms=` line. + let mut cmd = Command::new(env!("CARGO_BIN_EXE_timeshard_runner")); + cmd.arg(DEFAULT_DATA_PATH).arg(SYMBOL); + for (f, t) in &windows { + cmd.arg(format!("{f}:{t}")); + } + let out = cmd + .env("AILANG_RC_STATS", "1") + .output() + .expect("spawn timeshard_runner"); + let stdout = String::from_utf8_lossy(&out.stdout); + let stderr = String::from_utf8_lossy(&out.stderr); + assert!( + out.status.success(), + "timeshard_runner exited {:?}\nstdout:\n{stdout}\nstderr:\n{stderr}", + out.status.code() + ); + + let server = Arc::new(DataServer::new(DEFAULT_DATA_PATH)); + + // Parse RESULT lines into (idx -> (from,to,acc,n)). + let mut shards: Vec<(usize, i64, i64, f64, i64)> = Vec::new(); + for line in stdout.lines().filter(|l| l.starts_with("RESULT ")) { + let mut t = line.split_whitespace(); + let _ = t.next(); // RESULT + let idx: usize = t.next().unwrap().parse().unwrap(); + let from: i64 = t.next().unwrap().parse().unwrap(); + let to: i64 = t.next().unwrap().parse().unwrap(); + let acc = f64::from_bits( + u64::from_str_radix(t.next().unwrap(), 16).unwrap(), + ); + let n: i64 = t.next().unwrap().parse().unwrap(); + shards.push((idx, from, to, acc, n)); + } + shards.sort_by_key(|s| s.0); + assert_eq!( + shards.len(), + SHARDS.len(), + "expected {} shard RESULT lines, got {}", + SHARDS.len(), + shards.len() + ); + + // (a) Per-shard BIT-EXACT vs a single-thread host fold of that + // exact window. THE boundary-invisibility proof. + let mut sum_acc = 0.0_f64; + let mut sum_n = 0_i64; + let mut ref_sum_acc = 0.0_f64; + let mut ref_sum_n = 0_i64; + for (i, &(idx, from, to, acc, n)) in shards.iter().enumerate() { + assert_eq!(idx, i, "shard indices contiguous 0..N"); + assert_eq!((from, to), windows[i], "shard {i} window matches pin"); + let (r_acc, r_n) = reference_window(&server, from, to); + assert_eq!( + acc.to_bits(), + r_acc.to_bits(), + "shard {i} ({from}..{to}): kernel acc BIT-EXACT vs \ + single-thread host fold of that exact window \ + (boundary-invisibility)" + ); + assert_eq!(n, r_n, "shard {i}: tick count == reference"); + assert!(r_n > 0, "shard {i}: non-empty window"); + sum_acc += acc; + sum_n += n; + ref_sum_acc += r_acc; + ref_sum_n += r_n; + } + + // (b) Host-summed partials == identically-summed reference + // partials, BIT-EXACT by construction (same values, same + // host summation order). + assert_eq!( + sum_acc.to_bits(), + ref_sum_acc.to_bits(), + "Σ shard acc bit-exact vs Σ reference partials" + ); + assert_eq!(sum_n, ref_sum_n, "Σ shard n == Σ reference n"); + + // (c) Whole-window single-thread total within REL_TOL only — + // cross-shard re-association is host arithmetic, NOT a kernel + // property; asserting bit-exact here would be WRONG (spec §3). + let whole_from = windows.first().unwrap().0; + let whole_to = windows.last().unwrap().1; + let (whole_acc, whole_n) = reference_window(&server, whole_from, whole_to); + assert_eq!( + whole_n, sum_n, + "whole-window tick count == Σ shard n (windows are an exact \ + disjoint partition of the union)" + ); + let rel = ((whole_acc - sum_acc) / whole_acc).abs(); + assert!( + rel < REL_TOL, + "whole-window total {whole_acc} vs Σ partials {sum_acc}: \ + relative diff {rel:e} must be < {REL_TOL:e} (f64 \ + reassociation only — NOT a kernel discrepancy)" + ); + + // Global leak-Σ across ALL ailang_rc_stats: lines (deterministic + // since dbd76e5: atomic global counters + build.rs runtime-track). + let mut allocs: u64 = 0; + let mut frees: u64 = 0; + let mut seen = 0usize; + for line in stderr.lines().filter(|l| l.starts_with("ailang_rc_stats:")) { + seen += 1; + for tok in line.split_whitespace() { + if let Some(v) = tok.strip_prefix("allocs=") { + allocs += v.parse::().expect("allocs= u64"); + } else if let Some(v) = tok.strip_prefix("frees=") { + frees += v.parse::().expect("frees= u64"); + } + } + } + assert!(seen > 0, "no ailang_rc_stats line; stderr:\n{stderr}"); + assert_eq!( + allocs, frees, + "globally leak-free: Σallocs==Σfrees across {seen} stat line(s)" + ); + + // Friction harvest: journal-only observation, NO assertion (the + // value feeds the P2 batch-crossing decision; it is not a gate). + if let Some(tl) = stdout.lines().find(|l| l.starts_with("TIMING ")) { + eprintln!("m5.3 friction-harvest: {tl} (Σn={sum_n})"); + } +} diff --git a/bench/orchestrator-stats/2026-05-19-iter-embedding-abi-m5.3.json b/bench/orchestrator-stats/2026-05-19-iter-embedding-abi-m5.3.json new file mode 100644 index 0000000..8aeb46d --- /dev/null +++ b/bench/orchestrator-stats/2026-05-19-iter-embedding-abi-m5.3.json @@ -0,0 +1,21 @@ +{ + "iter_id": "embedding-abi-m5.3", + "date": "2026-05-19", + "mode": "standard", + "outcome": "DONE", + "tasks_total": 3, + "tasks_completed": 3, + "reloops_per_task": { "1": 0, "2": 0, "3": 0 }, + "review_loops_spec": 0, + "review_loops_quality": 0, + "blocked_reason": null, + "friction_harvest": { + "timing_line": "TIMING total_ticks=3192562 elapsed_ms=658", + "total_ticks": 3192562, + "elapsed_ms": 658, + "derived_ns_per_tick_ffi": 206, + "note": "journal-only observation, NO bench gate; Boss records derived per-tick-FFI cost as the P2 flat-array-decision input in the close-out journal" + }, + "determinism_runs_green": 6, + "tasks_with_concerns": ["2"] +} diff --git a/docs/journals/2026-05-19-iter-embedding-abi-m5.3.md b/docs/journals/2026-05-19-iter-embedding-abi-m5.3.md new file mode 100644 index 0000000..3af3547 --- /dev/null +++ b/docs/journals/2026-05-19-iter-embedding-abi-m5.3.md @@ -0,0 +1,101 @@ +# iter embedding-abi-m5.3 — time-shard boundary-invisibility proof + friction harvest + +**Date:** 2026-05-19 +**Started from:** 67027ab0f98210926ed30ffd551306d3db3645ee +**Status:** DONE +**Tasks completed:** 3 of 3 + +## Summary + +Final functional M5 iteration. Proves the chunk/window-boundary-invisibility +claim the M4 retirement rests on: one symbol (EURUSD), three disjoint +contiguous single-month windows (2017-03 / 2017-04 / 2017-05), each folded +on its own thread owning its own `Kernel` (`Ctx: !Send` makes +one-ctx-per-thread a compile-time guarantee), each shard's `(acc,n)` +asserted **bit-exact** vs a single-thread host fold of that exact window in +data-server stream order. Adds the windowed adapter sibling `fold_window` +(purely additive after `fold_symbol`), a new single-responsibility +`[[bin]] timeshard_runner`, and `tests/timeshard.rs` carrying the spec §3 +strength-ordered (a)/(b)/(c) assertions plus the deterministic leak-Σ and a +journal-only friction-timing capture (no bench gate, no timing assertion). +The shipped m5.2 symbol-fan path (`swarm_runner.rs`, `swarm.rs`) and m5.1 +core stay byte-untouched and green. Invariant 1 holds (the AILang workspace +graph has zero `data-server`; `chrono` enters only as an `ail-embed` +dev-dep). Determinism re-verified 6× (5× standalone + 1× in full suite), +all identical. Friction-harvest observed: +`TIMING total_ticks=3192562 elapsed_ms=658` → derived per-tick-FFI cost +≈ 658 ms / 3,192,562 ≈ **~206 ns/tick** (the Boss records this as the P2 +flat-array-decision input in the close-out journal). + +## Per-task notes + +- iter embedding-abi-m5.3.1: windowed adapter + chrono dev-dep + bin + declaration — `fold_window(server, symbol, from_ms, to_ms)` appended to + `ail-embed/src/adapter.rs` immediately after `fold_symbol`, before the + `#[cfg(test)] mod tests` block (windowed sibling, same + `MidPriceStream`+`Kernel` path, differs only by `stream_tick_windowed`); + `chrono = "0.4"` added to `[dev-dependencies]` + `[[bin]] timeshard_runner` + declared in `ail-embed/Cargo.toml`. `--lib` regression check + `2 passed; 0 failed` (exactly the plan's prediction). Spec compliant, + quality approved, 0 re-loops. +- iter embedding-abi-m5.3.2: the time-shard E2E test (RED) — + `ail-embed/tests/timeshard.rs` created verbatim from the plan + (boundary-invisibility (a)/(b)/(c) + leak-Σ + friction capture). + RED observed: the declared-but-absent `[[bin]] timeshard_runner` source + makes the test binary fail to build. Spec compliant, quality approved, + 0 re-loops. DONE_WITH_CONCERNS — see Concerns (RED-text discrepancy, + same cause). +- iter embedding-abi-m5.3.3: the `timeshard_runner` bin (GREEN) — + `ail-embed/src/bin/timeshard_runner.rs` created verbatim (one thread per + window, each building its own `Kernel`; prints `RESULT`/`TIMING`). + Step 2 GREEN (`1 passed; 0 failed`, ran 2.71s against real /mnt data — + did NOT skip). Step 3 determinism ×5: 5/5 `ok`. Step 4 full suite: + 0 failed AND 0 ignored across all binaries (m5.2 `symbol_fan_swarm_*` + un-ignored & green). Step 5: `cargo metadata` data-server count = 0; + path filter empty. Step 6: `embed_rc_global_stats_race` + `1 passed; 0 failed`. Spec compliant, quality approved, 0 re-loops. + +## Concerns + +- iter embedding-abi-m5.3.2 (DONE_WITH_CONCERNS): the Task 2 Step 2 RED + surfaced as `error: couldn't read \`src/bin/timeshard_runner.rs\`: No such + file or directory` rather than the plan's predicted `error: environment + variable \`CARGO_BIN_EXE_timeshard_runner\` not defined at compile time`. + Same root cause (the declared `[[bin]] timeshard_runner` target has no + source until Task 3), same trigger, same resolution (disappears exactly + when Task 3 lands the bin) — Cargo fails at the missing-source check + before reaching the `env!` expansion, so the surface error differs but + the RED is genuine, deterministic, and not a false-green. Observation + only; the plan's intent (a deterministic compile-failure RED isolating + Task 2 from Task 3) is fully satisfied. + +## Known debt + +- (none introduced by this iter) +- Pre-existing `docs/DESIGN.md:2358-2360` "additive M4 concern" drift is + explicitly out of m5.3 scope (spec §"Out of scope" / Boss decision) and + was deliberately not touched — the milestone-close audit handles it, + not this iter. + +## Blocked detail + +(not applicable — Status DONE) + +## Files touched + +- `ail-embed/src/adapter.rs` (modified — additive `fold_window`) +- `ail-embed/Cargo.toml` (modified — `chrono` dev-dep + `timeshard_runner` bin) +- `ail-embed/Cargo.lock` (modified — single line: `chrono` direct-dep edge; + automatic, strictly required by the Cargo.toml dep add, already resolved + transitively via data-server) +- `ail-embed/src/bin/timeshard_runner.rs` (created) +- `ail-embed/tests/timeshard.rs` (created) + +All within the plan's declared file set and the Step 5b allowed path scope +(`ail-embed/**`, `docs/`, `bench/orchestrator-stats/`). Zero diff to +`crates/ailang-*`, `crates/ail/`, `runtime/`, `examples/*.ail`, root +`Cargo.toml`, `docs/DESIGN.md`. + +## Stats + +bench/orchestrator-stats/2026-05-19-iter-embedding-abi-m5.3.json diff --git a/docs/journals/INDEX.md b/docs/journals/INDEX.md index 8b6aae0..cfec1cd 100644 --- a/docs/journals/INDEX.md +++ b/docs/journals/INDEX.md @@ -111,3 +111,4 @@ - 2026-05-19 — iter bugfix-rc-global-stats-race (RED→GREEN, debug→implement mini, DONE 1/1): the M5 iter-2 bounce-back resolution (user-approved Option A). The two null-ctx fallback RC-stats counters `g_rc_alloc_count`/`g_rc_free_count` (`runtime/rc.c:90-91`) were plain `static uint64_t` with a non-atomic `++` on the `__ail_tls_ctx == NULL` arm of `ailang_rc_alloc` (rc.c:161) / the to-zero branch of `ailang_rc_dec` (rc.c:212); a multi-threaded host driving the global fallback (no `ailang_ctx_new`) raced the read-modify-write and lost increments, so the `AILANG_RC_STATS` atexit Σ under-counted non-deterministically — exactly the deferred "concurrency arrived: atomic-vs-non-atomic" decision rc.c's own header (rc.c:44-49) names, M5 being AILang's first concurrent consumer. NOT a memory bug (`Ctx: !Send` keeps every box on one thread; the real refcount/free is correct; programs bit-exact). RED (debugger, audit-trail commit `427b687`): a C host — 8 threads × 2M alloc-then-dec, no ctx so the global path is taken, no box crossing a thread — + integration test asserting the atexit Σ is exact; Boss-verified RED `allocs=2141382 expected 16000000, live=-131242` (deterministic-fail under that contention, not flaky). GREEN (implement mini, this commit): the two globals are now `_Atomic uint64_t`, `atomic_fetch_add_explicit(.., memory_order_relaxed)` at the two fallback `++` sites, `atomic_load_explicit(.., relaxed)` in `ailang_rc_stats_atexit` (its sole reader) — relaxed is correct (pure stats, no happens-before; atexit reader runs post-join). Per-ctx counters + the per-object refcount header left non-atomic BY DESIGN (single-thread-per-ctx; boxes never cross threads) — explicitly out of scope, untouched; frozen value layout / ABI offsets / host-free rule untouched; the genuinely-still-single-threaded drop-worklist doc note correctly left unchanged (a false correction was refused); the two genuinely-stale atomicity doc blocks corrected for honesty. Scope held to `runtime/rc.c` ONLY (37+/14−). Boss-verified independently: RED→GREEN deterministically 3/3 (jitter gone, `allocs==frees==16_000_000`), full `cargo test -p ail` green (every binary 0 failed — the per-ctx tsan harnesses embed_swarm_tsan/embed_rc_accounting_tsan + embed_tick_e2e/embed_record_e2e unaffected since the per-ctx path is untouched; rc.c links into every ail binary so this is the strong regression gate), git-status scope = rc.c + journal + stats only, the RED files unchanged. Unblocks resuming the M5 leak-proof (un-`#[ignore]` `ail-embed`'s `symbol_fan_swarm_leak_free`). RED `427b687` → GREEN this commit. → 2026-05-19-iter-bugfix-rc-global-stats-race.md - 2026-05-19 — iter embedding-abi-m5.2-resume-attempt (RE-QUARANTINED; honesty fix only): attempted to un-`#[ignore]` the M5 swarm leak-proof `symbol_fan_swarm_leak_free` as the integration-level acceptance of the runtime fix `7bfa11e` (bugfix-rc-global-stats-race). Boss verification FALSIFIED the sufficiency premise: across 4–6 swarm runs the leak Σ is still non-deterministic — `Σfrees` stable-exact (12000003) every run, `Σallocs` short by a jittering ~1600–2260 in ~1/3 of runs (e.g. left:11998383 right:12000003 across 4 stat lines). The atomic-global fix is real, committed, NECESSARY (the isolated 8×2M pure-global RED `embed_rc_global_stats_race` is deterministically green) but NOT sufficient — a second, distinct alloc-side undercount remains that the isolated RED did not model (frees never lose ⇒ not a per-ctx race; not pure global contention ⇒ RED green; localisation beyond that is where Boss speculation stops per the debug Iron Law). Un-ignore reverted (never committed); the only shipped diff is the three now-stale `#[ignore]`-rationale breadcrumbs in `ail-embed/tests/swarm.rs` corrected from "un-ignore when the runtime is atomic" to the accurate necessary-but-insufficient state (a future agent seeing `7bfa11e` must NOT un-ignore on that basis — doc-honesty). Test body + the `#[ignore]` itself unchanged (still quarantined, NOT weakened); `symbol_fan_swarm_bit_exact` stays live+GREEN (the swarm IS actually correct and leak-free — only the accounting is wrong). Residual handed to a fresh debug RED-first cycle; two framings flagged for it (genuine residual runtime defect vs Σ-over-heterogeneous-global+per-ctx-lines methodology unsoundness — the latter, if so, is a post-root-cause design decision). main green at the honesty-fix commit; M5 stays open `[~]`; m5.3 remains downstream of a sound leak-proof. → 2026-05-19-iter-embedding-abi-m5.2-resume-attempt.md - 2026-05-19 — iter bugfix-swarm-rc-alloc-undercount (RED→GREEN, debug→implement mini, DONE 1/1): root-caused and fixed the m5.2-resume-attempt residual. The M5 swarm leak-proof was still non-deterministic after the atomic-counter runtime fix `7bfa11e` — NOT a second runtime bug and NOT test-methodology unsoundness, but a **build-dependency staleness gap**: `ail-embed/build.rs` declared `cargo:rerun-if-changed` only for the kernel `.ail` + `build.rs`, NOT the `runtime/` C sources `ail build --emit=staticlib` compiles into `libailang_rt.a`. So Cargo never re-ran `build.rs` after `7bfa11e`; `ail-embed` kept linking a stale pre-`7bfa11e` non-atomic `libailang_rt.a` whose `g_rc_alloc_count++` raced the swarm's concurrent TLS-NULL host allocs (explaining the frees-stable/allocs-jitter asymmetry, and why the isolated `crates/ail` RED — which rebuilds the staticlib fresh per run — was green while the `ail-embed` swarm was not). `runtime/rc.c` at HEAD was always correct; the entire production fix is one directory-level `cargo:rerun-if-changed=/runtime` line (preferred over a per-file list — the implementer inspected `crates/ail` `build_staticlib` confirming `libailang_rt.a = ar(rc.o,str.o)`, both under `runtime/`; directory tracking is future-proof against any new runtime source). RED (debugger, audit-trail `483117d`): `ail-embed/tests/rt_archive_freshness.rs` pins the *cause* deterministically (objdump the linked archive for `lock`-prefixed atomic increments + a fresh-build cross-check) since the *symptom* is non-deterministic by construction; Boss-verified RED = 0 lock insns ("STALE"). GREEN (implement mini, this commit): the build.rs fix + the cohesive consequence — `symbol_fan_swarm_leak_free` un-`#[ignore]`d (body byte-for-byte verbatim — it was always quarantined, never weakened; the three doc breadcrumbs rewritten to the resolved build-dep-staleness rationale), now the integration-level acceptance of both `7bfa11e` and this fix. Boss removed a pre-existing dead `DataFormat` import in swarm.rs inline (trivial; file committed this iter regardless). Boss-verified independently: RED→GREEN; **swarm determinism 5/5 — `symbol_fan_swarm_leak_free` + `symbol_fan_swarm_bit_exact` GREEN every run, 0 ignored, jitter gone, Σallocs==Σfrees==12000003**; isolated `embed_rc_global_stats_race` still GREEN (untouched); Invariant 1 data-server count 0; scope = `ail-embed/build.rs` + `ail-embed/tests/swarm.rs` + journal + stats only. The M5 swarm leak-proof bounce-back is now fully resolved end-to-end (bounce-back → Option A `7bfa11e` → resume-attempt "necessary-but-insufficient" → this build-dep root-cause+fix). RED `483117d` → GREEN this commit. M5 stays open `[~]`; m5.3 (time-shard + friction-harvest + close-out) remains. → 2026-05-19-iter-bugfix-swarm-rc-alloc-undercount.md +- 2026-05-19 — iter embedding-abi-m5.3 (DONE 3/3): final functional M5 iteration — the time-shard boundary-invisibility proof + friction harvest. One symbol (EURUSD), three disjoint contiguous single-month windows (2017-03/04/05), each folded on its own thread owning its own `Kernel` (`Ctx: !Send` ⇒ one-ctx-per-thread compile-time-enforced). Adds the purely-additive windowed adapter sibling `fold_window` (same `MidPriceStream`+`Kernel` path as `fold_symbol`, differs only by `stream_tick_windowed`), a new single-responsibility `[[bin]] timeshard_runner`, `chrono` as a **dev-dep** (the time-shard test derives `(y,m)→Unix-ms` bounds via data-server's own `NaiveDate…timestamp_millis()` precedent and passes raw ms to the bin, keeping the bin date-math-free and Invariant 1 untouched), and `tests/timeshard.rs` carrying the spec §3 strength-ordered assertions: (a) each shard's `(acc,n)` **bit-exact** vs a single-thread host fold of THAT EXACT window in data-server stream order — the first actual proof of the chunk/window-boundary-invisibility claim the entire M4 retirement rests on; (b) host-summed partials bit-exact by construction; (c) the whole-window [2017-03..2017-05] single-thread total within `REL_TOL=1e-6` only (cross-shard f64 re-association is host arithmetic, NOT a kernel property — asserting it bit-exact would be the bug; tolerance derived from the recursive-summation error bound with ~4 orders of safety, shown in the test). Plus the now-deterministic global leak-Σ (post-`dbd76e5`) and a journal-only friction-timing capture (no bench gate, no timing assertion — flake-free). RED was a genuine deterministic compile failure (declared-but-absent `[[bin]] timeshard_runner` source; surfaced as Cargo's missing-source error rather than the plan's predicted `env!` error — same root cause, same trigger, no false-green; recorded as a benign plan-text imprecision in Concerns). The shipped m5.2 symbol-fan path (`swarm_runner.rs`, `swarm.rs`) and m5.1 core stay byte-untouched and green. Boss-verified independently: **timeshard determinism 5/5** (consistent ~2.7s, no jitter); full ail-embed suite **0 failed AND 0 ignored** across all binaries (lib 2, swarm 2 — the m5.2 leak-proof stays un-ignored & green, the whole saga's resolution intact —, smoke 1, rt_archive 1, timeshard 1); isolated `embed_rc_global_stats_race` still green; AILang workspace `cargo metadata` data-server count 0; compiler-surface diff empty (zero diff to crates/ailang-*, crates/ail/, runtime/, examples/*.ail, root Cargo.toml, DESIGN.md). **Friction-harvest deliverable (P2 flat-array-decision input): host-per-tick-FFI ≈ 658 ms / 3 192 562 ticks ≈ ~206 ns/tick at real EURUSD volume.** M5's functional work is complete; the milestone closes next via the mandatory `audit` (which also handles the pre-existing `docs/DESIGN.md:2358-2360` "additive M4 concern" drift, explicitly out of m5.3 scope). → 2026-05-19-iter-embedding-abi-m5.3.md