plan: embedding-abi-m5.3 — time-shard boundary-invisibility proof + friction harvest
Final functional M5 iteration (spec ae905de; builds on green m5.1204c171+ m5.2b724cd1whose leak-proof is resolved throughdbd76e5). 3 tasks: (1) additive adapter::fold_window (windowed sibling of fold_symbol) + chrono dev-dep + a second [[bin]] timeshard_runner declaration; (2) tests/timeshard.rs written FIRST — RED is the deterministic `env!(CARGO_BIN_EXE_ timeshard_runner) not defined` compile failure (bin absent); (3) the timeshard_runner bin → GREEN, with the spec §3 strength-ordered assertions (a) per-shard bit-exact vs single-thread windowed host fold (THE boundary-invisibility proof), (b) Σ bit-exact by construction, (c) whole-window within REL_TOL=1e-6 (derived from the recursive-summation error bound, never bit-exact — spec explicit), plus the now-deterministic leak-Σ and a journal-only friction-timing capture. Boss decisions in-plan: new bin not an arg-mode (shipped symbol-fan path byte-untouched, no regression surface); chrono dev-only (date math in the test, bin date-math-free; Invariant 1 unaffected); concrete pin EURUSD 2017_03/04/05 (recon-verified present, no gaps; data-server has no month-listing API); friction is a journal observation, no bench gate, no timing assertion (flake-free); pre-existing DESIGN.md:2358-2360 M4-drift explicitly out of scope. Milestone close + mandatory audit run after Boss-verification.
This commit is contained in:
@@ -0,0 +1,588 @@
|
||||
# embedding-abi-m5.3 — time-shard boundary-invisibility proof + friction harvest — Implementation Plan
|
||||
|
||||
> **Parent spec:** `docs/specs/2026-05-19-embedding-abi-m5.md`
|
||||
>
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: use `skills/implement`
|
||||
> to run this plan. Steps use `- [ ]` checkboxes for tracking.
|
||||
|
||||
**Goal:** Prove the chunk/window-boundary-invisibility claim the M4
|
||||
retirement rests on — one symbol, N disjoint month-windows, each
|
||||
shard's `(acc,n)` bit-exact vs a single-thread host fold of that exact
|
||||
window — and harvest the per-tick-FFI cost at real volume for the P2
|
||||
decision. The final functional M5 iteration before milestone close.
|
||||
|
||||
**Architecture:** Builds on shipped+green m5.1/m5.2 (adapter,
|
||||
`Kernel`, `MidPriceStream`, atomic runtime, build.rs runtime-tracking,
|
||||
deterministic leak-proof). Adds a windowed adapter sibling
|
||||
`fold_window`, a new single-responsibility `[[bin]] timeshard_runner`
|
||||
(one thread per disjoint month-window, each owning its `Kernel` —
|
||||
`Ctx: !Send` makes one-ctx-per-thread a compile-time guarantee), and
|
||||
`tests/timeshard.rs` asserting the spec §3 strength-ordered (a)/(b)/(c)
|
||||
plus the (now-deterministic) leak-Σ, with a journal-only friction
|
||||
timing capture. The shipped symbol-fan path (`swarm_runner.rs`,
|
||||
`swarm.rs`) is NOT touched.
|
||||
|
||||
**Tech Stack:** Rust (edition 2021); `data-server`
|
||||
(`stream_tick_windowed`, `filter_files` month-granularity windowing);
|
||||
`chrono` (dev-dep, `(y,m)→unix_ms` bounds, data-server's own
|
||||
precedent); `std::thread`; `std::time::Instant`; the M3 staticlib via
|
||||
the m5.1 `build.rs`.
|
||||
|
||||
---
|
||||
|
||||
**Files this plan creates or modifies:**
|
||||
|
||||
- Modify: `ail-embed/Cargo.toml` — add `chrono = "0.4"` to `[dev-dependencies]`; add a second `[[bin]] timeshard_runner`
|
||||
- Modify: `ail-embed/src/adapter.rs` — add `fold_window` (windowed sibling of `fold_symbol`)
|
||||
- Create: `ail-embed/src/bin/timeshard_runner.rs` — one thread per disjoint window; prints `RESULT`/`TIMING`
|
||||
- Test: `ail-embed/tests/timeshard.rs` — boundary-invisibility (a)/(b)/(c) + leak-Σ + friction capture
|
||||
|
||||
Cross-reference anchors (read-only, do not edit):
|
||||
- `ail-embed/src/adapter.rs:62-69` — `fold_symbol` (the no-window sibling `fold_window` mirrors); `MidPriceStream` `:24-60`
|
||||
- `ail-embed/src/lib.rs:81-100` — `Kernel`/`Kernel::run<I:IntoIterator<Item=f64>>` (unchanged; reused)
|
||||
- `ail-embed/src/bin/swarm_runner.rs:1-73` — symbol-fan runner shape `timeshard_runner` parallels (NOT modified)
|
||||
- `ail-embed/tests/swarm.rs:42-44,65-86` — `skip_if_no_data` / `run_swarm_or_skip` idioms `timeshard.rs` mirrors (NOT modified)
|
||||
- `../../libs/data-server/src/lib.rs:241-268` — `stream_tick_windowed(self:&Arc<Self>,&str,Option<i64>,Option<i64>)->Option<SymbolChunkIter<TickParsed>>`; `filter_files:274-308` (month-granularity inclusive window)
|
||||
- `../../libs/data-server/src/lib.rs:696-699` — the `chrono::NaiveDate::from_ymd_opt(..).and_hms_opt(..).and_utc().timestamp_millis()` bound precedent
|
||||
- `docs/DESIGN.md:2358-2360` — pre-existing "additive M4 concern" drift; spec §"Out of scope" — NOT m5.3's to fix, audit must not charge it here
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Windowed adapter + chrono dev-dep + bin declaration
|
||||
|
||||
**Files:**
|
||||
- Modify: `ail-embed/Cargo.toml`
|
||||
- Modify: `ail-embed/src/adapter.rs`
|
||||
|
||||
- [ ] **Step 1: Add `fold_window` to `ail-embed/src/adapter.rs`**
|
||||
|
||||
Append to `ail-embed/src/adapter.rs`, immediately after the
|
||||
`fold_symbol` function (currently ends at `:69`), *before* the
|
||||
`#[cfg(test)] mod tests` block:
|
||||
|
||||
```rust
|
||||
/// Fold the mid-price stream of `symbol` restricted to the inclusive
|
||||
/// Unix-millisecond window `[from_ms, to_ms]` through the M3 kernel.
|
||||
/// `None` if the symbol has no tick stream in that window. The window
|
||||
/// is the bound (no tick cap): the result is `(Σ mid, count)` over
|
||||
/// exactly the records data-server yields for that window, in stream
|
||||
/// order — identical f64-add sequence to a single-thread host fold of
|
||||
/// the same window, hence bit-exact against it. The windowed sibling
|
||||
/// of `fold_symbol`; same `MidPriceStream` + `Kernel` path, only the
|
||||
/// data-server entry differs (`stream_tick_windowed`).
|
||||
pub fn fold_window(
|
||||
server: &Arc<DataServer>,
|
||||
symbol: &str,
|
||||
from_ms: i64,
|
||||
to_ms: i64,
|
||||
) -> Option<(f64, i64)> {
|
||||
let it = server.stream_tick_windowed(symbol, Some(from_ms), Some(to_ms))?;
|
||||
Some(Kernel::new().run(MidPriceStream::new(it)))
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Add `chrono` dev-dep + the `timeshard_runner` bin to `ail-embed/Cargo.toml`**
|
||||
|
||||
The exact current tail of `ail-embed/Cargo.toml` is:
|
||||
|
||||
```toml
|
||||
[dev-dependencies]
|
||||
zip = "2"
|
||||
tempfile = "3"
|
||||
|
||||
[[bin]]
|
||||
name = "swarm_runner"
|
||||
path = "src/bin/swarm_runner.rs"
|
||||
```
|
||||
|
||||
Replace it verbatim with:
|
||||
|
||||
```toml
|
||||
[dev-dependencies]
|
||||
zip = "2"
|
||||
tempfile = "3"
|
||||
# Test-only: the time-shard test derives (year,month) -> Unix-ms
|
||||
# window bounds via chrono (data-server's own bound precedent,
|
||||
# data-server/src/lib.rs:696-699) and passes raw ms to the bin, so
|
||||
# the bin itself stays date-math-free. chrono is already transitively
|
||||
# present via data-server; this is the test-direct declaration.
|
||||
chrono = "0.4"
|
||||
|
||||
[[bin]]
|
||||
name = "swarm_runner"
|
||||
path = "src/bin/swarm_runner.rs"
|
||||
|
||||
[[bin]]
|
||||
name = "timeshard_runner"
|
||||
path = "src/bin/timeshard_runner.rs"
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Verify the crate builds and m5.1/m5.2 stay green (no regression)**
|
||||
|
||||
Run: `cargo test --manifest-path ail-embed/Cargo.toml --lib`
|
||||
Expected: PASS — `kernel_run_sums_prices` + `adapter::tests::tick_to_px_is_mid` `ok`; `0 failed`. (`fold_window` is additive; `[[bin]] timeshard_runner` references a not-yet-created file but `--lib` does not build bins, so this isolates the adapter/lib regression check. The bin + full-suite gate is Task 3 Step 4.)
|
||||
|
||||
---
|
||||
|
||||
### Task 2: The time-shard E2E test (RED — bin does not exist yet)
|
||||
|
||||
**Files:**
|
||||
- Create: `ail-embed/tests/timeshard.rs`
|
||||
|
||||
- [ ] **Step 1: Write `ail-embed/tests/timeshard.rs`**
|
||||
|
||||
```rust
|
||||
//! Time-shard boundary-invisibility E2E (spec Architecture §3
|
||||
//! second assertion / Testing strategy §3). One symbol, N disjoint
|
||||
//! single-month windows, one thread per window (each owning its
|
||||
//! `Kernel` — `Ctx: !Send`). Asserts, in spec strength order:
|
||||
//! (a) each shard's `(acc,n)` == a single-thread host fold of THAT
|
||||
//! EXACT window in stream order, BIT-EXACT — the
|
||||
//! boundary-invisibility proof (the kernel result for a window
|
||||
//! is independent of how data-server chunked it);
|
||||
//! (b) the host-summed `(Σacc,Σn)` == the identically-summed
|
||||
//! reference partials, bit-exact by construction;
|
||||
//! (c) the whole-window single-thread total within an
|
||||
//! f64-reassociation tolerance only (cross-shard re-association
|
||||
//! is host arithmetic, NOT a kernel property — asserting it
|
||||
//! bit-exact would be WRONG, per the spec).
|
||||
//! Plus the global leak-Σ (deterministic since dbd76e5) and a
|
||||
//! journal-only friction timing capture (NO bench gate).
|
||||
//!
|
||||
//! Concrete pin: EURUSD, months 2017-03 / 2017-04 / 2017-05 (all
|
||||
//! present on /mnt, contiguous, no gaps — recon-verified). data-server
|
||||
//! has no public month-listing API, so the months are hard-pinned.
|
||||
//! Skips when /mnt absent — data-server's own precedent.
|
||||
|
||||
use std::process::Command;
|
||||
use std::sync::Arc;
|
||||
|
||||
use chrono::{NaiveDate, TimeZone, Utc};
|
||||
use data_server::{DataServer, DEFAULT_DATA_PATH};
|
||||
|
||||
const SYMBOL: &str = "EURUSD";
|
||||
/// Three disjoint single-month windows. Contiguous so the whole-window
|
||||
/// union is exactly 2017-03-01 .. 2017-05-31 23:59:59.999.
|
||||
const SHARDS: [(i32, u32); 3] = [(2017, 3), (2017, 4), (2017, 5)];
|
||||
/// Relative tolerance for the (c) whole-window cross-check. Derivation:
|
||||
/// recursive f64 summation of n terms has error <= (n-1)*eps*Σ|x_i|.
|
||||
/// One EURUSD month ~250k ticks => 3 shards ~750k adds of O(1) mids;
|
||||
/// Σ|x| ~ 8.3e5, (n-1)*eps ~ 8.3e-11 => abs err ~ 6.8e-5, rel ~ 8e-11.
|
||||
/// 1e-6 is ~4 orders of safety yet still catches any real bug
|
||||
/// (a genuine miscount is off by >> 1 ppm).
|
||||
const REL_TOL: f64 = 1e-6;
|
||||
|
||||
fn skip_if_no_data() -> bool {
|
||||
!std::path::Path::new(DEFAULT_DATA_PATH).exists()
|
||||
}
|
||||
|
||||
/// Inclusive Unix-ms bounds for the whole of calendar month `(y, m)`.
|
||||
/// `filter_files` windows at month granularity via
|
||||
/// `unix_ms_to_year_month`, and `filter_chunk` is record-level
|
||||
/// inclusive `[from, to]`; first-instant .. last-instant of the month
|
||||
/// selects exactly that month's file and all its records.
|
||||
fn month_window(y: i32, m: u32) -> (i64, i64) {
|
||||
let from = Utc
|
||||
.from_utc_datetime(
|
||||
&NaiveDate::from_ymd_opt(y, m, 1)
|
||||
.unwrap()
|
||||
.and_hms_opt(0, 0, 0)
|
||||
.unwrap(),
|
||||
)
|
||||
.timestamp_millis();
|
||||
let (ny, nm) = if m == 12 { (y + 1, 1) } else { (y, m + 1) };
|
||||
let next_from = Utc
|
||||
.from_utc_datetime(
|
||||
&NaiveDate::from_ymd_opt(ny, nm, 1)
|
||||
.unwrap()
|
||||
.and_hms_opt(0, 0, 0)
|
||||
.unwrap(),
|
||||
)
|
||||
.timestamp_millis();
|
||||
(from, next_from - 1)
|
||||
}
|
||||
|
||||
/// Single-thread host reference: fold the mid stream over EXACTLY the
|
||||
/// inclusive `[from_ms, to_ms]` window in data-server stream order
|
||||
/// (`acc += mid; n += 1`). Bit-exact with `adapter::fold_window`
|
||||
/// because the f64-add order is identical.
|
||||
fn reference_window(server: &Arc<DataServer>, from_ms: i64, to_ms: i64) -> (f64, i64) {
|
||||
let mut it = server
|
||||
.stream_tick_windowed(SYMBOL, Some(from_ms), Some(to_ms))
|
||||
.expect("EURUSD has a tick stream in the pinned window");
|
||||
let (mut acc, mut n) = (0.0_f64, 0_i64);
|
||||
while let Some(chunk) = it.next_chunk() {
|
||||
for r in chunk.iter() {
|
||||
acc += (r.ask + r.bid) / 2.0;
|
||||
n += 1;
|
||||
}
|
||||
}
|
||||
(acc, n)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn timeshard_boundary_invisibility_and_leak_free() {
|
||||
if skip_if_no_data() {
|
||||
eprintln!("skipping: {DEFAULT_DATA_PATH} absent (mirrors data-server)");
|
||||
return;
|
||||
}
|
||||
|
||||
let windows: Vec<(i64, i64)> =
|
||||
SHARDS.iter().map(|&(y, m)| month_window(y, m)).collect();
|
||||
|
||||
// Spawn the time-shard runner: <data_dir> <symbol> then one
|
||||
// "from:to" arg per shard. It folds each window on its own thread
|
||||
// (own Kernel) and prints `RESULT <idx> <from> <to> <accbits> <n>`
|
||||
// + a final `TIMING total_ticks=<N> elapsed_ms=<M>` line.
|
||||
let mut cmd = Command::new(env!("CARGO_BIN_EXE_timeshard_runner"));
|
||||
cmd.arg(DEFAULT_DATA_PATH).arg(SYMBOL);
|
||||
for (f, t) in &windows {
|
||||
cmd.arg(format!("{f}:{t}"));
|
||||
}
|
||||
let out = cmd
|
||||
.env("AILANG_RC_STATS", "1")
|
||||
.output()
|
||||
.expect("spawn timeshard_runner");
|
||||
let stdout = String::from_utf8_lossy(&out.stdout);
|
||||
let stderr = String::from_utf8_lossy(&out.stderr);
|
||||
assert!(
|
||||
out.status.success(),
|
||||
"timeshard_runner exited {:?}\nstdout:\n{stdout}\nstderr:\n{stderr}",
|
||||
out.status.code()
|
||||
);
|
||||
|
||||
let server = Arc::new(DataServer::new(DEFAULT_DATA_PATH));
|
||||
|
||||
// Parse RESULT lines into (idx -> (from,to,acc,n)).
|
||||
let mut shards: Vec<(usize, i64, i64, f64, i64)> = Vec::new();
|
||||
for line in stdout.lines().filter(|l| l.starts_with("RESULT ")) {
|
||||
let mut t = line.split_whitespace();
|
||||
let _ = t.next(); // RESULT
|
||||
let idx: usize = t.next().unwrap().parse().unwrap();
|
||||
let from: i64 = t.next().unwrap().parse().unwrap();
|
||||
let to: i64 = t.next().unwrap().parse().unwrap();
|
||||
let acc = f64::from_bits(
|
||||
u64::from_str_radix(t.next().unwrap(), 16).unwrap(),
|
||||
);
|
||||
let n: i64 = t.next().unwrap().parse().unwrap();
|
||||
shards.push((idx, from, to, acc, n));
|
||||
}
|
||||
shards.sort_by_key(|s| s.0);
|
||||
assert_eq!(
|
||||
shards.len(),
|
||||
SHARDS.len(),
|
||||
"expected {} shard RESULT lines, got {}",
|
||||
SHARDS.len(),
|
||||
shards.len()
|
||||
);
|
||||
|
||||
// (a) Per-shard BIT-EXACT vs a single-thread host fold of that
|
||||
// exact window. THE boundary-invisibility proof.
|
||||
let mut sum_acc = 0.0_f64;
|
||||
let mut sum_n = 0_i64;
|
||||
let mut ref_sum_acc = 0.0_f64;
|
||||
let mut ref_sum_n = 0_i64;
|
||||
for (i, &(idx, from, to, acc, n)) in shards.iter().enumerate() {
|
||||
assert_eq!(idx, i, "shard indices contiguous 0..N");
|
||||
assert_eq!((from, to), windows[i], "shard {i} window matches pin");
|
||||
let (r_acc, r_n) = reference_window(&server, from, to);
|
||||
assert_eq!(
|
||||
acc.to_bits(),
|
||||
r_acc.to_bits(),
|
||||
"shard {i} ({from}..{to}): kernel acc BIT-EXACT vs \
|
||||
single-thread host fold of that exact window \
|
||||
(boundary-invisibility)"
|
||||
);
|
||||
assert_eq!(n, r_n, "shard {i}: tick count == reference");
|
||||
assert!(r_n > 0, "shard {i}: non-empty window");
|
||||
sum_acc += acc;
|
||||
sum_n += n;
|
||||
ref_sum_acc += r_acc;
|
||||
ref_sum_n += r_n;
|
||||
}
|
||||
|
||||
// (b) Host-summed partials == identically-summed reference
|
||||
// partials, BIT-EXACT by construction (same values, same
|
||||
// host summation order).
|
||||
assert_eq!(
|
||||
sum_acc.to_bits(),
|
||||
ref_sum_acc.to_bits(),
|
||||
"Σ shard acc bit-exact vs Σ reference partials"
|
||||
);
|
||||
assert_eq!(sum_n, ref_sum_n, "Σ shard n == Σ reference n");
|
||||
|
||||
// (c) Whole-window single-thread total within REL_TOL only —
|
||||
// cross-shard re-association is host arithmetic, NOT a kernel
|
||||
// property; asserting bit-exact here would be WRONG (spec §3).
|
||||
let whole_from = windows.first().unwrap().0;
|
||||
let whole_to = windows.last().unwrap().1;
|
||||
let (whole_acc, whole_n) = reference_window(&server, whole_from, whole_to);
|
||||
assert_eq!(
|
||||
whole_n, sum_n,
|
||||
"whole-window tick count == Σ shard n (windows are an exact \
|
||||
disjoint partition of the union)"
|
||||
);
|
||||
let rel = ((whole_acc - sum_acc) / whole_acc).abs();
|
||||
assert!(
|
||||
rel < REL_TOL,
|
||||
"whole-window total {whole_acc} vs Σ partials {sum_acc}: \
|
||||
relative diff {rel:e} must be < {REL_TOL:e} (f64 \
|
||||
reassociation only — NOT a kernel discrepancy)"
|
||||
);
|
||||
|
||||
// Global leak-Σ across ALL ailang_rc_stats: lines (deterministic
|
||||
// since dbd76e5: atomic global counters + build.rs runtime-track).
|
||||
let mut allocs: u64 = 0;
|
||||
let mut frees: u64 = 0;
|
||||
let mut seen = 0usize;
|
||||
for line in stderr.lines().filter(|l| l.starts_with("ailang_rc_stats:")) {
|
||||
seen += 1;
|
||||
for tok in line.split_whitespace() {
|
||||
if let Some(v) = tok.strip_prefix("allocs=") {
|
||||
allocs += v.parse::<u64>().expect("allocs= u64");
|
||||
} else if let Some(v) = tok.strip_prefix("frees=") {
|
||||
frees += v.parse::<u64>().expect("frees= u64");
|
||||
}
|
||||
}
|
||||
}
|
||||
assert!(seen > 0, "no ailang_rc_stats line; stderr:\n{stderr}");
|
||||
assert_eq!(
|
||||
allocs, frees,
|
||||
"globally leak-free: Σallocs==Σfrees across {seen} stat line(s)"
|
||||
);
|
||||
|
||||
// Friction harvest: journal-only observation, NO assertion (the
|
||||
// value feeds the P2 batch-crossing decision; it is not a gate).
|
||||
if let Some(tl) = stdout.lines().find(|l| l.starts_with("TIMING ")) {
|
||||
eprintln!("m5.3 friction-harvest: {tl} (Σn={sum_n})");
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run the test — verify it is RED (bin does not exist)**
|
||||
|
||||
Run: `cargo test --manifest-path ail-embed/Cargo.toml --test timeshard`
|
||||
Expected: FAIL to compile — `error: environment variable
|
||||
\`CARGO_BIN_EXE_timeshard_runner\` not defined at compile time`
|
||||
(the `[[bin]] timeshard_runner` target's source file does not exist
|
||||
yet, so Cargo does not set the env var `env!` reads). This is the
|
||||
deterministic RED.
|
||||
|
||||
---
|
||||
|
||||
### Task 3: The `timeshard_runner` bin (GREEN)
|
||||
|
||||
**Files:**
|
||||
- Create: `ail-embed/src/bin/timeshard_runner.rs`
|
||||
|
||||
- [ ] **Step 1: Write `ail-embed/src/bin/timeshard_runner.rs`**
|
||||
|
||||
```rust
|
||||
//! Time-shard swarm harness (spec §3). Run as a subprocess by
|
||||
//! `tests/timeshard.rs` under `AILANG_RC_STATS=1`.
|
||||
//!
|
||||
//! argv: `<data_dir> <symbol> <from:to> [<from:to> ...]` — one
|
||||
//! inclusive Unix-ms window per shard. Date math lives in the test
|
||||
//! (chrono); this bin is date-math-free. Spawns one thread per
|
||||
//! window, each building its OWN `Kernel` (`Ctx: !Send` forces
|
||||
//! one-ctx-per-thread — a shared-ctx design is `error[E0277]`).
|
||||
//! Prints, in shard order, `RESULT <idx> <from> <to> <accbits:016x>
|
||||
//! <n>`, then one `TIMING total_ticks=<Σn> elapsed_ms=<wall>` line
|
||||
//! (the friction-harvest observation; the test records it in the
|
||||
//! close-out journal, it is not asserted).
|
||||
|
||||
use std::sync::Arc;
|
||||
use std::time::Instant;
|
||||
|
||||
use ail_embed::adapter::fold_window;
|
||||
use data_server::DataServer;
|
||||
|
||||
fn main() {
|
||||
let mut args = std::env::args().skip(1);
|
||||
let data_dir = args.next().expect("argv[1] = data dir");
|
||||
let symbol = args.next().expect("argv[2] = symbol");
|
||||
let windows: Vec<(i64, i64)> = args
|
||||
.map(|a| {
|
||||
let (f, t) = a.split_once(':').expect("window arg is from:to");
|
||||
(
|
||||
f.parse::<i64>().expect("from_ms i64"),
|
||||
t.parse::<i64>().expect("to_ms i64"),
|
||||
)
|
||||
})
|
||||
.collect();
|
||||
assert!(
|
||||
windows.len() >= 2,
|
||||
"need >= 2 disjoint windows for a time-shard, got {}",
|
||||
windows.len()
|
||||
);
|
||||
|
||||
let server = Arc::new(DataServer::new(&data_dir));
|
||||
|
||||
let started = Instant::now();
|
||||
let handles: Vec<_> = windows
|
||||
.iter()
|
||||
.copied()
|
||||
.enumerate()
|
||||
.map(|(idx, (from, to))| {
|
||||
let srv = Arc::clone(&server);
|
||||
let sym = symbol.clone();
|
||||
// Each thread builds its own Kernel inside the closure —
|
||||
// Kernel/Ctx are !Send, so this is the ONLY shape that
|
||||
// compiles (one ctx per thread, type-enforced).
|
||||
std::thread::spawn(move || {
|
||||
let (acc, n) = fold_window(&srv, &sym, from, to)
|
||||
.expect("window has a tick stream");
|
||||
(idx, from, to, acc, n)
|
||||
})
|
||||
})
|
||||
.collect();
|
||||
|
||||
let mut results: Vec<(usize, i64, i64, f64, i64)> = handles
|
||||
.into_iter()
|
||||
.map(|h| h.join().expect("shard thread panicked"))
|
||||
.collect();
|
||||
let elapsed_ms = started.elapsed().as_millis();
|
||||
results.sort_by_key(|r| r.0);
|
||||
|
||||
let mut total_ticks: i64 = 0;
|
||||
for (idx, from, to, acc, n) in &results {
|
||||
total_ticks += *n;
|
||||
println!("RESULT {idx} {from} {to} {:016x} {n}", acc.to_bits());
|
||||
}
|
||||
println!("TIMING total_ticks={total_ticks} elapsed_ms={elapsed_ms}");
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run the time-shard test — verify GREEN**
|
||||
|
||||
Run: `cargo test --manifest-path ail-embed/Cargo.toml --test timeshard`
|
||||
Expected: PASS — `test timeshard_boundary_invisibility_and_leak_free
|
||||
... ok`; `1 passed; 0 failed`. The `eprintln!` shows the
|
||||
friction-harvest `TIMING` line (captured for the journal).
|
||||
|
||||
- [ ] **Step 3: Determinism — run the time-shard test ×5**
|
||||
|
||||
Run: `for i in 1 2 3 4 5; do cargo test --manifest-path ail-embed/Cargo.toml --test timeshard 2>&1 | grep 'test result:'; done`
|
||||
Expected: `test result: ok. 1 passed; 0 failed; 0 ignored; ...`
|
||||
**every** run (5/5). The leak-Σ is deterministic post-`dbd76e5`;
|
||||
per-shard bit-exactness is deterministic by same-order f64 adds.
|
||||
Any FAIL here is a hard stop (the leak-proof saga's lesson: one
|
||||
green run is insufficient).
|
||||
|
||||
- [ ] **Step 4: Full ail-embed suite + the M5 acceptance gates**
|
||||
|
||||
Run: `cargo test --manifest-path ail-embed/Cargo.toml`
|
||||
Expected: PASS — lib (`kernel_run_sums_prices`,
|
||||
`adapter::tests::tick_to_px_is_mid`), `smoke`
|
||||
(`hermetic_smoke_data_server_roundtrip`), `swarm`
|
||||
(`symbol_fan_swarm_bit_exact`, `symbol_fan_swarm_leak_free`),
|
||||
`timeshard` (`timeshard_boundary_invisibility_and_leak_free`),
|
||||
`rt_archive_freshness` all `ok`; **0 failed, 0 ignored** across all
|
||||
test binaries (the m5.2 leak-proof stays green; nothing
|
||||
re-quarantined).
|
||||
|
||||
- [ ] **Step 5: Invariant 1 + no-compiler-surface-diff gate**
|
||||
|
||||
Run: `cargo metadata --format-version 1 --manifest-path Cargo.toml --no-deps 2>/dev/null | grep -c data-server | cat`
|
||||
Expected: `0` — the AILang workspace graph has no `data-server`
|
||||
(adding `chrono` as an `ail-embed` dev-dep does not enter it;
|
||||
`ail-embed` is its own workspace root).
|
||||
|
||||
Run: `git status --porcelain | grep -E '^\?\?|^ ?M' | grep -vE '^.. (ail-embed/|docs/|bench/orchestrator-stats/)'`
|
||||
Expected: empty output — m5.3 touches **only** `ail-embed/**`,
|
||||
`docs/`, and the stats file. **Zero** diff to `crates/ailang-*`,
|
||||
`crates/ail/`, `runtime/`, `examples/*.ail`, root `Cargo.toml`.
|
||||
|
||||
- [ ] **Step 6: Isolated runtime RED still green (no regression)**
|
||||
|
||||
Run: `cargo test -p ail --test embed_rc_global_stats_race 2>&1 | grep 'test result:'`
|
||||
Expected: `test result: ok. 1 passed; 0 failed; ...` — the
|
||||
runtime/build are untouched by m5.3; the audit-trail RED stays GREEN.
|
||||
|
||||
---
|
||||
|
||||
## Boss decisions (recorded for the audit)
|
||||
|
||||
- **Friction-harvest is journal-only.** `timeshard_runner` emits a
|
||||
`TIMING total_ticks=N elapsed_ms=M` line; the test `eprintln!`s it
|
||||
and the **Boss records the derived per-tick-FFI cost (elapsed /
|
||||
total_ticks) in the m5.3 close-out journal** as the P2
|
||||
flat-array-decision input. There is NO timing assertion and NO
|
||||
`bench/` baseline touched (recon confirmed no bench harness is
|
||||
invoked by `ail-embed`) — a wall-clock gate would flake; the spec
|
||||
(`:302-305`) is explicit it is an observation, not a perf gate.
|
||||
- **`chrono` is a dev-dependency only.** The date math lives in the
|
||||
test; the bin takes raw `from:to` ms args and is date-math-free.
|
||||
Invariant 1 is unaffected (compiler crates untouched; the AILang
|
||||
workspace graph stays `data-server`/`chrono`-free since `ail-embed`
|
||||
is its own workspace root — gated in Task 3 Step 5).
|
||||
- **New bin, not an arg-mode on `swarm_runner`.** Keeps the
|
||||
now-green m5.2 symbol-fan path byte-untouched (no regression
|
||||
surface) and each runner single-responsibility; its own
|
||||
`CARGO_BIN_EXE_timeshard_runner` is the established subprocess
|
||||
wiring.
|
||||
- **(c) is a tolerance check, never bit-exact.** Cross-shard
|
||||
re-association is host f64 arithmetic, not a kernel property; the
|
||||
spec and the m5-spec self-review are explicit. `REL_TOL = 1e-6` is
|
||||
derived from the recursive-summation error bound with ~4 orders of
|
||||
safety margin (shown in the test).
|
||||
- **Pre-existing DESIGN.md drift out of scope.** `docs/DESIGN.md:2358-2360`
|
||||
("additive M4 concern") is pre-existing M4-retirement drift, NOT
|
||||
m5.3's to fix; the audit must not charge it to m5.3 (spec §"Out of
|
||||
scope"). m5.3 ships zero `runtime/`/`crates/`/DESIGN.md change.
|
||||
- **Milestone close next.** m5.3 is the last functional M5 iteration;
|
||||
after Boss-verification the milestone closes and `audit` (mandatory)
|
||||
runs separately.
|
||||
|
||||
## Self-review (planner Step 5)
|
||||
|
||||
1. **Spec coverage.** Architecture §3 time-shard second assertion →
|
||||
Task 2/3 (per-shard bit-exact (a), Σ bit-exact (b), whole-window
|
||||
tolerance (c)). Testing §3 → `tests/timeshard.rs`. Friction-harvest
|
||||
(`:302-305`, Acceptance `:331-332`) → `TIMING` line + journal
|
||||
record (Boss decision). Leak-Σ acceptance (`:323-324`) → the
|
||||
leak-Σ block. Invariant 1 / no-surface-diff (`:309-314`,
|
||||
`:328-330`) → Task 3 Step 5. skip-if-absent (`:326-327`) →
|
||||
`skip_if_no_data`. Symbol-fan / hermetic smoke already shipped
|
||||
(m5.2/m5.1) — not re-implemented; verified non-regressed in Task 3
|
||||
Step 4. Audit runs separately (not an m5.3 task).
|
||||
2. **Placeholder scan.** No "TBD/TODO/implement later/similar
|
||||
to/add appropriate". `SYMBOL`/`SHARDS`/`REL_TOL` are concrete
|
||||
pinned constants with recon-verified data + a derived tolerance,
|
||||
not placeholders.
|
||||
3. **Type/name consistency.** `fold_window`, `MidPriceStream`,
|
||||
`Kernel`, `timeshard_runner`, `CARGO_BIN_EXE_timeshard_runner`,
|
||||
`RESULT <idx> <from> <to> <accbits:016x> <n>` (written in Task 3 /
|
||||
parsed in Task 2), `TIMING total_ticks=<N> elapsed_ms=<M>`,
|
||||
`month_window`, `reference_window`, `SHARDS=[(2017,3),(2017,4),(2017,5)]`,
|
||||
`SYMBOL="EURUSD"`, `REL_TOL=1e-6`,
|
||||
`timeshard_boundary_invisibility_and_leak_free` — identical across
|
||||
tasks and verification filters.
|
||||
4. **Step granularity.** Each step is one file write or one command;
|
||||
2-5 min. Largest paste (Task 2 Step 1) is a single verbatim test.
|
||||
5. **No commit steps.** None. Work stays in the working tree.
|
||||
6. **Pin/replacement substring contiguity.** The one verbatim
|
||||
replacement (Task 1 Step 2, the `[dev-dependencies]`/`[[bin]]`
|
||||
block) is not paired with any `contains(...)`/grep over its own
|
||||
body. The greps (Task 3 Step 5) assert over `cargo metadata` /
|
||||
`git status` output, not over a verbatim text body this plan
|
||||
writes — no contiguity hazard.
|
||||
7. **Compile-gate vs deferred-caller.** No signature change to an
|
||||
existing function (only the additive `fold_window`; `fold_symbol`
|
||||
and all callers untouched). Task 1 Step 3 uses `--lib` precisely
|
||||
because the `[[bin]] timeshard_runner` path does not exist until
|
||||
Task 3 — a full `cargo test` in Task 1 would fail to build the
|
||||
declared bin. The full-suite/bin gate is correctly placed in
|
||||
Task 3 Step 4 (after the bin exists). This ordering is deliberate
|
||||
and internally consistent — `--lib` is satisfiable at Task 1
|
||||
(bins are not built by `--lib`); the RED in Task 2 Step 2 is the
|
||||
*intended* `env!` compile failure, not a plan defect.
|
||||
8. **Verification filter strings resolve.** `--lib`,
|
||||
`--test timeshard`, `--test embed_rc_global_stats_race`,
|
||||
`timeshard_boundary_invisibility_and_leak_free` are all defined
|
||||
*by this plan* (or are the shipped isolated RED) in the same
|
||||
tasks that filter on them — resolve by construction. Task 3
|
||||
Steps 4/5/6 use the unfiltered suite or explicit `grep -c … == 0`
|
||||
/ empty-output / `1 passed` assertions, so "nothing ran" cannot
|
||||
masquerade as success. Task 2 Step 2's "RED" is a *compile*
|
||||
failure (deterministic, not a 0-filtered false-green).
|
||||
Reference in New Issue
Block a user