Final functional M5 iteration (spec ae905de, plan 67027ab; on
green m5.1 204c171 + m5.2 b724cd1 whose leak-proof resolved via
dbd76e5). Proves the chunk/window-boundary-invisibility claim the
M4 retirement rests on:
- adapter::fold_window — purely additive windowed sibling of
fold_symbol (same MidPriceStream+Kernel path; stream_tick_windowed).
fold_symbol/MidPriceStream/Kernel byte-untouched.
- chrono as a DEV-dep only — the time-shard test derives
(y,m)->Unix-ms bounds (data-server's own precedent) and passes
raw ms to the bin; the bin is date-math-free; Invariant 1
unaffected.
- new single-responsibility [[bin]] timeshard_runner — one thread
per disjoint window, each owning its Kernel (Ctx:!Send =>
compile-time per-thread-ctx).
- tests/timeshard.rs — spec §3 strength-ordered: (a) per-shard
(acc,n) BIT-EXACT vs single-thread host fold of that exact window
in stream order (THE boundary-invisibility proof); (b) Σ partials
bit-exact by construction; (c) whole-window within REL_TOL=1e-6
only (cross-shard f64 reassociation is host arithmetic, never a
kernel property — bit-exact there would be the bug; tolerance
derived from the recursive-summation error bound). Plus the
now-deterministic leak-Σ + a journal-only friction timing capture
(no bench gate, no timing assertion).
Concrete pin: EURUSD 2017-03/04/05 (recon-verified present,
contiguous, no gaps; data-server has no month-listing API). RED was
a genuine deterministic compile failure (declared-but-absent bin
source; Cargo missing-source error vs the plan's predicted env!
error — same cause, no false-green; benign plan-text imprecision in
Concerns).
Boss-verified independently: timeshard determinism 5/5 (no jitter);
full ail-embed suite 0 failed AND 0 ignored across all binaries
(m5.2 symbol_fan_swarm_leak_free stays un-ignored & green — the
saga's resolution intact); isolated embed_rc_global_stats_race
still green; AILang workspace data-server count 0; compiler-surface
diff empty (zero diff to crates/ailang-*, crates/ail/, runtime/,
examples/*.ail, root Cargo.toml, DESIGN.md).
Friction-harvest deliverable (P2 flat-array-decision input):
host-per-tick-FFI ~= 658 ms / 3,192,562 ticks ~= ~206 ns/tick at
real EURUSD volume.
M5 functional work complete; milestone closes next via the
mandatory audit (which also handles the pre-existing
DESIGN.md:2358-2360 "additive M4 concern" drift, out of m5.3 scope).
Includes the per-iter journal, stats, and the INDEX.md line.
bugfix-swarm-rc-alloc-undercount, GREEN stage. RED is the separate
audit-trail commit 483117d.
Root cause of the m5.2-resume-attempt residual: ail-embed/build.rs
declared cargo:rerun-if-changed only for the kernel .ail + build.rs
itself — NOT the runtime/ C sources ail build --emit=staticlib
compiles into libailang_rt.a. Cargo never re-ran build.rs after the
atomic-counter fix 7bfa11e, so ail-embed kept linking a stale
pre-7bfa11e non-atomic libailang_rt.a; the swarm's concurrent
TLS-NULL host allocs raced the stale non-atomic g_rc_alloc_count++
(frees-stable/allocs-jitter). NOT a second runtime bug and NOT
test-methodology unsoundness — runtime/rc.c at HEAD is correct; the
isolated crates/ail RED was green only because it rebuilds the
staticlib fresh per run.
Fix (ail-embed/build.rs, +7 lines): directory-level
cargo:rerun-if-changed=<repo>/runtime (Cargo recurses mtimes under
the dir). Preferred over a per-file list — the implementer inspected
crates/ail build_staticlib (libailang_rt.a = ar(rc.o, str.o), both
under runtime/) and chose the directory mechanism so any future
runtime source cannot silently re-introduce the same staleness.
Existing kernel/build.rs/AIL_BIN rerun-if-changed lines kept.
Cohesive consequence (in-scope): symbol_fan_swarm_leak_free
un-#[ignore]d — the test BODY is byte-for-byte unchanged (it was
quarantined, never weakened); the module-doc + fn-doc breadcrumbs
rewritten to the resolved build-dep-staleness rationale. It is now
the integration-level acceptance of both 7bfa11e and this fix. Boss
also removed a pre-existing unused `DataFormat` import in swarm.rs
inline (trivial; the file is committed this iter regardless;
warning-clean after).
Boss-verified independently: RED rt_archive_freshness -> GREEN;
swarm determinism 5/5 (symbol_fan_swarm_leak_free +
symbol_fan_swarm_bit_exact GREEN every run, 0 ignored, jitter gone,
Sallocs==Sfrees==12000003); isolated embed_rc_global_stats_race
still GREEN (untouched, runtime untouched); Invariant 1 data-server
count 0; scope = ail-embed/build.rs + ail-embed/tests/swarm.rs +
journal + stats only.
The M5 swarm leak-proof bounce-back is fully resolved end-to-end.
M5 stays open [~]; m5.3 (time-shard + friction-harvest + close-out)
remains. Includes the per-iter journal, stats, and INDEX.md line.
bugfix-swarm-rc-alloc-undercount, RED stage (audit trail; GREEN
follows separately via implement mini-mode).
Root cause of the m5.2-resume-attempt residual (the M5 swarm
leak-proof still non-deterministic after the atomic-counter fix
7bfa11e): ail-embed/build.rs declares cargo:rerun-if-changed only
for the kernel .ail + build.rs itself — NOT runtime/rc.c. So Cargo
never re-ran build.rs after 7bfa11e; the swarm keeps linking a
stale pre-7bfa11e libailang_rt.a whose g_rc_*count++ is the old
non-atomic racing increment. 7bfa11e was correct and is unchanged;
it just never reached the ail-embed binary. This explains why the
isolated crates/ail RED (rebuilds the staticlib fresh per run) is
green while the ail-embed swarm is not, and the
frees-stable/allocs-jitter asymmetry (few TLS-NULL frees = no
contention; millions of TLS-NULL host allocs = race on the stale
non-atomic counter).
Framing: a genuine defect (build-dependency completeness gap), NOT
test-methodology unsoundness — with a fresh archive the swarm
Σ-over-stat-lines balances deterministically (Σallocs==Σfrees==
12000003, 5/5 per the debugger). runtime/rc.c at HEAD is correct;
the GREEN fix is in the build script, not the runtime.
RED pins the CAUSE deterministically (the symptom is
non-deterministic by construction): objdump the libailang_rt.a that
build.rs emitted for this crate; assert >=2 lock-prefixed
instructions (the atomic lock incq for g_rc_alloc_count +
g_rc_free_count from current rc.c) and that it matches a
freshly-built archive. Boss-verified RED: linked archive has 0 lock
insns ("STALE pre-7bfa11e archive linked").
Does NOT touch the committed #[ignore] of symbol_fan_swarm_leak_free
nor the isolated embed_rc_global_stats_race RED (both correct,
unchanged).
Attempted to un-#[ignore] the M5 swarm leak-proof as the
integration-level acceptance of bugfix-rc-global-stats-race
(7bfa11e). Boss verification falsified the sufficiency premise:
across 4-6 swarm runs the leak Σ is STILL non-deterministic —
Σfrees stable-exact (12000003) every run, Σallocs short by a
jittering ~1600-2260 in ~1/3 of runs (e.g. left:11998383
right:12000003, 4 stat lines).
The atomic-global runtime fix (7bfa11e) is real, committed, and
NECESSARY — the isolated 8x2M pure-global RED
embed_rc_global_stats_race is deterministically green. It is NOT
sufficient for the swarm: a second, distinct alloc-side undercount
remains that the isolated RED did not model (Σfrees never loses =>
not a per-ctx race; not pure global contention => RED green).
The un-#[ignore] was reverted (never committed). The ONLY shipped
diff is the three now-stale #[ignore]-rationale breadcrumbs in
ail-embed/tests/swarm.rs corrected from "un-ignore when the runtime
makes the counters atomic" to the accurate necessary-but-insufficient
state — a future agent seeing 7bfa11e must NOT un-ignore on that
basis (doc-honesty rule). The test body and the #[ignore] attribute
itself are unchanged: still quarantined, NOT weakened.
symbol_fan_swarm_bit_exact stays live and GREEN — the swarm IS
actually correct and leak-free; only the accounting is wrong.
Residual handed to a fresh debug RED-first cycle. Two framings
recorded for it (genuine residual runtime defect vs Σ-over-
heterogeneous-lines test-methodology unsoundness; the latter, if so,
is a post-root-cause design decision). main stays green; M5 open
[~]; m5.3 remains downstream of a sound leak-proof.
Includes the resume-attempt journal + INDEX line.
M5 iteration 2 (spec ae905de, plan 9cc9d9c). Tasks 1+2 clean and
committable; Task 3 BLOCKED on a real finding → M5 bounce-back.
Shipped:
- data-server promoted [dev-dependencies] -> [dependencies]
(Invariant-1 sanctioned: ail-embed is the sole meeting point; the
AILang workspace graph still has data-server count 0; zero
compiler-surface diff — not even root Cargo.toml this iter).
- additive `adapter` module (ail-embed/src/adapter.rs): tick_to_px,
MidPriceStream (lazy Iterator<f64> over SymbolChunkIter),
fold_symbol; RED-first (E0432 -> GREEN). m5.1 core untouched
except `pub mod adapter;`.
- swarm_runner [[bin]]: one thread per symbol, each owning its
Kernel. The clean compile IS the compile-time per-thread-ctx
proof (Ctx: !Send => a shared-ctx swarm is E0277).
- tests/swarm.rs: real-data symbol-fan E2E. `symbol_fan_swarm_bit_exact`
is GREEN and live — per-symbol kernel (acc,n) bit-exact vs an
independent same-order host reference (EURUSD/GER40/XAUUSD, ~4s).
The Task 3 finding (Boss independently confirmed by reading
runtime/rc.c): the global Σallocs==Σfrees leak measurement is an
INSTRUMENTATION race, not a memory bug. No box crosses a thread
(Ctx: !Send), the real refcount/free is correct (bit-exact GREEN
every run); only the global g_rc_* stat counters (rc.c:90-91,
161,212) are non-atomic BY rc.c's own documented single-threaded
design, losing ++s when 3 worker threads hit the null-__ail_tls_ctx
host-side path. M5 is AILang's first concurrent consumer; rc.c's
header (rc.c:44-49) explicitly defers exactly this atomic-vs-non-
atomic decision to "when it acquires concurrency primitives".
The leak assertion was Boss-split into `symbol_fan_swarm_leak_free`
(#[ignore], body preserved VERBATIM — quarantined not weakened;
un-ignore = the runtime fix's acceptance criterion) so main stays
green and the finding is pinned as a regression marker. The earlier
plan/journal claim that embed_swarm_tsan.rs covers this path was
wrong and is corrected on the record (that test uses the scalar
kernel — zero box allocs — so it never exercised the host-side
global-counter path).
Escalated as an M5 bounce-back: resolution touches M3-frozen
runtime/ and re-frames M5's "zero runtime change" commitment;
multiple substantive options, not unilaterally Boss's to pick in
frozen-runtime territory. Boss recommendation on the record =
Option A (make only the global-fallback g_rc_* counters atomic, as
a standalone RED-first runtime micro-iteration; M5 framing amended).
Time-shard + friction-harvest remain m5.3.
Includes the per-iter journal (with Boss disposition), stats, and
the INDEX.md line.
M5 iteration 1 (spec ae905de, plan 22f02aa). Stands up the
workspace-excluded `ail-embed` crate:
- zero-dependency embedding core (`ail-embed/src/lib.rs`): extern "C"
to the M3-frozen ABI + frozen-layout State/Tick box helpers + a
Kernel price fold; the Rust port of the audited
crates/ail/tests/embed/tick_roundtrip.c. Raw pointers never escape
the type.
- build.rs (no in-repo precedent): AIL_BIN env override else nested
`cargo build -p ail` against the parent workspace (separate target
dir → no cargo-lock deadlock), `ail build --emit=staticlib`, link
directives.
- hermetic data-server smoke (ail-embed/tests/smoke.rs): synthetic
Pepperstone-format ZIP fixture via data-server's own public
RawTickRecord type → real DataServer → Kernel, bit-exact vs a
same-order host reference fold; runs with no /mnt.
- `ail-embed` is its own cargo workspace root (empty [workspace]
table); data-server is a dev-dependency only. Root Cargo.toml gains
only a 4-line non-membership comment.
Invariant 1 Boss-verified independently: full+no-deps cargo metadata
on the AILang workspace shows data-server count 0; git status path
filter empty (zero diff to crates/ailang-*, crates/ail/, runtime/,
examples/*.ail); src/lib.rs zero code-level data_server; AILang
cargo build --workspace still clean. ail-embed suite 2/2 green
(kernel_run_sums_prices unit RED-first + hermetic_smoke integration),
verified by me, not just the agent report.
Two toolchain-forced corrections to the plan's verbatim
ail-embed/Cargo.toml (added empty [workspace] table; sibling dev-dep
path ../libs -> ../../libs, manifest-relative) — confined to the
plan-created manifest, no acceptance gate altered, 0 review re-loops.
Journal Concerns records the planner-recon implication for the next
workspace-excluded-nested-crate plan. Adapter API + thread-swarm
deferred to M5 iter 2+ per spec/plan.
Includes the per-iter journal, stats, and the INDEX.md line.
Brummel