Final functional M5 iteration (spec ae905de, plan 67027ab; on
green m5.1 204c171 + m5.2 b724cd1 whose leak-proof resolved via
dbd76e5). Proves the chunk/window-boundary-invisibility claim the
M4 retirement rests on:
- adapter::fold_window — purely additive windowed sibling of
fold_symbol (same MidPriceStream+Kernel path; stream_tick_windowed).
fold_symbol/MidPriceStream/Kernel byte-untouched.
- chrono as a DEV-dep only — the time-shard test derives
(y,m)->Unix-ms bounds (data-server's own precedent) and passes
raw ms to the bin; the bin is date-math-free; Invariant 1
unaffected.
- new single-responsibility [[bin]] timeshard_runner — one thread
per disjoint window, each owning its Kernel (Ctx:!Send =>
compile-time per-thread-ctx).
- tests/timeshard.rs — spec §3 strength-ordered: (a) per-shard
(acc,n) BIT-EXACT vs single-thread host fold of that exact window
in stream order (THE boundary-invisibility proof); (b) Σ partials
bit-exact by construction; (c) whole-window within REL_TOL=1e-6
only (cross-shard f64 reassociation is host arithmetic, never a
kernel property — bit-exact there would be the bug; tolerance
derived from the recursive-summation error bound). Plus the
now-deterministic leak-Σ + a journal-only friction timing capture
(no bench gate, no timing assertion).
Concrete pin: EURUSD 2017-03/04/05 (recon-verified present,
contiguous, no gaps; data-server has no month-listing API). RED was
a genuine deterministic compile failure (declared-but-absent bin
source; Cargo missing-source error vs the plan's predicted env!
error — same cause, no false-green; benign plan-text imprecision in
Concerns).
Boss-verified independently: timeshard determinism 5/5 (no jitter);
full ail-embed suite 0 failed AND 0 ignored across all binaries
(m5.2 symbol_fan_swarm_leak_free stays un-ignored & green — the
saga's resolution intact); isolated embed_rc_global_stats_race
still green; AILang workspace data-server count 0; compiler-surface
diff empty (zero diff to crates/ailang-*, crates/ail/, runtime/,
examples/*.ail, root Cargo.toml, DESIGN.md).
Friction-harvest deliverable (P2 flat-array-decision input):
host-per-tick-FFI ~= 658 ms / 3,192,562 ticks ~= ~206 ns/tick at
real EURUSD volume.
M5 functional work complete; milestone closes next via the
mandatory audit (which also handles the pre-existing
DESIGN.md:2358-2360 "additive M4 concern" drift, out of m5.3 scope).
Includes the per-iter journal, stats, and the INDEX.md line.
Final functional M5 iteration (spec ae905de; builds on green
m5.1 204c171 + m5.2 b724cd1 whose leak-proof is resolved through
dbd76e5). 3 tasks: (1) additive adapter::fold_window (windowed
sibling of fold_symbol) + chrono dev-dep + a second
[[bin]] timeshard_runner declaration; (2) tests/timeshard.rs
written FIRST — RED is the deterministic `env!(CARGO_BIN_EXE_
timeshard_runner) not defined` compile failure (bin absent); (3)
the timeshard_runner bin → GREEN, with the spec §3 strength-ordered
assertions (a) per-shard bit-exact vs single-thread windowed host
fold (THE boundary-invisibility proof), (b) Σ bit-exact by
construction, (c) whole-window within REL_TOL=1e-6 (derived from
the recursive-summation error bound, never bit-exact — spec
explicit), plus the now-deterministic leak-Σ and a journal-only
friction-timing capture.
Boss decisions in-plan: new bin not an arg-mode (shipped symbol-fan
path byte-untouched, no regression surface); chrono dev-only (date
math in the test, bin date-math-free; Invariant 1 unaffected);
concrete pin EURUSD 2017_03/04/05 (recon-verified present, no
gaps; data-server has no month-listing API); friction is a journal
observation, no bench gate, no timing assertion (flake-free);
pre-existing DESIGN.md:2358-2360 M4-drift explicitly out of scope.
Milestone close + mandatory audit run after Boss-verification.
bugfix-swarm-rc-alloc-undercount, GREEN stage. RED is the separate
audit-trail commit 483117d.
Root cause of the m5.2-resume-attempt residual: ail-embed/build.rs
declared cargo:rerun-if-changed only for the kernel .ail + build.rs
itself — NOT the runtime/ C sources ail build --emit=staticlib
compiles into libailang_rt.a. Cargo never re-ran build.rs after the
atomic-counter fix 7bfa11e, so ail-embed kept linking a stale
pre-7bfa11e non-atomic libailang_rt.a; the swarm's concurrent
TLS-NULL host allocs raced the stale non-atomic g_rc_alloc_count++
(frees-stable/allocs-jitter). NOT a second runtime bug and NOT
test-methodology unsoundness — runtime/rc.c at HEAD is correct; the
isolated crates/ail RED was green only because it rebuilds the
staticlib fresh per run.
Fix (ail-embed/build.rs, +7 lines): directory-level
cargo:rerun-if-changed=<repo>/runtime (Cargo recurses mtimes under
the dir). Preferred over a per-file list — the implementer inspected
crates/ail build_staticlib (libailang_rt.a = ar(rc.o, str.o), both
under runtime/) and chose the directory mechanism so any future
runtime source cannot silently re-introduce the same staleness.
Existing kernel/build.rs/AIL_BIN rerun-if-changed lines kept.
Cohesive consequence (in-scope): symbol_fan_swarm_leak_free
un-#[ignore]d — the test BODY is byte-for-byte unchanged (it was
quarantined, never weakened); the module-doc + fn-doc breadcrumbs
rewritten to the resolved build-dep-staleness rationale. It is now
the integration-level acceptance of both 7bfa11e and this fix. Boss
also removed a pre-existing unused `DataFormat` import in swarm.rs
inline (trivial; the file is committed this iter regardless;
warning-clean after).
Boss-verified independently: RED rt_archive_freshness -> GREEN;
swarm determinism 5/5 (symbol_fan_swarm_leak_free +
symbol_fan_swarm_bit_exact GREEN every run, 0 ignored, jitter gone,
Sallocs==Sfrees==12000003); isolated embed_rc_global_stats_race
still GREEN (untouched, runtime untouched); Invariant 1 data-server
count 0; scope = ail-embed/build.rs + ail-embed/tests/swarm.rs +
journal + stats only.
The M5 swarm leak-proof bounce-back is fully resolved end-to-end.
M5 stays open [~]; m5.3 (time-shard + friction-harvest + close-out)
remains. Includes the per-iter journal, stats, and INDEX.md line.
bugfix-swarm-rc-alloc-undercount, RED stage (audit trail; GREEN
follows separately via implement mini-mode).
Root cause of the m5.2-resume-attempt residual (the M5 swarm
leak-proof still non-deterministic after the atomic-counter fix
7bfa11e): ail-embed/build.rs declares cargo:rerun-if-changed only
for the kernel .ail + build.rs itself — NOT runtime/rc.c. So Cargo
never re-ran build.rs after 7bfa11e; the swarm keeps linking a
stale pre-7bfa11e libailang_rt.a whose g_rc_*count++ is the old
non-atomic racing increment. 7bfa11e was correct and is unchanged;
it just never reached the ail-embed binary. This explains why the
isolated crates/ail RED (rebuilds the staticlib fresh per run) is
green while the ail-embed swarm is not, and the
frees-stable/allocs-jitter asymmetry (few TLS-NULL frees = no
contention; millions of TLS-NULL host allocs = race on the stale
non-atomic counter).
Framing: a genuine defect (build-dependency completeness gap), NOT
test-methodology unsoundness — with a fresh archive the swarm
Σ-over-stat-lines balances deterministically (Σallocs==Σfrees==
12000003, 5/5 per the debugger). runtime/rc.c at HEAD is correct;
the GREEN fix is in the build script, not the runtime.
RED pins the CAUSE deterministically (the symptom is
non-deterministic by construction): objdump the libailang_rt.a that
build.rs emitted for this crate; assert >=2 lock-prefixed
instructions (the atomic lock incq for g_rc_alloc_count +
g_rc_free_count from current rc.c) and that it matches a
freshly-built archive. Boss-verified RED: linked archive has 0 lock
insns ("STALE pre-7bfa11e archive linked").
Does NOT touch the committed #[ignore] of symbol_fan_swarm_leak_free
nor the isolated embed_rc_global_stats_race RED (both correct,
unchanged).
Attempted to un-#[ignore] the M5 swarm leak-proof as the
integration-level acceptance of bugfix-rc-global-stats-race
(7bfa11e). Boss verification falsified the sufficiency premise:
across 4-6 swarm runs the leak Σ is STILL non-deterministic —
Σfrees stable-exact (12000003) every run, Σallocs short by a
jittering ~1600-2260 in ~1/3 of runs (e.g. left:11998383
right:12000003, 4 stat lines).
The atomic-global runtime fix (7bfa11e) is real, committed, and
NECESSARY — the isolated 8x2M pure-global RED
embed_rc_global_stats_race is deterministically green. It is NOT
sufficient for the swarm: a second, distinct alloc-side undercount
remains that the isolated RED did not model (Σfrees never loses =>
not a per-ctx race; not pure global contention => RED green).
The un-#[ignore] was reverted (never committed). The ONLY shipped
diff is the three now-stale #[ignore]-rationale breadcrumbs in
ail-embed/tests/swarm.rs corrected from "un-ignore when the runtime
makes the counters atomic" to the accurate necessary-but-insufficient
state — a future agent seeing 7bfa11e must NOT un-ignore on that
basis (doc-honesty rule). The test body and the #[ignore] attribute
itself are unchanged: still quarantined, NOT weakened.
symbol_fan_swarm_bit_exact stays live and GREEN — the swarm IS
actually correct and leak-free; only the accounting is wrong.
Residual handed to a fresh debug RED-first cycle. Two framings
recorded for it (genuine residual runtime defect vs Σ-over-
heterogeneous-lines test-methodology unsoundness; the latter, if so,
is a post-root-cause design decision). main stays green; M5 open
[~]; m5.3 remains downstream of a sound leak-proof.
Includes the resume-attempt journal + INDEX line.
bugfix-rc-global-stats-race, GREEN stage. RED is the separate
audit-trail commit 427b687.
g_rc_alloc_count / g_rc_free_count (runtime/rc.c:90-91) were plain
static uint64_t; the __ail_tls_ctx == NULL fallback at rc.c:161
(alloc) and rc.c:212 (dec-to-zero) did a non-atomic ++. A
multi-threaded host using the global fallback path (no
ailang_ctx_new) raced the read-modify-write and lost increments, so
the AILANG_RC_STATS atexit Σ under-counted non-deterministically.
Not a memory bug — Ctx:!Send keeps every box on one thread; only
the global statistics Σ was wrong (the M5 iter-2 symbol-fan
leak-proof finding, b724cd1; user-approved Option A bounce-back
resolution).
Fix (runtime/rc.c only, 37+/14-): the two globals are now
_Atomic uint64_t; atomic_fetch_add_explicit(.., memory_order_relaxed)
at the two null-ctx fallback ++ sites; atomic_load_explicit(..,
relaxed) snapshot in ailang_rc_stats_atexit (its sole reader).
Relaxed is correct and sufficient — pure statistics, no
happens-before obligation, atexit reader runs after all worker
threads joined. Per-ctx counters and the per-object refcount header
stay non-atomic BY DESIGN (single-thread-per-ctx; boxes never cross
threads) — out of scope, untouched. Frozen value layout / ABI
offsets / host-free rule untouched. The genuinely-still-
single-threaded drop-worklist doc note was correctly left unchanged
(false correction refused); the two stale atomicity doc blocks
corrected for honesty.
Boss-verified independently: RED -> GREEN deterministically 3/3
(jitter gone, allocs==frees==16_000_000); full cargo test -p ail
green (every binary 0 failed — per-ctx tsan harnesses + embed
e2e unaffected; rc.c links into every ail binary, strong regression
gate); scope = runtime/rc.c + journal + stats only; RED files
unchanged. Unblocks resuming the M5 leak-proof.
Includes the per-iter journal, stats, and the INDEX.md line.
bugfix-rc-global-stats-race, RED stage (audit trail; GREEN follows
separately via implement mini-mode).
`g_rc_alloc_count`/`g_rc_free_count` (runtime/rc.c:90-91) are plain
`static uint64_t`; the `__ail_tls_ctx == NULL` fallback at rc.c:161
and rc.c:212 does a non-atomic `++`. Concurrent host-side
ailang_rc_alloc/ailang_rc_dec outside a bound ctx race the
read-modify-write and silently drop updates. New C host (8 threads
x 2_000_000 alloc-then-dec, no ailang_ctx_new so the global path is
taken, no box crosses a thread) + integration test asserting the
atexit Σ is exact. Boss-verified RED: allocs=2141382 expected
16000000, live=-131242 (deterministic-fail under this contention,
not flaky).
NOT a memory bug — Ctx:!Send keeps every box on one thread, the
per-object refcount header op is correct, programs are bit-exact;
the only defect is the under-counted statistics Σ (the M5 iter 2
symbol-fan leak-proof finding, b724cd1). Existing green per-ctx
harnesses (embed_swarm_tsan.rs, embed_rc_accounting_tsan.rs) always
bind __ail_tls_ctx, exercising the zero-contention per-thread
counters and never this global fallback — which is why the suite is
green while the bug ships.
M5 iteration 2 (spec ae905de, plan 9cc9d9c). Tasks 1+2 clean and
committable; Task 3 BLOCKED on a real finding → M5 bounce-back.
Shipped:
- data-server promoted [dev-dependencies] -> [dependencies]
(Invariant-1 sanctioned: ail-embed is the sole meeting point; the
AILang workspace graph still has data-server count 0; zero
compiler-surface diff — not even root Cargo.toml this iter).
- additive `adapter` module (ail-embed/src/adapter.rs): tick_to_px,
MidPriceStream (lazy Iterator<f64> over SymbolChunkIter),
fold_symbol; RED-first (E0432 -> GREEN). m5.1 core untouched
except `pub mod adapter;`.
- swarm_runner [[bin]]: one thread per symbol, each owning its
Kernel. The clean compile IS the compile-time per-thread-ctx
proof (Ctx: !Send => a shared-ctx swarm is E0277).
- tests/swarm.rs: real-data symbol-fan E2E. `symbol_fan_swarm_bit_exact`
is GREEN and live — per-symbol kernel (acc,n) bit-exact vs an
independent same-order host reference (EURUSD/GER40/XAUUSD, ~4s).
The Task 3 finding (Boss independently confirmed by reading
runtime/rc.c): the global Σallocs==Σfrees leak measurement is an
INSTRUMENTATION race, not a memory bug. No box crosses a thread
(Ctx: !Send), the real refcount/free is correct (bit-exact GREEN
every run); only the global g_rc_* stat counters (rc.c:90-91,
161,212) are non-atomic BY rc.c's own documented single-threaded
design, losing ++s when 3 worker threads hit the null-__ail_tls_ctx
host-side path. M5 is AILang's first concurrent consumer; rc.c's
header (rc.c:44-49) explicitly defers exactly this atomic-vs-non-
atomic decision to "when it acquires concurrency primitives".
The leak assertion was Boss-split into `symbol_fan_swarm_leak_free`
(#[ignore], body preserved VERBATIM — quarantined not weakened;
un-ignore = the runtime fix's acceptance criterion) so main stays
green and the finding is pinned as a regression marker. The earlier
plan/journal claim that embed_swarm_tsan.rs covers this path was
wrong and is corrected on the record (that test uses the scalar
kernel — zero box allocs — so it never exercised the host-side
global-counter path).
Escalated as an M5 bounce-back: resolution touches M3-frozen
runtime/ and re-frames M5's "zero runtime change" commitment;
multiple substantive options, not unilaterally Boss's to pick in
frozen-runtime territory. Boss recommendation on the record =
Option A (make only the global-fallback g_rc_* counters atomic, as
a standalone RED-first runtime micro-iteration; M5 framing amended).
Time-shard + friction-harvest remain m5.3.
Includes the per-iter journal (with Boss disposition), stats, and
the INDEX.md line.
M5 iteration 2 (spec ae905de; builds on m5.1 204c171). 3 tasks:
(1) promote data-server dev-dep -> real dep + additive `adapter`
module (tick_to_px / MidPriceStream lazy Iterator / fold_symbol),
RED-first on tick_to_px; (2) `swarm_runner` [[bin]] — one thread per
symbol, each owning its Kernel (Ctx: !Send makes one-ctx-per-thread
a compile-time guarantee), prints RESULT <sym> <acc_bits> <n>;
(3) integration test spawning that bin as a subprocess under
AILANG_RC_STATS=1 (atexit stat line is unobservable in-process —
runtime/rc.c:132-138), asserting per-symbol bit-exact vs an
independent same-order host reference + Σallocs==Σfrees (ported from
embed_tick_e2e.rs:84-104), skip-if-absent mirroring data-server's
own precedent.
Boss decisions recorded in-plan for the audit: TSan needs NO new
nightly harness — per-thread-ctx race-freedom is compile-time
enforced by Ctx:!Send AND already green-tested by the C-host
embed_swarm_tsan.rs (same pattern); m5.2 adds no new AILang-side
concurrency primitive. data-server dev->real dep is Invariant-1
sanctioned (compiler crates untouched; gate shifts to "AILang
workspace graph data-server count == 0" + zero compiler-surface
diff). Time-shard + friction-harvest deferred to m5.3.
M5 iteration 1 (spec ae905de, plan 22f02aa). Stands up the
workspace-excluded `ail-embed` crate:
- zero-dependency embedding core (`ail-embed/src/lib.rs`): extern "C"
to the M3-frozen ABI + frozen-layout State/Tick box helpers + a
Kernel price fold; the Rust port of the audited
crates/ail/tests/embed/tick_roundtrip.c. Raw pointers never escape
the type.
- build.rs (no in-repo precedent): AIL_BIN env override else nested
`cargo build -p ail` against the parent workspace (separate target
dir → no cargo-lock deadlock), `ail build --emit=staticlib`, link
directives.
- hermetic data-server smoke (ail-embed/tests/smoke.rs): synthetic
Pepperstone-format ZIP fixture via data-server's own public
RawTickRecord type → real DataServer → Kernel, bit-exact vs a
same-order host reference fold; runs with no /mnt.
- `ail-embed` is its own cargo workspace root (empty [workspace]
table); data-server is a dev-dependency only. Root Cargo.toml gains
only a 4-line non-membership comment.
Invariant 1 Boss-verified independently: full+no-deps cargo metadata
on the AILang workspace shows data-server count 0; git status path
filter empty (zero diff to crates/ailang-*, crates/ail/, runtime/,
examples/*.ail); src/lib.rs zero code-level data_server; AILang
cargo build --workspace still clean. ail-embed suite 2/2 green
(kernel_run_sums_prices unit RED-first + hermetic_smoke integration),
verified by me, not just the agent report.
Two toolchain-forced corrections to the plan's verbatim
ail-embed/Cargo.toml (added empty [workspace] table; sibling dev-dep
path ../libs -> ../../libs, manifest-relative) — confined to the
plan-created manifest, no acceptance gate altered, 0 review re-loops.
Journal Concerns records the planner-recon implication for the next
workspace-excluded-nested-crate plan. Adapter API + thread-swarm
deferred to M5 iter 2+ per spec/plan.
Includes the per-iter journal, stats, and the INDEX.md line.
First iteration of M5 (spec ae905de). 3 tasks, RED-first on the
core: (1) scaffold the workspace-EXCLUDED ail-embed crate +
build.rs (de-risks the no-in-repo-precedent ail-binary resolution
in isolation with a stub lib), (2) the lean embedding core
(Ctx/Kernel/frozen-layout helpers, Rust port of the audited
tick_roundtrip.c:31-75, zero data_server use) with an in-source
data-server-free RED unit test, (3) hermetic data-server smoke
(synthetic Pepperstone-format ZIP fixture via the public
RawTickRecord type → real DataServer → Kernel, bit-exact vs host
reference).
Boss decomposition calls recorded in-plan: data-server is dev-only
(iter-1 library has ZERO deps — Invariant 1 maximally honest);
build.rs resolves ail via AIL_BIN-else-cargo-build-parent-workspace
(separate target dir, no nested-cargo lock); fixture writer mirrors
data-server/src/loader.rs:110-124. Invariant-1/no-language-change
mechanically gated (cargo metadata graph check + git status path
check). Adapter + swarm explicitly deferred to iter 2+.
User-approved 2026-05-19; grounding-check PASS (all load-bearing
assumptions are already-shipped M3 ABI behaviour, each pinned by a
green non-ignored integration test: embed_tick_e2e own+borrow,
embed/tick_roundtrip.c, embed_staticlib_cli, embed_swarm_tsan,
embed_staticlib_alloc_guard). Terminal embedding milestone, zero
language/compiler/runtime change — ail-embed is a lean reusable
embedding module (Rust port of the audited tick_roundtrip.c) + the
real E2E; in-repo, [workspace]-excluded so the compiler workspace
owes nothing to ../libs/data-server or /mnt (Invariant 1 in the
dependency graph, not just on paper). Real data-server + real
Pepperstone ticks; symbol-fan headline + time-shard
boundary-invisibility proof (per-shard bit-exact; whole-window only
within f64-reassociation tolerance — self-review fix).
roadmap: M5 [ ]->[~] open in-flight, spec pointer added, dangling
"todo above" dependency reconciled to shipped 170464f; M4-retired
and Tick-coverage struck/done entries pruned (their own text
scheduled deletion at next-milestone-start; permanent rationale
survives in 2026-05-18-brainstorm-embedding-abi-m4-retired.md, still
pointed to by M5's context line). Mirrors the M3 1fbb9c4 precedent
(spec + roadmap-flip in one commit).
Drives the RED test from a11cb7c to GREEN. Bug-fix iter
bugfix-over-strict-mode-ctor-rebuild-consume (debug -> implement
mini). Check-only lint-precision fix; zero codegen / runtime / ABI /
schema / DESIGN.md change.
The [over-strict-mode] lint's consume-detection
(any_sub_binder_consumed_for / pattern_has_consumed_heap_binder,
crates/ailang-check/src/linearity.rs) only recognised a consume of an
(own (con T)) param when a heap-typed pattern-binder was moved out of
`match p`. When p was destructured into purely primitive fields fed
into a Term::Ctor rebuilding p's own ctor, that genuine
dismantle+rebuild consume was invisible, so the lint spuriously
advised "(borrow ...) would suffice". An LLM author "fixing" that by
flipping an export own->borrow would silently invert the ABI
ownership contract — why a low-severity advisory FP got a real
RED-first fix.
Fix: a second recognition path in any_sub_binder_consumed_for — a
`match p` arm destructuring binders out of p's ctor that references
any of them (primitive or not) inside a Term::Ctor's args in the arm
body genuinely consumes p. Two pure helpers: ctor_uses_any_binder
(finds a fresh-allocation Term::Ctor reachable in the arm body) +
term_mentions_any_binder (deep free-var scan, so binder flow through
an intervening expr like (+ acc px) is recognised, not only a bare
Var arg). Conservative toward NOT suppressing: a ctor ignoring p's
payload still warns (negative-control verified). Over-strict-only —
by-name-shadowing imprecision is extra-silence, never under-strict
(Known debt in the journal + the fn doc). The two stale doc comments
that mis-attributed this FP to a nested `match` corrected for
doc-honesty (debugger concern #2 — same code region, in-scope).
Boss-verified independently: RED test now GREEN; full
cargo test -p ailang-check 108/0 lib + every binary 0-failed with NO
existing test modified; ail check examples/embed_backtest_step_tick.ail
no longer warns over-strict on st/tick (exit 0);
embed_backtest_step_tick_borrow.ail + M3 embed_backtest_step_record.ail
still clean; embed_tick_e2e + bench posture untouched.
+166/-20 in crates/ailang-check/src/linearity.rs only. Journal +
stats + INDEX line in this commit.
RED-first audit trail (debug skill; fix follows as a separate GREEN
commit). Test-only, +97 lines in the linearity in-source test module,
no production change.
Bug: the [over-strict-mode] lint emits a conservative false-positive.
The consume-detection (any_sub_binder_consumed_for /
pattern_has_consumed_heap_binder, crates/ailang-check/src/linearity.rs
:853,942) only recognises a consume of an (own (con T)) param when a
heap-typed pattern-binder is moved out of `match p`. When p is
destructured into purely primitive fields (e.g. Float/Int) that are
fed into a Term::Ctor rebuilding p's own ctor, that rebuild's
re-consumption of the dismantled allocation is invisible, so the lint
wrongly tells the author `(borrow ...)` would suffice. It is
over-strict, never under-strict (exit 0; codegen/ABI unaffected) — but
an LLM author "fixing" the spurious warning by flipping an export's
declared mode own->borrow would silently invert the ABI ownership
contract, which is why a low-severity advisory FP gets a real
RED-first fix.
The carrier's initial hypothesis (nested inner `match` is the
discriminator) was disproved during diagnosis: the M3
embed_backtest_step_record.ail is silent only because its
implicit-mode scalar Float param disables the lint via the activation
gate (linearity.rs:327), not because it handles the rebuild; the
defect reproduces with a single param and no nesting. The RED unit is
therefore the general synthetic shape — an (own (con T)) param
consumed by a term-ctor rebuild of its own ctor — not the embed
fixtures.
RED test (fails now, goes GREEN when consume-detection sees the
ctor-rebuild re-consumption):
crates/ailang-check/src/linearity.rs ::
linearity::tests::over_strict_mode_silent_when_ctor_rebuilt_from_primitive_fields
Coverage backfill for an already-shipped capability — NO language /
checker / codegen / schema / runtime change (4 new files, all under
examples/ and crates/ail/tests/).
Property protected: the M3 export gate is_c_abi_type runs as a
per-parameter loop, accepting a single-ctor all-scalar record
independently per param; the staticlib forwarder maps every
non-scalar Type::Con to a bare ptr (M3-frozen). So a kernel
(State, Tick) -> State with BOTH params single-ctor all-scalar
records is callable from C today — a record Tick pushed per call,
not just a scalar sample. Every shipped M3 fixture pushes a scalar
Float; none pinned the two-record-param shape (the actual minimal
data-server binding). This is the residual of the retired M4.
Added:
- examples/embed_backtest_step_tick.ail (own Tick)
- examples/embed_backtest_step_tick_borrow.ail (borrow Tick)
- crates/ail/tests/embed/tick_roundtrip.c (C host, -DBORROW)
- crates/ail/tests/embed_tick_e2e.rs (2 E2E tests)
Both E2E tests pass on HEAD (Boss-verified, not just agent-claimed):
own + borrow, globally leak-free (Sigma allocs == Sigma frees across
the ctx readback + g_rc atexit lines, the M3 dual-stat-line model),
acc/n value-correct, exit 0. Independently re-confirms the M4
retirement reasoning: the capability is genuinely present in M3.
Honest, deliberately NOT silenced: the new fixtures emit an
advisory [over-strict-mode] warning (ail check still exits 0,
non-gating) that the M3 scalar-sample original does not. Root cause
characterized: a conservative over-strict-mode false-positive whose
discriminator is the nested second (match tick ...) — structurally
identical `st` handling warns when its term-ctor rebuild is nested
inside an inner match. It is advisory only and does not affect
codegen, the ABI, or the proven property. The fixture is kept in
its LLM-natural form (dont-adapt-tests-to-bugs); the lint precision
wart is surfaced as a separate finding, not papered over.
M4 (sequence crossing via the List ADT) was struck during its own
/boss brainstorm, in the Step-2/3 Q&A, before any spec — the
"problem mis-framed -> don't ratify a known-unneeded shape into the
roadmap" brainstorm path. No spec, no grounding-check, no planner
handoff.
Why: held against DESIGN.md feature-acceptance criterion clause 2,
M4 removes no redundancy. The shipped M3 export gate is_c_abi_type
(crates/ailang-check/src/lib.rs:1934-1953) is a per-parameter loop
accepting a C scalar OR a single-ctor all-scalar record
independently per param; the forwarder's llvm_scalar maps every
non-scalar Type::Con to ptr (M3-frozen). So (State, Tick) -> State
with both as single-ctor all-scalar records is already
gate-accepted and forwarder-supported today. The minimal
data-server binding is M3 (shipped) + a host-side per-tick loop;
cons-list crossing would add a 2N+1-box-per-chunk host builder and
the deferred flat-array perf debt, with no named consumer (M5's
adapter unrolls each chunk host-side; whole-chunk in-kernel
visibility is semantically void — State threads across calls
regardless of chunk boundaries).
Outcome: M4 retired in docs/roadmap.md (struck entry kept one
cycle for context, never [x] — never shipped); M5 reconciled
(depends on M4 -> M3 + the Tick-coverage todo; adapter unrolls
host-side; friction feeds the host-per-tick-FFI vs. batch-crossing
P2 perf decision); residual recorded as a new [todo] Tick-coverage
on M3 — an E2E + fixture pinning the two-record-param per-tick
(State, Tick) -> State shape (test backfill, no brainstorm).
Rationale + the honest mid-Q&A correction (I had wrongly asserted
M5 needs M4) in
docs/journals/2026-05-18-brainstorm-embedding-abi-m4-retired.md;
INDEX.md mirrors it.
[medium] DESIGN.md §"Embedding ABI": surgically replaced ONLY the
contradicted M1-era parenthetical (asserted the export boundary is
scalar-only — false post-M3, a single-ctor record is an accepted
export type) with the present-tense truth pointing at the frozen-layout
SSOT + the own/borrow mode contract. The parenthetical shared physical
line :2300 with the docs_honesty_pin:135 pinned bare-scalar sentence;
the edit kept every pinned word (planner Step-5 item-6 pin-safety, the
M2.tidy precedent — docs_honesty_pin stayed 5/5 green, RED-restore path
never triggered). [low] codegen forwarder-body comment honesty fix
(comment-only; the IR byte-pins assert generated code not comments —
byte-identical before/after).
Boss-verified: both stale fragments grep-absent; 4 standing pins green
at the recon baseline; workspace 639/77 byte-unchanged; diff exactly
2 files. No behaviour change; the pins are the coverage (M2.tidy
precedent, no audit/fieldtest gate). bench already carry-on/NO-ratify
at the M3 audit (causally exonerated). One non-gating planner-quality
grep defect recorded in the journal Concerns (orchestrator handled it
correctly, no code bent).
Tasks 1-5 GREEN. T1 baseline pins (re-point annotation + @ailang_rc_alloc
heap-box byte-pin: size=8+n*8, tag@0, fields@8/16). T2 export gate widened
(is_c_scalar -> two-level is_c_abi_type: single-ctor all-Int/Float record;
multi-ctor/Str/List/nested still RED; gate suite 10/10; M1 adt-ret must-fail
re-pointed to multi-ctor+Str Reading). T3 codegen forwarder widened
(llvm_scalar record Type::Con -> ptr; M2 forwarder body byte-unchanged;
3/3 staticlib pins). T4/T5 E2E record round-trip own+borrow, global
leak-freedom.
Boss spec-consistency repair (M2.1-precedent class): orchestrator
correctly BLOCKED Task 5 on a genuine spec defect -- the single-ctx-readback
allocs==frees proof model is unsatisfiable for borrow (and only
coincidentally passes for own) because M2's TLS-ctx is bound only during
the synchronous forwarder call, so host-side decs land on g_rc_*, not ctx.
Boss-verified globally leak-free + value-correct both modes. Spec + plan +
harness amended to the stronger global model (sum all ailang_rc_stats:
lines; the M2-TLS cross-attribution documented as correct behaviour). No
fresh grounding-check (removes an over-strong measurement assumption).
Tasks 6 (DESIGN.md frozen-layout SSOT + lockstep pointers + freeze wording
+ enforceability demo) and 7 (workspace-green gate) re-dispatched on the
amended plan. Bench/architect milestone-close is audit-owned.
iter embedding-abi-m3.1 (PARTIAL); INDEX.md line deferred to the DONE commit
Resolves the M2 milestone-close audit's only actionable drift item
(architect [medium] DESIGN.md:2266 stale (M1) header vs its now-M2
present-tense body + coupled [low] :2354 xref). Current-state-mirror:
the section describes the current embedding ABI (M1+M2), so the
milestone tag is internal history, not current state.
5 live-coupling sites renamed in lockstep, 3 files:
- docs/DESIGN.md:2266 header '## Embedding ABI (M1)' -> '## Embedding ABI'
- docs/DESIGN.md:2354 schema-block xref 'see §"Embedding ABI"'
- docs_honesty_pin.rs:134 test comment + :136 assert-message
- crates/ailang-core/specs/form_a.md:89 live forward-xref (Boss-added
5th site: plan-recon-undercount countermeasure surfaced a live
dangling-xref the architect's 3-site scope missed; folded in on
the merits — a half-rename contradicts the tidy's own thesis)
docs_honesty_pin.rs:135 asserted substring (the rename-immune
DESIGN.md body sentence) byte-verbatim untouched. Committed
append-only history NOT falsified (Boss-verified diff = 3 content
files + journal + stats only). Pins green before AND after
(docs_honesty_pin 5/0, design_schema_drift 8/0). Zero
language/checker/codegen/schema change.
Architect drift_found: the only actionable item is the stale
'## Embedding ABI (M1)' DESIGN.md header (:2266) vs its now-M2 body
+ coupled :2354 xref — a current-state-mirror honesty desync the
iter journal flagged truthfully (Task-7-scope-deferred to preserve
the docs_honesty_pin:135 verbatim sentence). Everything substantive
holds (Invariant 1, internal-convention byte-invariance, Decision-10
note present-tense-correct, lockstep, no unrequested extras).
Bench check.py exit 1 (bump_s + 2x rc-latency max_us). M2 changed
the benched runtime path so byte-identical exoneration did not apply
-> ailang-bencher localised: H0 refuted, all 3 firings are the two
tracked roadmap-P2 noise families (bump_s causally impossible from
M2's diff; rc.c hunk branchless+free; max_us median/p99 flat +
non-reproducible). NO baseline ratified; not M2 artefacts.
Resolution: doc-honesty drift -> tidy iter embedding-abi-m2.tidy;
bench -> carry-on. Roadmap [~]->[x] after the tidy lands clean.
Completes Embedding ABI — M2.1 (Tasks 1-4 committed c9a84b3 as the
known-good subset; this commit lands Tasks 5-9 from the re-dispatch
against the Boss-corrected plan).
- T5: swarm.c per-thread-ctx scalar-swarm capability demo (no
SHARED_CTX) + rc_accounting_shared_ctx_is_flagged_by_tsan added to
the item-1 harness driver (de-globalisation negative-control teeth,
where a shared ctx genuinely races on ctx->alloc_count).
- T6: M1 host.c migrated to the ctx ABI; embed_e2e green (s==25
through the ctx-threaded forwarder).
- T7: DESIGN.md current-state update (provisional-until-M3 narrowed
to the value/record layout; Decision-10 per-thread-ctx note);
docs_honesty-pinned sentence + (con Int) snippet preserved verbatim.
- T8: regression gate — workspace green; null-ctx fallback
byte-behaviour preserved.
- T9: bench-trio — compile_check/cross_lang clean; check.py
tracked-P2 noise only, causally exonerated (zero codegen/runtime
diff vs HEAD), NO baseline ratified.
Boss-verified directly (not on report trust): docs_honesty_pin 5/0,
design_schema_drift 8/0, embed_export_hash_stable 1/0,
embed_export_gate 6/0, embed_e2e 1/0, embed_staticlib_alloc_guard
2/0, print_no_leak_pin 1/0 + e2e rc_stats 4/0; the tsan coherent-stop
proven by hand — per-ctx clean (exit 0, 0 warnings), shared-ctx a
real ThreadSanitizer data race at rc.c:157/:208 (exit 66).
Known debt (journal + roadmap): DESIGN.md '## Embedding ABI (M1)'
header still says (M1) while the body mirrors M2 — out of Task 7's
scope-limited regions by design; post-audit doc-honesty tidy.
Milestone-close audit next.
Tasks 1-4 land fully review-green and are the cohesive shippable
subset (the per-thread embedding ABI, fully wired + proven):
- T1: FIXED-FIRST alloc-guard baseline pin (RED captured -> GREEN at T4).
- T2: runtime/rc.c gains ailang_ctx_t {alloc_count,free_count} +
ailang_ctx_new/_free + __thread __ail_tls_ctx; the two increment
sites become 'if (_ctx) _ctx->... else g_rc_...'. g_rc_* statics +
ailang_rc_stats_atexit + the constructor RETAINED VERBATIM as the
null-ctx (single-threaded executable) fallback. rc_accounting_tsan.c
+ driver: per-ctx 8x200000 tsan-clean (de-globalisation positive proof).
- T3: codegen Target::StaticLib forwarder gains a leading 'ptr %ctx'
+ one '@__ail_tls_ctx = external thread_local global ptr' decl +
TLS save/store/restore around the BYTE-UNCHANGED internal call;
_adapter/_clos + internal arg vector untouched (M1 decision held);
doc-comment provisionality narrowed to the value/record layout.
- T4: build_staticlib rejects --alloc != rc (RC-only swarm artefact).
Boss adjudication of the orchestrator's Task-5 BLOCKED (spec-defect,
correctly surfaced not hacked): swarm.c's -DSHARED_CTX negative
control is structurally impossible — examples/embed_backtest_step.ail
is a non-allocating scalar kernel that never writes a ctx field, and
__ail_tls_ctx is __thread, so a shared-ctx scalar swarm has no
shared-memory write to race on (the spec's OWN item-1 honesty point).
The de-globalisation teeth belong to the item-1 rc_accounting harness
(shared ctx genuinely races on ctx->alloc_count; orchestrator
independently confirmed tsan exit 66). Spec amended in lockstep
(Goal coherent-stop, must-fail-axis item 2, Testing item 3,
Acceptance) so item 3's negative control is item-1's by the
de-globalisation-proof / capability-demo split; plan Task 5
restructured (swarm.c per-ctx capability demo only; negative-control
standing test relocated into the item-1 harness driver) + Task 3's
plan-transcription error (@ail_backtest_step ->
@ail_embed_backtest_step_step) corrected. This is a Boss
consistency-repair of an internally-contradictory clause whose intent
is already correctly realised in the same spec — not a redesign; no
brainstorm bounce. Task-5 working files discarded (corrected plan
reproduces them; a structurally-RED test must not enter main).
INDEX.md + final journal land at iter completion (re-dispatch [5,9]).
Restores the Decision-5 IR-readability affordance for main-free
kernels: ail emit-ir gains --emit=staticlib, symmetric with
ail build, reusing the M1-audited Target::StaticLib path via a new
one-line lower_workspace_staticlib convenience. Zero-export guard
byte-identical to build_staticlib's. DESIGN.md widened (not
narrowed) + a pre-existing M1 synopsis omission corrected. 3 new
E2E tests; no fixture minted, no doc pin (E2E is the coverage).
Plan 03493c9.
Resolves the M1-fieldtest [friction] + [spec_gap]#1 shared root:
form_a.md stated an unconditional 'every (fn ...) param needs an
(own/borrow) mode' rule in 4 places, contradicting shipped checker
behaviour (scalars take and require bare (con Int); a mode on a
scalar trips body-pointing use-after-consume/consume-while-borrowed).
All 4 sites rewritten symmetric to the existing return-type carve-out;
DESIGN.md gains the bare-scalar export-param rule + a corpus-grounded
Form-A snippet; RED-first anti-regrowth pin via FORM_A_SPEC + norm().
Zero language/checker/codegen change. Plan 4c266a6.
Surface-touch fieldtest of M1's (export "<sym>") + --emit=staticlib
+ scalar/effect-free gate. DESIGN.md + public examples only (no
compiler source). The M1 thesis is empirically substantiated:
- [working] Int EMA + Float leaky-integrator both first-try clean
end-to-end (author -> --emit=staticlib -> C link -> typed scalar
return; s==67 / s==1.875, exit 0). Clause-1 confirmed.
- [working] Both clause-3 rejections precise + self-correcting
(export-non-scalar-signature on IntList param;
export-has-effects on !IO). Clause-3 confirmed.
- [working] (export) orthogonality + zero-export staticlib guard
match spec.
- [friction] form_a.md 'wrap every param in own/borrow' misdirects
the headline scalar export -> body-pointing use-after-consume /
consume-while-borrowed; only bare (con Int) checks. Pre-existing
form_a.md defect M1 made acute (not introduced).
- [spec_gap] form_a.md item 1 unconditional; scalar params take no
mode (shares root with the friction).
- [spec_gap] no public emit-ir --emit=staticlib though Decision 5
makes kernel-IR readability load-bearing.
Boss verified independently via the public CLI (positive E2E builds
both archives; both rejections fire the documented codes). 0 bugs:
M1 capability + gate sound. Findings route to the roadmap as
follow-up (not M1-blocking, no debug).
Tasks 1-3 of docs/plans/embedding-abi-m1.1.md, a verified known-good
subset (cargo build --workspace exit 0; full workspace green; the
roundtrip_cli all-examples gate green):
- Task 1: CLI build-path MissingEntryMain baseline pin
(examples/embed_noentry_baseline.ail + embed_missing_main_baseline.rs;
triple-assertion, layered on the green codegen-unit pin lib.rs:3314).
- Task 2: additive FnDef.export: Option<String> (serde default +
skip_serializing_if, modelled on FnDef.doc); compile-driven thread
of export: None across ~85 FnDef/AstFnDef literals / 18 files incl.
all in-source #[cfg(test)] mod tests + drift/hash/spec-drift
literals + the codegen lib.rs:3320 in-source pin; hash-stability
golden pin; DESIGN.md fn-JSON "export" line. DONE_WITH_CONCERNS
(plan symbol content_hash_fn -> def_hash per hash_pin.rs, plan-
pseudo-vs-reality class, plan corrected).
- Task 3: Form-A (export "<sym>") modifier — parse_export +
parse_fn thread + write_fn_def emission + form_a.md grammar;
byte-identical round-trip.
Task 4 BLOCKED (Boss plan-defect: fixtures declared module != file
stem; AILang's loader hard-enforces module==stem at workspace.rs:438,
so ail check panicked on load before the gate). Boss resolution
(Option 2, overriding the orchestrator's Option 1, rationale in the
journal + plan Design-decision 6): keep descriptive embed_* filenames,
rename fixture MODULES to their stems — the C symbol backtest_step
stays via (export ...), demonstrating spec Decision 2's mangling-
decoupling rather than violating it; archive becomes
libembed_backtest_step.a, internal symbol @ail_embed_backtest_step_step,
host.c unchanged. Plan fixed (Design-decision 6 + Task-4 module==stem
+ Task-5/6/7 dependent strings + the no-*. Float-head + content_hash_fn
concerns folded). Headline fixture corrected; blocked-Task-4 WIP
discarded (recreated by the [4,7] re-dispatch from the fixed plan).
PARTIAL commit (not the iter's final commit): the implement Repair
path mandates committing the known-good subset so the [4,7]
re-dispatch starts from a clean tree that includes Tasks 1-3 (Tasks
4-7 structurally depend on the schema field + surface). INDEX line
added when the full iter lands post-re-dispatch.
User-authored proposal added during the embedding-abi-m1.1 implement
run (path-disjoint from the iter; Boss roadmap maintenance, not
bundled into the iter commit). Role-split of the 2905-line DESIGN.md
into a typed contract ledger + per-model whitepapers + decision-
records rehomed to docs/journals/, on the consumer/lifetime axis.
Brainstorm complete for Embedding ABI M1 (first of the M1–M5 arc).
User-approved 2026-05-18. Grounding-check PASS on 3rd dispatch
(dispatch 1 false-BLOCK on an in-source unit test it missed;
dispatch 2 PASS + located the pin; dispatch 3 PASS after honest
prose correction).
7 design decisions pinned, all derived from the M1–M5 arc except
the one genuine fork the user decided:
- additive FnDef.export: Option<String> (FnDef.doc precedent)
- (export "<sym>") author-chosen, mangling-decoupled C symbol
- lib<entry>.a (program only) + separate libailang_rt.a
- no runtime lifecycle API — documented no-call contract
- --emit=staticlib suppresses @main + MissingEntryMain; needs >=1 (export)
- export-signature gate: scalar-only + effect-free (clause-3 discriminator)
- Task 1 fixed: MissingEntryMain build-path baseline pin (RED-first)
Roadmap M1 flipped [ ] -> [~]. Next: planner in a fresh session
(user-chosen session shape).
The data-server-driven goal: a compiled AILang kernel callable
in-process from a concurrent Rust host so a backtest swarm runs
each backtest as an independent single-threaded computation.
Decomposed into five chronologically-bounded milestones (M1
linkable artifact+scalar entrypoint, M2 per-thread ctx+concurrency
safety, M3 frozen value layout+ADT crossing, M4 List-ADT sequence
crossing, M5 ail-embed adapter+swarm fieldtest), each its own full
brainstorm->plan->implement->audit cycle. Shared motivation + three
audited invariants (clean core / single-threaded per computation /
native-AOT-only) carried on the M1 entry. The flat array/slice
primitive is recorded P2 as a measured performance follow-up
gated on M5's fieldtest, not a capability gap — keeping the ABI
arc's scope clean. Decision made in the 2026-05-18 data-server
analysis chat; no spec yet (brainstorm produces M1's spec at start).
Architect clean (one [medium] advisory Sweep-5 self-match wart filed
as P3, not a fix iteration). Bench causally exonerated: HEAD vs
5bb7211/de66eb7 — all 5 firing bench binaries cmp-byte-identical, ail
sha256 identical at all 3 commits; the check.py firings are the two
tracked-P2 environmental classes (*.bump_s anchor staleness; *.max_us
-n5 tail-jitter, 4th on a zero-runtime-change milestone), NOT this
milestone — NO baseline ratify (Iron Law forbids ratifying noise).
compile_check 0/24, cross_lang 0/25 EXIT 0. No fieldtest (zero
authoring-surface change). docs-honesty-lint CLOSED: roadmap [~]->[x],
WhatsNew appended, INDEX line added.
Brummel