spec: embedding-abi-m5 — ail-embed adapter + data-server wiring + thread-swarm backtest

User-approved 2026-05-19; grounding-check PASS (all load-bearing
assumptions are already-shipped M3 ABI behaviour, each pinned by a
green non-ignored integration test: embed_tick_e2e own+borrow,
embed/tick_roundtrip.c, embed_staticlib_cli, embed_swarm_tsan,
embed_staticlib_alloc_guard). Terminal embedding milestone, zero
language/compiler/runtime change — ail-embed is a lean reusable
embedding module (Rust port of the audited tick_roundtrip.c) + the
real E2E; in-repo, [workspace]-excluded so the compiler workspace
owes nothing to ../libs/data-server or /mnt (Invariant 1 in the
dependency graph, not just on paper). Real data-server + real
Pepperstone ticks; symbol-fan headline + time-shard
boundary-invisibility proof (per-shard bit-exact; whole-window only
within f64-reassociation tolerance — self-review fix).

roadmap: M5 [ ]->[~] open in-flight, spec pointer added, dangling
"todo above" dependency reconciled to shipped 170464f; M4-retired
and Tick-coverage struck/done entries pruned (their own text
scheduled deletion at next-milestone-start; permanent rationale
survives in 2026-05-18-brainstorm-embedding-abi-m4-retired.md, still
pointed to by M5's context line). Mirrors the M3 1fbb9c4 precedent
(spec + roadmap-flip in one commit).
This commit is contained in:
2026-05-19 00:52:38 +02:00
parent 208d7095bc
commit ae905de00c
2 changed files with 386 additions and 44 deletions
+23 -44
View File
@@ -304,50 +304,29 @@ work progresses.
`docs/journals/2026-05-18-audit-embedding-abi-m3.md`;
`docs/journals/2026-05-18-iter-embedding-abi-m3.tidy.md`.
- **\[milestone — RETIRED 2026-05-18, never speced\]** Embedding ABI
— M4: sequence crossing via the `List` ADT. Struck during its own
brainstorm Q&A, before any spec: failed feature-acceptance
clause 2. The minimal data-server binding is M3 (shipped) + a
host-side per-tick loop over each chunk's records —
`(State, Tick) -> State` is *already* gate-accepted and
forwarder-supported under M3 (the gate loop accepts each record
param independently; the forwarder maps each to `ptr`). Cons-list
crossing would *add* a 2N+1-box-per-chunk host builder plus the
deferred flat-array perf debt rather than remove redundancy, and
builds something no named consumer needs (M5's adapter unrolls
the chunk host-side; whole-chunk in-kernel visibility buys nothing
`State` threads across calls regardless of chunk boundaries).
Removed (not `[x]` — never shipped); rationale in the journal.
Stays one cycle for context, then delete.
- context: `docs/journals/2026-05-18-brainstorm-embedding-abi-m4-retired.md`.
- [x] **\[todo\]** Tick-coverage on M3 — DONE 2026-05-18 (`170464f`).
E2E + fixtures pinning the two-record-param per-tick crossing
`(State, Tick) -> State` on the shipped M3 ABI (`own` + `borrow`
Tick, globally leak-free, Boss-verified both tests green on HEAD;
zero language change — independently re-confirms the M4-retirement
reasoning). Surfaced a conservative `[over-strict-mode]` advisory
false-positive (nested-`match` scrutinee; non-gating, exit 0) —
recorded as a separate finding, not silenced (fixtures kept
LLM-natural). Stays one cycle for context, then delete.
- context: `docs/journals/2026-05-18-brainstorm-embedding-abi-m4-retired.md`;
commit `170464f`.
- [ ] **\[milestone\]** Embedding ABI — M5: `ail-embed` adapter +
`data-server` wiring + thread-swarm backtest (fieldtest
target). The host-side adapter crate (sole meeting point of
`data-server` and AILang, per Invariant 1) wires
`data-server`'s `while let next_chunk` loop to the AILang
kernel via the M3 ABI by **unrolling each chunk host-side** into
per-tick `(State, Tick) -> State` calls (M4's cons-list crossing
was retired — see above; the adapter, not the kernel, owns chunk
iteration); an actual small backtest runs over a thread swarm.
Natural `fieldtest` milestone: existence proof of the whole goal
+ friction harvest feeding the host-per-tick-FFI vs. batch-
crossing perf decision (the P2 flat-array item). Coherent stop:
the goal itself.
- depends on: Embedding ABI — M3; Tick-coverage on M3 (todo above).
- context: 2026-05-18 chat; M4 retirement
- [~] **\[milestone\]** Embedding ABI — M5: `ail-embed` adapter +
`data-server` wiring + thread-swarm backtest. **In-flight**: spec
written + user-approved + grounding-check PASS 2026-05-19. The
host-side adapter — a lean reusable embedding module (`extern "C"`
to the M3-frozen ABI + frozen-layout box helpers, zero finance
knowledge) **plus** the real E2E on top; in-repo but
`[workspace]`-excluded; sole `data-server`↔AILang meeting point
(Invariant 1) — wires the **real external** `data-server`'s
`while let next_chunk` loop to the shipped M3 kernel by
**unrolling each chunk host-side** into per-tick
`(State, Tick) -> State` calls (the adapter, not the kernel, owns
chunk iteration; M4's cons-list crossing was retired). **Zero
language/compiler/runtime change.** A real backtest runs over real
Pepperstone tick data on a thread swarm: symbol-fan (headline) +
time-shard (first actual proof of the chunk-boundary invisibility
the M4 retirement rests on). Terminal embedding milestone:
existence proof of the whole goal + friction harvest feeding the
host-per-tick-FFI vs. batch-crossing perf decision (the P2
flat-array item). Coherent stop: the goal itself.
- depends on: Embedding ABI — M3 (shipped); Tick-coverage on M3
(shipped `170464f`).
- context: `docs/specs/2026-05-19-embedding-abi-m5.md`;
M4 retirement
`docs/journals/2026-05-18-brainstorm-embedding-abi-m4-retired.md`.
- [x] **\[milestone\]** Heap-`Str` ABI — runtime infrastructure for