From 750f97e4553f4b31d5f07d65467d4cce857c9be7 Mon Sep 17 00:00:00 2001 From: Brummel Date: Tue, 12 May 2026 16:24:12 +0200 Subject: [PATCH] =?UTF-8?q?spec:=20heap-str-abi=20=E2=80=94=20runtime-allo?= =?UTF-8?q?cated=20Str=20slab=20+=20sentinel-static=20ABI?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two builtins ship together: int_to_str (new) and float_to_str (today type-installed but codegen-deferred). Both produce malloc-backed, refcounted Str slabs via new ailang_int_to_str / ailang_float_to_str runtime exports. Slab layout is {rc_header, len, bytes, NUL}. Static-Str globals migrate to the same packed-struct shape with a UINT64_MAX sentinel rc_header; ailang_rc_inc / _dec short-circuit on the sentinel. From every caller's POV static and heap Str values are indistinguishable, so existing code paths (eq__Str, compare__Str, io/print_str, per-type drop walks) keep working with a single +8 GEP at the three C-API callsites and no change to drop dispatch. Out of scope: ++ concat operator, Show typeclass + print rewire, length-aware comparison for embedded-NUL Strs. --- docs/specs/2026-05-12-heap-str-abi.md | 509 ++++++++++++++++++++++++++ 1 file changed, 509 insertions(+) create mode 100644 docs/specs/2026-05-12-heap-str-abi.md diff --git a/docs/specs/2026-05-12-heap-str-abi.md b/docs/specs/2026-05-12-heap-str-abi.md new file mode 100644 index 0000000..2030c23 --- /dev/null +++ b/docs/specs/2026-05-12-heap-str-abi.md @@ -0,0 +1,509 @@ +# Heap-`Str` ABI — Design Spec + +**Date:** 2026-05-12 +**Status:** Draft — awaiting Step 7.5 grounding-check +**Authors:** Brummel (orchestrator) + Claude + +## Goal + +Add a second realisation of `Str` values — `malloc`-backed, +reference-counted heap slabs — alongside the existing static +`@.str_*` globals, *without* enlarging the language surface. +From the LLM-author's view, `Str` remains a single type; from +the IR caller's view, a single ABI handles both realisations. + +The milestone exists to unblock two builtins that are already +visible at the type level but currently broken at codegen: + +- `float_to_str : (Float) -> Str` is type-installed today but + raises `CodegenError::Internal` on call (`crates/ailang-codegen/src/lib.rs:1829`) + because there is no runtime-allocated `Str` path. +- `int_to_str : (Int) -> Str` is a natural counterpart that is + not even type-installed today; its absence forces the LLM-author + into awkward Bool-based formatting workarounds whenever an Int + needs to flow into print. + +Both ship together; both exercise the new heap-allocation path +identically (produce a fresh refcounted slab containing +runtime-formatted bytes). + +Out of scope (each its own follow-up milestone or todo): + +- `++` (Str concatenation operator). Would exercise the + *consume-and-produce* heap-Str path; the operator does not + exist in the language yet and needs its own feature-shape + decision (surface operator vs. builtin fn vs. typeclass + method). +- `Show` typeclass and `print`-rewire. Depends on this milestone + per the roadmap (P1 entry "Post-22 Prelude — Show + print + rewire"). +- Length-aware `==`/`<` semantics that handle embedded `\0` bytes + correctly. The status quo (`@strcmp`/`@ail_str_eq`/ + `@ail_str_compare`) carries forward unchanged. + +## Feature-acceptance check + +Per DESIGN.md §"Feature-acceptance criterion": + +1. **An LLM author naturally produces code that uses it.** Yes. + `int_to_str(42)` is the canonical way to convert an integer + into a printable representation; `float_to_str` is already + announced in DESIGN.md as a Float builtin. Both surface in + real LLM-authored programs whenever a numeric value needs to + reach `io/print_str`. + +2. **It measurably improves correctness or removes redundancy.** + Yes. Today `float_to_str` typechecks but fails codegen with a + structured internal error; this milestone makes it a working + path. `int_to_str` removes the redundant Bool-formatting + workaround needed today. + +Neither criterion rests on ergonomic appeal. The milestone ships +infrastructure that turns two announced-but-broken builtins into +working ones. + +## Architecture + +The milestone adds work in three coordinated layers without +enlarging the type-surface. + +### Runtime layer (`runtime/str.c`, `runtime/rc.c`) + +Two new extern functions in `runtime/str.c`: + +- `char *ailang_int_to_str(int64_t n)` +- `char *ailang_float_to_str(double x)` + +Both format their input into a stack buffer, allocate a heap +slab of matching size via a private `static char *str_alloc(uint64_t len)` +helper, copy the bytes (plus terminating `NUL`), and return the +payload pointer. + +`str_alloc` itself calls `ailang_rc_alloc(8 + len + 1)`, writes +the length into the first 8 bytes of the payload, and returns +the payload pointer. The caller fills bytes at offset 8 and the +`NUL` at offset `8 + len`. + +`runtime/rc.c` `ailang_rc_inc` and `ailang_rc_dec` gain a +**sentinel-header fast path**: when `*header_of(payload) == UINT64_MAX`, +both return immediately. This is the discriminator between +"heap-allocated, real refcount" and "static literal, treat as +immortal". + +### Codegen layer (`crates/ailang-codegen/src/lib.rs`) + +Four change-points. + +1. **Static-Str globals** (`intern_string`, ~`crates/ailang-codegen/src/lib.rs:2487`): + emitted not as `[N+1 x i8] c"…\00"` but as a packed struct + `<{i64, i64, [N+1 x i8]}>` with values `<{i64 -1, i64 N, c"…\00"}>`. + The callsite pointer is `getelementptr` to the second field + of this struct (= where the `len` slot starts), not to the + slab start. The pointer that flows through the IR as a `Str` + value points at the *payload* (= len-field), exactly mirroring + the heap-Str case. + +2. **`float_to_str` lowering** (~`crates/ailang-codegen/src/lib.rs:1829`): + the `CodegenError::Internal` fork is replaced by + `call ptr @ailang_float_to_str(double %a)`. + +3. **`int_to_str` lowering** (new): parallel to `float_to_str`, + emits `call ptr @ailang_int_to_str(i64 %a)`. + +4. **`drop__` walks** (per-type drop fns) — no code change. + `Str` fields are already included in the dec walk today: + `field_drop_call` at `crates/ailang-codegen/src/drop.rs:372` + maps `Type::Con { name: "Str", .. }` to bare `ailang_rc_dec`. + The comment at that site explicitly justifies the routing on + the assumption that *"Str payloads are NUL-terminated bytes in + static memory; nothing to recurse into"* — exactly the + assumption that breaks once heap-Str exists. With the current + static-only path, dropping an ADT field of type `Str` therefore + already calls `ailang_rc_dec(static_str_ptr)`, which reads + `header_of(payload) = payload - 8` — bytes that fall *outside* + the static `[N x i8]` global. This is latent undefined + behaviour today; no shipping test exercises an ADT-with-Str- + field under `--alloc=rc`, so it has not surfaced. The + sentinel-header check proposed in (item 1 above implicitly + + the `runtime/rc.c` change below) turns that latent UB into a + defined no-op: with the new packed-struct global, `payload - 8` + lands on a valid `i64` slot whose value is `UINT64_MAX`, and + `ailang_rc_dec` short-circuits. **The change here is in + correctness, not in code: no edit to `drop.rs` is required.** + +The existing extern declarations `@strcmp`, `@ail_str_eq`, +`@ail_str_compare`, `@puts`, `@printf` stay. They still expect a +`NUL`-terminated bytes pointer, which is *not* the same as the +payload pointer that flows through the IR (the payload pointer +points at the `len` field; bytes start 8 bytes later). The three +codegen sites that today call these C functions on a `Str` value +(`compare__Str`, `eq__Str`, `io/print_str` — at +`crates/ailang-codegen/src/lib.rs:2152`, `2262`, `2313`) emit a +`getelementptr i8, ptr %s, i64 8` immediately before the call to +land on the bytes pointer. + +### Checker layer (`crates/ailang-check/src/builtins.rs`) + +`int_to_str : (Int) -> Str` is added to the builtin signature +table next to the existing `float_to_str` entry +(`crates/ailang-check/src/builtins.rs:194`). The unit test +`install_int_to_str_signature` mirrors the existing +`install_float_to_str_signature` (line 445). + +No mode annotations, no effects — like `int_to_float`, this is a +pure conversion. + +### What this milestone does *not* change + +- The `Type::Str` variant, the `Literal::Str` JSON encoding, the + Form-A / Form-B mapping, the prose serialisation. +- The uniqueness inferencer. `Str` is immutable; no Own/Bor + modes apply. +- The `@strcmp`-based comparison semantics. Embedded `\0` bytes + in a `Str` value continue to cause comparison to stop early; + this is the status quo, and the milestone explicitly inherits + it. + +## Components + +File-by-file inventory of the change surface. + +### `runtime/str.c` + +Add: + +- `static char *str_alloc(uint64_t len)` — private helper. + Calls `ailang_rc_alloc(8 + len + 1)`, writes `len` at payload + offset 0, returns payload pointer. +- `char *ailang_int_to_str(int64_t n)` — extern. `snprintf` into + a 64-byte stack buffer with format `"%lld"`; defensively + asserts no truncation (the longest i64 is 20 chars, the buffer + is 64); calls `str_alloc` with the resulting length; memcpy's + bytes + writes the `NUL`; returns the payload pointer. +- `char *ailang_float_to_str(double x)` — extern. Same shape + with format `"%g"`. `%g` matches `io/print_float`'s existing + format for IEEE consistency; NaN/±Inf render via libc default. + +### `runtime/rc.c` + +Modify `ailang_rc_inc` and `ailang_rc_dec`: after the existing +`NULL` guard, add `if (*header_of(payload) == UINT64_MAX) return;` +to short-circuit on the sentinel. + +### `crates/ailang-codegen/src/lib.rs` + +Modify: + +- `intern_string` (~`:2487`) — switch the LLVM-IR emission of + static-Str globals from `[N+1 x i8] c"…\00"` to + `<{i64, i64, [N+1 x i8]}> <{i64 -1, i64 N, c"…\00"}>`. The + global name stays the same; the way callsites materialise a + pointer to it changes to a `getelementptr` landing on the + `len`-field. +- IR-header preamble (~`:455`–`:498`): add + `declare ptr @ailang_int_to_str(i64)` and + `declare ptr @ailang_float_to_str(double)`. +- `float_to_str` arm (~`:1829`): drop the `Err(CodegenError::Internal(…))`, + emit the call instead. +- New arm parallel to it for `int_to_str`. +- `compare__Str`, `eq__Str`, `io/print_str` codegen sites + (~`:2152`, `:2262`, `:2313`): emit a +8 `getelementptr` to + land on the bytes pointer before calling `@ail_str_compare`, + `@ail_str_eq`, `@puts` respectively. +- **Per-type drop walks are unchanged.** `Str` is already routed + to `ailang_rc_dec` by `field_drop_call` + (`crates/ailang-codegen/src/drop.rs:372`). The stale comment + at that site (*"NUL-terminated bytes in static memory; nothing + to recurse into"*) becomes incorrect once heap-Str ships and + must be updated to reflect the new dual-realisation reality + — but the dispatched symbol (`ailang_rc_dec`) is already + correct. + +### `crates/ailang-check/src/builtins.rs` + +Add `int_to_str` to the builtin signatures table (~`:194`, +mirroring `float_to_str`'s entry), and add the type-installation +unit test parallel to `install_float_to_str_signature`. + +### `crates/ailang-codegen/src/synth.rs` + +Add an `int_to_str` arm parallel to the `float_to_str` arm at +`:167` so the codegen's local type-replay knows the signature. + +### `docs/DESIGN.md` + +- §"Float semantics", last paragraph: remove the "codegen lowering + deferred" caveat on `float_to_str`. +- §"What is not (yet) supported" and the builtin inventory under + "What **is** supported": add `int_to_str : (Int) -> Str` next + to `float_to_str`. +- New top-level subsection "Str ABI" (or under an appropriate + Decision-10 / runtime-contracts anchor): document the unified + slab layout `{rc_header, len, bytes…, NUL}`, the sentinel- + header convention (`UINT64_MAX` = static / immortal), and the + inherited limitation that `==` / `<` use `strcmp` semantics + and do not support embedded `\0` bytes (this is a future- + milestone enhancement, not a regression). + +## Data flow + +Three representative IR cases pin the after-state. + +**Static string literal.** Today, `"hello"` is emitted as: + +```llvm +@.str_M_hello_0 = private unnamed_addr constant [6 x i8] c"hello\00" +… +%1 = call i32 @puts(ptr @.str_M_hello_0) +``` + +After the milestone: + +```llvm +@.str_M_hello_0 = private unnamed_addr constant <{i64, i64, [6 x i8]}> + <{ i64 -1, i64 5, [6 x i8] c"hello\00" }> +… +%payload = getelementptr inbounds <{i64, i64, [6 x i8]}>, + ptr @.str_M_hello_0, i32 0, i32 1 +%bytes = getelementptr inbounds i8, ptr %payload, i64 8 +%1 = call i32 @puts(ptr %bytes) +``` + +`%payload` is the value that flows through the IR as a `Str`. The +`%bytes` adjustment is local to the `@puts` callsite. + +**Heap-allocated string from `int_to_str(42)`.** + +```llvm +%s = call ptr @ailang_int_to_str(i64 42) +; …print: +%bytes = getelementptr inbounds i8, ptr %s, i64 8 +%1 = call i32 @puts(ptr %bytes) +; …at scope close: +call void @ailang_rc_dec(ptr %s) +``` + +`%s` is a fresh heap slab with `rc_header == 1`. `ailang_rc_dec` +accesses the header at `%s - 8`, decrements to zero, frees. + +**`eq Str` with mixed-origin operands.** + +```llvm +define i1 @eq__Str(ptr %a, ptr %b) { + %a_bytes = getelementptr inbounds i8, ptr %a, i64 8 + %b_bytes = getelementptr inbounds i8, ptr %b, i64 8 + %r = call zeroext i1 @ail_str_eq(ptr %a_bytes, ptr %b_bytes) + ret i1 %r +} +``` + +Either operand may be static (sentinel header) or heap (real +header); the body does not distinguish. `@ail_str_eq` keeps its +`strcmp`-based body unchanged. + +**ADT drop with a `Str` field.** For `type Boxed = | Box(Str)`, +the per-type `drop_M_Boxed(ptr %cell)` already emits a +`call void @ailang_rc_dec(ptr %str_field)` today — `Str` fields +are dispatched to `ailang_rc_dec` by `field_drop_call` regardless +of whether the value originated from a static or a heap source. +What changes is the *meaning* of that call on a static-Str +pointer: pre-milestone it reads bytes outside the `[N x i8]` +global (latent UB; no shipping test exercises the path); +post-milestone it reads the new global's leading `i64 -1` slot, +hits the sentinel short-circuit in `ailang_rc_dec`, and returns +without modifying anything. For a heap-Str field, the normal +refcount drop runs unchanged. The single behavioural shift — +from latent UB to defined no-op on static fields — is gated by +the new `str_field_in_adt_drops_static_str_noop` E2E test (see +§Testing strategy). + +## Error handling + +Four failure modes; most are inherited or structurally impossible. + +**Out-of-memory at heap-Str allocation.** `ailang_rc_alloc` +already `abort()`s on OOM, matching Boehm. `str_alloc`, +`ailang_int_to_str`, `ailang_float_to_str` propagate. No new +recovery path. + +**`snprintf` truncation in the runtime formatters.** The 64-byte +stack buffer is strictly larger than the worst-case width of +either `%lld` (longest i64 = 20 chars) or `%g` on a `double` +(~25 chars worst case). The runtime body asserts no truncation +defensively (the `snprintf` return tells us how many bytes +*would* have been written; an `abort()` fires if that exceeds +the buffer). The defensive check survives format-string changes +that might widen the output without anyone noticing the buffer +is too small. + +**Sentinel collision.** A real refcount can never reach +`UINT64_MAX` — `2^64` references would require more memory than +exists on any machine the program can run on. The convention +`rc_header == UINT64_MAX` ⇒ static is structurally safe; no +runtime check beyond the equality test in `inc`/`dec` is needed. +DESIGN.md §"Str ABI" records this as an invariant. + +**Embedded `\0` bytes in static-Str literals from JSON.** Status +quo: a literal containing `""` ends up with a `NUL` in the +middle of the slab; `strcmp`-based comparison treats it as the +end of string. This is the *current* behaviour of static-Str and +heap-Str inherits it unchanged. Heap-Str values produced by +`int_to_str` and `float_to_str` themselves cannot contain +embedded `\0` (neither `%lld` nor `%g` emits one), so the +milestone scope is not affected. DESIGN.md §"Str ABI" notes the +limitation explicitly with a forward pointer to a future +length-aware-compare milestone. + +`float_to_str`'s `CodegenError::Internal` path is *removed* +entirely after the milestone — neither as a caught nor as an +uncaught error. The symbol-installation in the builtin tier and +the codegen lowering are landed in the same iteration; there is +no intermediate state where one exists without the other. + +## Testing strategy + +Three layers. + +### IR-shape tests in the codegen crate + +Pin the structural guarantees of §"Data flow" with substring +matches against emitted IR: + +- `static_str_global_uses_packed_struct_with_sentinel_and_len` — + pins the new slab format. +- `static_str_callsite_pointer_is_payload_not_slab` — pins the + `getelementptr`-to-second-field convention. +- `eq_str_calls_ail_str_eq_with_bytes_pointer`, + `compare_str_calls_ail_str_compare_with_bytes_pointer`, + `print_str_calls_puts_with_bytes_pointer` — pin the +8 GEP at + each existing C-API callsite. +- `int_to_str_lowers_to_ailang_int_to_str_call` — pins the new + builtin's lowering. +- `float_to_str_no_longer_errors_internal` — converts the + existing "lowering deferred"-pinning test into a positive + green test. + +These tests are fast but sensitive to SSA-naming changes. The +existing test style in `crates/ailang-codegen/src/lib.rs` already +accepts this risk for analogous pins (e.g. +`eq_str_mono_symbol_emits_ail_str_eq_call`); the new tests match. + +### Stdout E2E tests in `crates/ail/tests/e2e.rs` + +Observable program behaviour: + +- `int_to_str_smoke` — `int_to_str(42)` prints `"42\n"`; plus + edge cases `0`, `-1`, `i64::MAX`, `i64::MIN`. +- `float_to_str_smoke` — `float_to_str(3.5)` prints something + starting with `3.5`; plus `0.0`, `NaN`, `+Inf`, `-Inf` (pinned + against libc `%g` rendering — implementations are permitted to + print NaN as `nan`, `-nan`, or other casing; the test pins the + exact bytes observed on the dev target with a comment + acknowledging the libc-defined freedom). +- `static_str_print_unchanged` — an existing example + (`examples/hello.ail` or similar) produces byte-identical + stdout before and after the milestone, sourced from + pre-milestone `expected/`. + +### RC-discipline E2E tests via `AILANG_RC_STATS` + +Catch leaks and double-frees: + +- `int_to_str_drop_balances_rc_stats` — `int_to_str` is called + N times in a loop, each result dropped without aliasing; + `AILANG_RC_STATS=1` reports `allocs=K frees=K live=0` for some + K matched to the loop bound + setup. Pins that heap-Str + participates in RC normally. +- `str_field_in_adt_drops_heap_str_correctly` — a value of + `type Boxed = | Box(Str)` is instantiated with a heap-Str + (from `int_to_str`) and dropped; `live=0`. +- `str_field_in_adt_drops_static_str_noop` — same ADT + instantiated with a static literal under `--alloc=rc`; `live=0`, + no crash. **Load-bearing:** no precursor test today exercises + this path (the existing `alloc_rc_*` tests do not instantiate + an ADT with a `Str` field), so today's behaviour on this path + is undefined. The test gates the milestone-level safety + property that static-Str pointers in ADT fields drop safely + via the sentinel short-circuit. + +### Bench gates + +At audit close (per `skills/audit/SKILL.md`): + +- `bench/check.py` — existing latency baselines green, with the + known `latency.explicit_at_rc.*` nondeterminism tolerance + documented in recent audits. +- `bench/compile_check.py` — entire `examples/` corpus compiles. +- `bench/cross_lang.py` — byte-identical outputs vs. the C + reference corpus in `bench/reference/`. This is the strongest + "we did not break anything" gate because it compares output + against a second-language implementation, not against AILang + itself. + +### Out of testing scope + +Embedded-`\0` behaviour. Concurrent RC (AILang is single- +threaded). Allocator stress beyond representative program sizes. + +## Acceptance criteria + +The milestone closes when all the following are observable. + +**Language surface.** + +- `int_to_str : (Int) -> Str` is installed in + `crates/ailang-check/src/builtins.rs` and resolved by the + codegen builtin lookup. +- `float_to_str` no longer lowers to `CodegenError::Internal`; + calling it produces a working binary. +- DESIGN.md §"Float semantics" carries no "codegen lowering + deferred" caveat on `float_to_str`; the builtin inventory + under "What **is** supported" lists `int_to_str` alongside + `float_to_str`. + +**Runtime / ABI.** + +- `runtime/str.c` exports `ailang_int_to_str(int64_t) -> char*` + and `ailang_float_to_str(double) -> char*`. The private + `str_alloc(uint64_t)` is *not* in the export symbol set. +- `runtime/rc.c` `ailang_rc_inc`/`ailang_rc_dec` short-circuit + on `*header == UINT64_MAX`. +- Static-Str globals are emitted as + `<{i64, i64, [N+1 x i8]}>` with values `{i64 -1, i64 N, c"…\00"}`. +- DESIGN.md has a "Str ABI" anchor (or equivalent) documenting + the unified layout, sentinel convention, and inherited + `strcmp`-based comparison semantics. + +**Observable program behaviour.** + +- A trivial `.ail` program `do io/print_str(int_to_str(42))` + builds, runs, and prints `42`. +- The analogous `float_to_str(3.5)` program builds, runs, and + prints `%g`-conformant output (exact bytes pinned by the E2E + test). +- All existing `examples/` programs produce byte-identical + stdout vs. pre-milestone (no regressions). + +**RC discipline.** + +- The three new `AILANG_RC_STATS`-based E2E tests pass. +- No existing drop-relevant test breaks. + +**Bench gates.** + +- `bench/check.py`, `bench/compile_check.py`, `bench/cross_lang.py` + all green at audit close. + +**Documentation / journal.** + +- One journal entry per iteration with per-task notes. +- `WhatsNew.md` gets a milestone-close entry (verbatim with the + done-state Notify), written for the user-as-reader without + iteration codes or crate names. + +**Explicitly *not* part of acceptance.** + +- `++` / Str concatenation operator (own milestone). +- `Show` typeclass and `print` rewire (own milestone, depends-on + this milestone). +- Length-aware comparison semantics / embedded-`\0` support.