spec: heap-str-abi — runtime-allocated Str slab + sentinel-static ABI

Two builtins ship together: int_to_str (new) and float_to_str (today type-installed but codegen-deferred). Both produce malloc-backed, refcounted Str slabs via new ailang_int_to_str / ailang_float_to_str runtime exports. Slab layout is {rc_header, len, bytes, NUL}. Static-Str globals migrate to the same packed-struct shape with a UINT64_MAX sentinel rc_header; ailang_rc_inc / _dec short-circuit on the sentinel. From every caller's POV static and heap Str values are indistinguishable, so existing code paths (eq__Str, compare__Str, io/print_str, per-type drop walks) keep working with a single +8 GEP at the three C-API callsites and no change to drop dispatch. Out of scope: ++ concat operator, Show typeclass + print rewire, length-aware comparison for embedded-NUL Strs.
2026-05-12 16:24:12 +02:00
parent 78c26b3596
commit 750f97e455
1 changed files with 509 additions and 0 deletions
@@ -0,0 +1,509 @@
+# Heap-`Str` ABI — Design Spec
+
+**Date:** 2026-05-12
+**Status:** Draft — awaiting Step 7.5 grounding-check
+**Authors:** Brummel (orchestrator) + Claude
+
+## Goal
+
+Add a second realisation of `Str` values — `malloc`-backed,
+reference-counted heap slabs — alongside the existing static
+`@.str_*` globals, *without* enlarging the language surface.
+From the LLM-author's view, `Str` remains a single type; from
+the IR caller's view, a single ABI handles both realisations.
+
+The milestone exists to unblock two builtins that are already
+visible at the type level but currently broken at codegen:
+
+- `float_to_str : (Float) -> Str` is type-installed today but
+  raises `CodegenError::Internal` on call (`crates/ailang-codegen/src/lib.rs:1829`)
+  because there is no runtime-allocated `Str` path.
+- `int_to_str : (Int) -> Str` is a natural counterpart that is
+  not even type-installed today; its absence forces the LLM-author
+  into awkward Bool-based formatting workarounds whenever an Int
+  needs to flow into print.
+
+Both ship together; both exercise the new heap-allocation path
+identically (produce a fresh refcounted slab containing
+runtime-formatted bytes).
+
+Out of scope (each its own follow-up milestone or todo):
+
+- `++` (Str concatenation operator). Would exercise the
+  *consume-and-produce* heap-Str path; the operator does not
+  exist in the language yet and needs its own feature-shape
+  decision (surface operator vs. builtin fn vs. typeclass
+  method).
+- `Show` typeclass and `print`-rewire. Depends on this milestone
+  per the roadmap (P1 entry "Post-22 Prelude — Show + print
+  rewire").
+- Length-aware `==`/`<` semantics that handle embedded `\0` bytes
+  correctly. The status quo (`@strcmp`/`@ail_str_eq`/
+  `@ail_str_compare`) carries forward unchanged.
+
+## Feature-acceptance check
+
+Per DESIGN.md §"Feature-acceptance criterion":
+
+1. **An LLM author naturally produces code that uses it.** Yes.
+   `int_to_str(42)` is the canonical way to convert an integer
+   into a printable representation; `float_to_str` is already
+   announced in DESIGN.md as a Float builtin. Both surface in
+   real LLM-authored programs whenever a numeric value needs to
+   reach `io/print_str`.
+
+2. **It measurably improves correctness or removes redundancy.**
+   Yes. Today `float_to_str` typechecks but fails codegen with a
+   structured internal error; this milestone makes it a working
+   path. `int_to_str` removes the redundant Bool-formatting
+   workaround needed today.
+
+Neither criterion rests on ergonomic appeal. The milestone ships
+infrastructure that turns two announced-but-broken builtins into
+working ones.
+
+## Architecture
+
+The milestone adds work in three coordinated layers without
+enlarging the type-surface.
+
+### Runtime layer (`runtime/str.c`, `runtime/rc.c`)
+
+Two new extern functions in `runtime/str.c`:
+
+- `char *ailang_int_to_str(int64_t n)`
+- `char *ailang_float_to_str(double x)`
+
+Both format their input into a stack buffer, allocate a heap
+slab of matching size via a private `static char *str_alloc(uint64_t len)`
+helper, copy the bytes (plus terminating `NUL`), and return the
+payload pointer.
+
+`str_alloc` itself calls `ailang_rc_alloc(8 + len + 1)`, writes
+the length into the first 8 bytes of the payload, and returns
+the payload pointer. The caller fills bytes at offset 8 and the
+`NUL` at offset `8 + len`.
+
+`runtime/rc.c` `ailang_rc_inc` and `ailang_rc_dec` gain a
+**sentinel-header fast path**: when `*header_of(payload) == UINT64_MAX`,
+both return immediately. This is the discriminator between
+"heap-allocated, real refcount" and "static literal, treat as
+immortal".
+
+### Codegen layer (`crates/ailang-codegen/src/lib.rs`)
+
+Four change-points.
+
+1. **Static-Str globals** (`intern_string`, ~`crates/ailang-codegen/src/lib.rs:2487`):
+   emitted not as `[N+1 x i8] c"…\00"` but as a packed struct
+   `<{i64, i64, [N+1 x i8]}>` with values `<{i64 -1, i64 N, c"…\00"}>`.
+   The callsite pointer is `getelementptr` to the second field
+   of this struct (= where the `len` slot starts), not to the
+   slab start. The pointer that flows through the IR as a `Str`
+   value points at the *payload* (= len-field), exactly mirroring
+   the heap-Str case.
+
+2. **`float_to_str` lowering** (~`crates/ailang-codegen/src/lib.rs:1829`):
+   the `CodegenError::Internal` fork is replaced by
+   `call ptr @ailang_float_to_str(double %a)`.
+
+3. **`int_to_str` lowering** (new): parallel to `float_to_str`,
+   emits `call ptr @ailang_int_to_str(i64 %a)`.
+
+4. **`drop_<m>_<T>` walks** (per-type drop fns) — no code change.
+   `Str` fields are already included in the dec walk today:
+   `field_drop_call` at `crates/ailang-codegen/src/drop.rs:372`
+   maps `Type::Con { name: "Str", .. }` to bare `ailang_rc_dec`.
+   The comment at that site explicitly justifies the routing on
+   the assumption that *"Str payloads are NUL-terminated bytes in
+   static memory; nothing to recurse into"* — exactly the
+   assumption that breaks once heap-Str exists. With the current
+   static-only path, dropping an ADT field of type `Str` therefore
+   already calls `ailang_rc_dec(static_str_ptr)`, which reads
+   `header_of(payload) = payload - 8` — bytes that fall *outside*
+   the static `[N x i8]` global. This is latent undefined
+   behaviour today; no shipping test exercises an ADT-with-Str-
+   field under `--alloc=rc`, so it has not surfaced. The
+   sentinel-header check proposed in (item 1 above implicitly +
+   the `runtime/rc.c` change below) turns that latent UB into a
+   defined no-op: with the new packed-struct global, `payload - 8`
+   lands on a valid `i64` slot whose value is `UINT64_MAX`, and
+   `ailang_rc_dec` short-circuits. **The change here is in
+   correctness, not in code: no edit to `drop.rs` is required.**
+
+The existing extern declarations `@strcmp`, `@ail_str_eq`,
+`@ail_str_compare`, `@puts`, `@printf` stay. They still expect a
+`NUL`-terminated bytes pointer, which is *not* the same as the
+payload pointer that flows through the IR (the payload pointer
+points at the `len` field; bytes start 8 bytes later). The three
+codegen sites that today call these C functions on a `Str` value
+(`compare__Str`, `eq__Str`, `io/print_str` — at
+`crates/ailang-codegen/src/lib.rs:2152`, `2262`, `2313`) emit a
+`getelementptr i8, ptr %s, i64 8` immediately before the call to
+land on the bytes pointer.
+
+### Checker layer (`crates/ailang-check/src/builtins.rs`)
+
+`int_to_str : (Int) -> Str` is added to the builtin signature
+table next to the existing `float_to_str` entry
+(`crates/ailang-check/src/builtins.rs:194`). The unit test
+`install_int_to_str_signature` mirrors the existing
+`install_float_to_str_signature` (line 445).
+
+No mode annotations, no effects — like `int_to_float`, this is a
+pure conversion.
+
+### What this milestone does *not* change
+
+- The `Type::Str` variant, the `Literal::Str` JSON encoding, the
+  Form-A / Form-B mapping, the prose serialisation.
+- The uniqueness inferencer. `Str` is immutable; no Own/Bor
+  modes apply.
+- The `@strcmp`-based comparison semantics. Embedded `\0` bytes
+  in a `Str` value continue to cause comparison to stop early;
+  this is the status quo, and the milestone explicitly inherits
+  it.
+
+## Components
+
+File-by-file inventory of the change surface.
+
+### `runtime/str.c`
+
+Add:
+
+- `static char *str_alloc(uint64_t len)` — private helper.
+  Calls `ailang_rc_alloc(8 + len + 1)`, writes `len` at payload
+  offset 0, returns payload pointer.
+- `char *ailang_int_to_str(int64_t n)` — extern. `snprintf` into
+  a 64-byte stack buffer with format `"%lld"`; defensively
+  asserts no truncation (the longest i64 is 20 chars, the buffer
+  is 64); calls `str_alloc` with the resulting length; memcpy's
+  bytes + writes the `NUL`; returns the payload pointer.
+- `char *ailang_float_to_str(double x)` — extern. Same shape
+  with format `"%g"`. `%g` matches `io/print_float`'s existing
+  format for IEEE consistency; NaN/±Inf render via libc default.
+
+### `runtime/rc.c`
+
+Modify `ailang_rc_inc` and `ailang_rc_dec`: after the existing
+`NULL` guard, add `if (*header_of(payload) == UINT64_MAX) return;`
+to short-circuit on the sentinel.
+
+### `crates/ailang-codegen/src/lib.rs`
+
+Modify:
+
+- `intern_string` (~`:2487`) — switch the LLVM-IR emission of
+  static-Str globals from `[N+1 x i8] c"…\00"` to
+  `<{i64, i64, [N+1 x i8]}> <{i64 -1, i64 N, c"…\00"}>`. The
+  global name stays the same; the way callsites materialise a
+  pointer to it changes to a `getelementptr` landing on the
+  `len`-field.
+- IR-header preamble (~`:455`–`:498`): add
+  `declare ptr @ailang_int_to_str(i64)` and
+  `declare ptr @ailang_float_to_str(double)`.
+- `float_to_str` arm (~`:1829`): drop the `Err(CodegenError::Internal(…))`,
+  emit the call instead.
+- New arm parallel to it for `int_to_str`.
+- `compare__Str`, `eq__Str`, `io/print_str` codegen sites
+  (~`:2152`, `:2262`, `:2313`): emit a +8 `getelementptr` to
+  land on the bytes pointer before calling `@ail_str_compare`,
+  `@ail_str_eq`, `@puts` respectively.
+- **Per-type drop walks are unchanged.** `Str` is already routed
+  to `ailang_rc_dec` by `field_drop_call`
+  (`crates/ailang-codegen/src/drop.rs:372`). The stale comment
+  at that site (*"NUL-terminated bytes in static memory; nothing
+  to recurse into"*) becomes incorrect once heap-Str ships and
+  must be updated to reflect the new dual-realisation reality
+  — but the dispatched symbol (`ailang_rc_dec`) is already
+  correct.
+
+### `crates/ailang-check/src/builtins.rs`
+
+Add `int_to_str` to the builtin signatures table (~`:194`,
+mirroring `float_to_str`'s entry), and add the type-installation
+unit test parallel to `install_float_to_str_signature`.
+
+### `crates/ailang-codegen/src/synth.rs`
+
+Add an `int_to_str` arm parallel to the `float_to_str` arm at
+`:167` so the codegen's local type-replay knows the signature.
+
+### `docs/DESIGN.md`
+
+- §"Float semantics", last paragraph: remove the "codegen lowering
+  deferred" caveat on `float_to_str`.
+- §"What is not (yet) supported" and the builtin inventory under
+  "What **is** supported": add `int_to_str : (Int) -> Str` next
+  to `float_to_str`.
+- New top-level subsection "Str ABI" (or under an appropriate
+  Decision-10 / runtime-contracts anchor): document the unified
+  slab layout `{rc_header, len, bytes…, NUL}`, the sentinel-
+  header convention (`UINT64_MAX` = static / immortal), and the
+  inherited limitation that `==` / `<` use `strcmp` semantics
+  and do not support embedded `\0` bytes (this is a future-
+  milestone enhancement, not a regression).
+
+## Data flow
+
+Three representative IR cases pin the after-state.
+
+**Static string literal.** Today, `"hello"` is emitted as:
+
+```llvm
+@.str_M_hello_0 = private unnamed_addr constant [6 x i8] c"hello\00"
+…
+%1 = call i32 @puts(ptr @.str_M_hello_0)
+```
+
+After the milestone:
+
+```llvm
+@.str_M_hello_0 = private unnamed_addr constant <{i64, i64, [6 x i8]}>
+                  <{ i64 -1, i64 5, [6 x i8] c"hello\00" }>
+…
+%payload = getelementptr inbounds <{i64, i64, [6 x i8]}>,
+                                  ptr @.str_M_hello_0, i32 0, i32 1
+%bytes   = getelementptr inbounds i8, ptr %payload, i64 8
+%1       = call i32 @puts(ptr %bytes)
+```
+
+`%payload` is the value that flows through the IR as a `Str`. The
+`%bytes` adjustment is local to the `@puts` callsite.
+
+**Heap-allocated string from `int_to_str(42)`.**
+
+```llvm
+%s = call ptr @ailang_int_to_str(i64 42)
+; …print:
+%bytes = getelementptr inbounds i8, ptr %s, i64 8
+%1     = call i32 @puts(ptr %bytes)
+; …at scope close:
+call void @ailang_rc_dec(ptr %s)
+```
+
+`%s` is a fresh heap slab with `rc_header == 1`. `ailang_rc_dec`
+accesses the header at `%s - 8`, decrements to zero, frees.
+
+**`eq Str` with mixed-origin operands.**
+
+```llvm
+define i1 @eq__Str(ptr %a, ptr %b) {
+  %a_bytes = getelementptr inbounds i8, ptr %a, i64 8
+  %b_bytes = getelementptr inbounds i8, ptr %b, i64 8
+  %r       = call zeroext i1 @ail_str_eq(ptr %a_bytes, ptr %b_bytes)
+  ret i1 %r
+}
+```
+
+Either operand may be static (sentinel header) or heap (real
+header); the body does not distinguish. `@ail_str_eq` keeps its
+`strcmp`-based body unchanged.
+
+**ADT drop with a `Str` field.** For `type Boxed = | Box(Str)`,
+the per-type `drop_M_Boxed(ptr %cell)` already emits a
+`call void @ailang_rc_dec(ptr %str_field)` today — `Str` fields
+are dispatched to `ailang_rc_dec` by `field_drop_call` regardless
+of whether the value originated from a static or a heap source.
+What changes is the *meaning* of that call on a static-Str
+pointer: pre-milestone it reads bytes outside the `[N x i8]`
+global (latent UB; no shipping test exercises the path);
+post-milestone it reads the new global's leading `i64 -1` slot,
+hits the sentinel short-circuit in `ailang_rc_dec`, and returns
+without modifying anything. For a heap-Str field, the normal
+refcount drop runs unchanged. The single behavioural shift —
+from latent UB to defined no-op on static fields — is gated by
+the new `str_field_in_adt_drops_static_str_noop` E2E test (see
+§Testing strategy).
+
+## Error handling
+
+Four failure modes; most are inherited or structurally impossible.
+
+**Out-of-memory at heap-Str allocation.** `ailang_rc_alloc`
+already `abort()`s on OOM, matching Boehm. `str_alloc`,
+`ailang_int_to_str`, `ailang_float_to_str` propagate. No new
+recovery path.
+
+**`snprintf` truncation in the runtime formatters.** The 64-byte
+stack buffer is strictly larger than the worst-case width of
+either `%lld` (longest i64 = 20 chars) or `%g` on a `double`
+(~25 chars worst case). The runtime body asserts no truncation
+defensively (the `snprintf` return tells us how many bytes
+*would* have been written; an `abort()` fires if that exceeds
+the buffer). The defensive check survives format-string changes
+that might widen the output without anyone noticing the buffer
+is too small.
+
+**Sentinel collision.** A real refcount can never reach
+`UINT64_MAX` — `2^64` references would require more memory than
+exists on any machine the program can run on. The convention
+`rc_header == UINT64_MAX` ⇒ static is structurally safe; no
+runtime check beyond the equality test in `inc`/`dec` is needed.
+DESIGN.md §"Str ABI" records this as an invariant.
+
+**Embedded `\0` bytes in static-Str literals from JSON.** Status
+quo: a literal containing `""` ends up with a `NUL` in the
+middle of the slab; `strcmp`-based comparison treats it as the
+end of string. This is the *current* behaviour of static-Str and
+heap-Str inherits it unchanged. Heap-Str values produced by
+`int_to_str` and `float_to_str` themselves cannot contain
+embedded `\0` (neither `%lld` nor `%g` emits one), so the
+milestone scope is not affected. DESIGN.md §"Str ABI" notes the
+limitation explicitly with a forward pointer to a future
+length-aware-compare milestone.
+
+`float_to_str`'s `CodegenError::Internal` path is *removed*
+entirely after the milestone — neither as a caught nor as an
+uncaught error. The symbol-installation in the builtin tier and
+the codegen lowering are landed in the same iteration; there is
+no intermediate state where one exists without the other.
+
+## Testing strategy
+
+Three layers.
+
+### IR-shape tests in the codegen crate
+
+Pin the structural guarantees of §"Data flow" with substring
+matches against emitted IR:
+
+- `static_str_global_uses_packed_struct_with_sentinel_and_len` —
+  pins the new slab format.
+- `static_str_callsite_pointer_is_payload_not_slab` — pins the
+  `getelementptr`-to-second-field convention.
+- `eq_str_calls_ail_str_eq_with_bytes_pointer`,
+  `compare_str_calls_ail_str_compare_with_bytes_pointer`,
+  `print_str_calls_puts_with_bytes_pointer` — pin the +8 GEP at
+  each existing C-API callsite.
+- `int_to_str_lowers_to_ailang_int_to_str_call` — pins the new
+  builtin's lowering.
+- `float_to_str_no_longer_errors_internal` — converts the
+  existing "lowering deferred"-pinning test into a positive
+  green test.
+
+These tests are fast but sensitive to SSA-naming changes. The
+existing test style in `crates/ailang-codegen/src/lib.rs` already
+accepts this risk for analogous pins (e.g.
+`eq_str_mono_symbol_emits_ail_str_eq_call`); the new tests match.
+
+### Stdout E2E tests in `crates/ail/tests/e2e.rs`
+
+Observable program behaviour:
+
+- `int_to_str_smoke` — `int_to_str(42)` prints `"42\n"`; plus
+  edge cases `0`, `-1`, `i64::MAX`, `i64::MIN`.
+- `float_to_str_smoke` — `float_to_str(3.5)` prints something
+  starting with `3.5`; plus `0.0`, `NaN`, `+Inf`, `-Inf` (pinned
+  against libc `%g` rendering — implementations are permitted to
+  print NaN as `nan`, `-nan`, or other casing; the test pins the
+  exact bytes observed on the dev target with a comment
+  acknowledging the libc-defined freedom).
+- `static_str_print_unchanged` — an existing example
+  (`examples/hello.ail` or similar) produces byte-identical
+  stdout before and after the milestone, sourced from
+  pre-milestone `expected/`.
+
+### RC-discipline E2E tests via `AILANG_RC_STATS`
+
+Catch leaks and double-frees:
+
+- `int_to_str_drop_balances_rc_stats` — `int_to_str` is called
+  N times in a loop, each result dropped without aliasing;
+  `AILANG_RC_STATS=1` reports `allocs=K frees=K live=0` for some
+  K matched to the loop bound + setup. Pins that heap-Str
+  participates in RC normally.
+- `str_field_in_adt_drops_heap_str_correctly` — a value of
+  `type Boxed = | Box(Str)` is instantiated with a heap-Str
+  (from `int_to_str`) and dropped; `live=0`.
+- `str_field_in_adt_drops_static_str_noop` — same ADT
+  instantiated with a static literal under `--alloc=rc`; `live=0`,
+  no crash. **Load-bearing:** no precursor test today exercises
+  this path (the existing `alloc_rc_*` tests do not instantiate
+  an ADT with a `Str` field), so today's behaviour on this path
+  is undefined. The test gates the milestone-level safety
+  property that static-Str pointers in ADT fields drop safely
+  via the sentinel short-circuit.
+
+### Bench gates
+
+At audit close (per `skills/audit/SKILL.md`):
+
+- `bench/check.py` — existing latency baselines green, with the
+  known `latency.explicit_at_rc.*` nondeterminism tolerance
+  documented in recent audits.
+- `bench/compile_check.py` — entire `examples/` corpus compiles.
+- `bench/cross_lang.py` — byte-identical outputs vs. the C
+  reference corpus in `bench/reference/`. This is the strongest
+  "we did not break anything" gate because it compares output
+  against a second-language implementation, not against AILang
+  itself.
+
+### Out of testing scope
+
+Embedded-`\0` behaviour. Concurrent RC (AILang is single-
+threaded). Allocator stress beyond representative program sizes.
+
+## Acceptance criteria
+
+The milestone closes when all the following are observable.
+
+**Language surface.**
+
+- `int_to_str : (Int) -> Str` is installed in
+  `crates/ailang-check/src/builtins.rs` and resolved by the
+  codegen builtin lookup.
+- `float_to_str` no longer lowers to `CodegenError::Internal`;
+  calling it produces a working binary.
+- DESIGN.md §"Float semantics" carries no "codegen lowering
+  deferred" caveat on `float_to_str`; the builtin inventory
+  under "What **is** supported" lists `int_to_str` alongside
+  `float_to_str`.
+
+**Runtime / ABI.**
+
+- `runtime/str.c` exports `ailang_int_to_str(int64_t) -> char*`
+  and `ailang_float_to_str(double) -> char*`. The private
+  `str_alloc(uint64_t)` is *not* in the export symbol set.
+- `runtime/rc.c` `ailang_rc_inc`/`ailang_rc_dec` short-circuit
+  on `*header == UINT64_MAX`.
+- Static-Str globals are emitted as
+  `<{i64, i64, [N+1 x i8]}>` with values `{i64 -1, i64 N, c"…\00"}`.
+- DESIGN.md has a "Str ABI" anchor (or equivalent) documenting
+  the unified layout, sentinel convention, and inherited
+  `strcmp`-based comparison semantics.
+
+**Observable program behaviour.**
+
+- A trivial `.ail` program `do io/print_str(int_to_str(42))`
+  builds, runs, and prints `42`.
+- The analogous `float_to_str(3.5)` program builds, runs, and
+  prints `%g`-conformant output (exact bytes pinned by the E2E
+  test).
+- All existing `examples/` programs produce byte-identical
+  stdout vs. pre-milestone (no regressions).
+
+**RC discipline.**
+
+- The three new `AILANG_RC_STATS`-based E2E tests pass.
+- No existing drop-relevant test breaks.
+
+**Bench gates.**
+
+- `bench/check.py`, `bench/compile_check.py`, `bench/cross_lang.py`
+  all green at audit close.
+
+**Documentation / journal.**
+
+- One journal entry per iteration with per-task notes.
+- `WhatsNew.md` gets a milestone-close entry (verbatim with the
+  done-state Notify), written for the user-as-reader without
+  iteration codes or crate names.
+
+**Explicitly *not* part of acceptance.**
+
+- `++` / Str concatenation operator (own milestone).
+- `Show` typeclass and `print` rewire (own milestone, depends-on
+  this milestone).
+- Length-aware comparison semantics / embedded-`\0` support.