diff --git a/docs/plans/0112-reserved-dollar-in-names.md b/docs/plans/0112-reserved-dollar-in-names.md new file mode 100644 index 0000000..b0bb7e2 --- /dev/null +++ b/docs/plans/0112-reserved-dollar-in-names.md @@ -0,0 +1,291 @@ +# Reserve `$` in the Form-A lexer — Implementation Plan + +> **Parent spec:** `docs/specs/0057-reserved-dollar-in-names.md` +> +> **For agentic workers:** REQUIRED SUB-SKILL: use the +> `implement` skill to run this plan. Steps use `- [ ]` +> checkboxes for tracking. + +**Goal:** Reject `$` in any Form-A identifier token at the lexer, so +the compiler's `$`-synthetic namespace (`$mp_N`, `$lr_N`, +`$`) can never collide with an authored name (closes the +latent collision class behind #44). + +**Architecture:** A single new `LexError::ReservedDollar` variant plus +a one-line guard inline in `tokenize`, raised on any token run +containing `$`, before int/float/ident classification. The error +surfaces through the existing `ParseError::Lex` → `W::SurfaceParse` → +`surface-parse-error` channel with zero new wiring. String- and +comment-internal `$` stay legal because both are consumed by earlier +branches of the scan loop, before a run is ever sliced. A stale +doc-comment on `fresh_binder` is corrected to state the now-enforced +invariant. + +**Tech Stack:** `crates/ailang-surface/src/lex.rs` (lexer + in-source +tests), `crates/ailang-surface/tests/` (integration test), +`crates/ailang-core/src/desugar.rs` (doc-comment). + +--- + +**Files this plan creates or modifies:** + +- Modify: `crates/ailang-surface/src/lex.rs:75` — add `ReservedDollar { token: String, start: usize }` to the `LexError` enum (after the `InvalidEscape` variant at line 75, before the enum's closing `}` at line 76). +- Modify: `crates/ailang-surface/src/lex.rs:184` — insert the `$`-reject guard between `let raw = &input[start..i];` (line 183) and `let first = raw.as_bytes()[0];` (line 185). +- Test: `crates/ailang-surface/src/lex.rs:~509` — add 3 in-source tests inside `mod tests` (before its closing `}` at line 510): `dollar_in_ident_is_reserved`, `dollar_in_string_literal_is_allowed`, `dollar_in_comment_is_allowed`. +- Create: `crates/ailang-surface/tests/reserved_dollar_pin.rs` — integration test feeding a full `(module …)` with a `$` binder through `parse`, asserting `Err(ParseError::Lex(LexError::ReservedDollar { … }))`. +- Modify: `crates/ailang-core/src/desugar.rs:528-536` — correct the `fresh_binder` doc-comment (the now-false "an authored binder may legally contain `$`" / "tracked as a follow-up" claims). + +--- + +## Task 1: Lexer `$` reservation (variant + guard + in-source tests) + +**Files:** +- Modify: `crates/ailang-surface/src/lex.rs:75` (enum variant) +- Modify: `crates/ailang-surface/src/lex.rs:184` (guard) +- Test: `crates/ailang-surface/src/lex.rs:~509` (3 in-source tests) + +- [ ] **Step 1: Add the `ReservedDollar` variant to `LexError`** + +In `crates/ailang-surface/src/lex.rs`, the `LexError` enum currently +ends: + +```rust + #[error("invalid escape sequence \\{ch} in string literal at byte {pos}")] + InvalidEscape { ch: char, pos: usize }, +} +``` + +Add the new variant immediately before the closing `}` (after +`InvalidEscape`): + +```rust + #[error("invalid escape sequence \\{ch} in string literal at byte {pos}")] + InvalidEscape { ch: char, pos: usize }, + #[error("reserved character `$` in token {token:?} at byte {start}; \ + `$` is reserved for compiler-synthetic names")] + ReservedDollar { token: String, start: usize }, +} +``` + +- [ ] **Step 2: Write the failing test `dollar_in_ident_is_reserved`** + +In the `#[cfg(test)] mod tests` block at the bottom of +`crates/ailang-surface/src/lex.rs` (closes at line 510), add this test +before the module's closing `}`: + +```rust + #[test] + fn dollar_in_ident_is_reserved() { + let err = tokenize("x$1").expect_err("`$` in an ident must be a lex error"); + assert!( + matches!(err, LexError::ReservedDollar { ref token, .. } if token == "x$1"), + "expected ReservedDollar(\"x$1\"), got {err:?}", + ); + } +``` + +- [ ] **Step 3: Run the test to verify it fails** + +Run: `cargo test -p ailang-surface dollar_in_ident_is_reserved` +Expected: FAIL — the test panics with +`` `$` in an ident must be a lex error `` (because the guard does not +exist yet, `tokenize("x$1")` returns `Ok`, so `expect_err` panics). + +- [ ] **Step 4: Add the `$`-reject guard inline in `tokenize`** + +In `crates/ailang-surface/src/lex.rs`, the run-classification currently +reads (lines 183-186): + +```rust + let raw = &input[start..i]; + // Classify. + let first = raw.as_bytes()[0]; + let is_int = first.is_ascii_digit() +``` + +Insert the guard immediately after the `let raw` line, before the +`// Classify.` comment: + +```rust + let raw = &input[start..i]; + // raw-buf.#44: `$` is reserved for compiler-synthetic names + // (`$mp_N`, `$lr_N`, `$`). Reject any authored + // token run containing it, before int/float/ident classification. + if raw.contains('$') { + return Err(LexError::ReservedDollar { token: raw.to_string(), start }); + } + // Classify. + let first = raw.as_bytes()[0]; + let is_int = first.is_ascii_digit() +``` + +- [ ] **Step 5: Run the test to verify it passes** + +Run: `cargo test -p ailang-surface dollar_in_ident_is_reserved` +Expected: PASS. + +- [ ] **Step 6: Add the two exemption tests** + +In the same `mod tests` block (before its closing `}`), add: + +```rust + #[test] + fn dollar_in_string_literal_is_allowed() { + // The `$` lives inside a string; it must NOT trip the reservation. + let toks = tokenize(r#""price: $5""#).expect("string-internal `$` is legal"); + assert!(matches!(toks.as_slice(), [Token { tok: Tok::Str(s), .. }] if s == "price: $5")); + } + + #[test] + fn dollar_in_comment_is_allowed() { + // Comment is stripped before tokenization; the ident `x` survives. + let toks = tokenize("x ; mentions loop$lr_0\n").expect("comment `$` is legal"); + assert!(matches!(toks.as_slice(), [Token { tok: Tok::Ident(s), .. }] if s == "x")); + } +``` + +- [ ] **Step 7: Run the exemption tests + the full lexer suite** + +Run: `cargo test -p ailang-surface --lib` +Expected: PASS — all lexer tests green, including the 19 pre-existing +ones plus the 3 new (`dollar_in_ident_is_reserved`, +`dollar_in_string_literal_is_allowed`, `dollar_in_comment_is_allowed`). +Confirm the run reports 22 lexer tests, not 19 (the count proves the 3 +new tests were collected and ran). + +--- + +## Task 2: Correct the `fresh_binder` doc-comment + +**Files:** +- Modify: `crates/ailang-core/src/desugar.rs:528-536` + +- [ ] **Step 1: Replace the stale doc paragraph** + +In `crates/ailang-core/src/desugar.rs`, the `fresh_binder` doc-comment +currently reads (lines 528-536): + +```rust + /// `$` is the project's convention for synthetic names, but it is + /// NOT lexer-enforced — an authored binder may legally contain `$`. + /// The probe rejects a `$` that is already in `used` or + /// currently in `scope`, but it does NOT see an authored + /// `$` that is out of scope at mint time yet later binds + /// under the same `(def, name)` key. No fixture exercises an + /// authored `$` binder; closing that latent gap (enforce the + /// reservation, or seed `used` with every authored binder name in + /// the def) is tracked as a follow-up. +``` + +Replace those 9 lines with: + +```rust + /// `$` is reserved for the compiler's synthetic namespace and is + /// rejected in any authored Form-A identifier by the lexer + /// (`ailang_surface::lex::tokenize` → `LexError::ReservedDollar`). + /// So a `$` candidate minted here cannot alias any + /// authored name — no authored name can contain `$`. The probe + /// against `used` and `scope` therefore guards only against prior + /// synthetic mints and currently in-scope renamed binders. (The + /// reservation is on *Form-A authored* identifiers; the `.ail.json` + /// deserialization path is intentionally not guarded — it is the + /// orchestrator's self-responsible channel, not client-reachable.) +``` + +- [ ] **Step 2: Verify `ailang-core` still builds** + +Run: `cargo build -p ailang-core` +Expected: build succeeds (doc-comment-only change; 0 errors). + +--- + +## Task 3: Integration test — the surfacing chain end-to-end + +**Files:** +- Create: `crates/ailang-surface/tests/reserved_dollar_pin.rs` + +- [ ] **Step 1: Write the integration test** + +Create `crates/ailang-surface/tests/reserved_dollar_pin.rs` with: + +```rust +//! #44: a `$` in an authored Form-A identifier is rejected at the +//! lexer and surfaces through `parse` as `ParseError::Lex`. Pins that +//! the reservation reaches the public `parse` entry point (not just +//! the internal `tokenize`), so a hallucinating Form-A author cannot +//! land a `$` binder in a `Module`. + +// `parse`, `ParseError`, `LexError` are all re-exported at the crate +// root (`pub use` in lib.rs); the `lex` / `parse` modules themselves +// are private, so import from the crate root, not the module paths. +use ailang_surface::{parse, LexError, ParseError}; + +#[test] +fn dollar_binder_in_module_is_rejected_at_parse() { + // (module bad (fn main (type (fn-type (params) (ret (con Int)))) (params) (body (let x$1 7 x$1)))) + let src = "(module bad\n (fn main\n (type (fn-type (params) (ret (con Int))))\n (params)\n (body\n (let x$1 7 x$1))))\n"; + let err = parse(src).expect_err("authored `$` binder must be rejected at parse"); + assert!( + matches!(err, ParseError::Lex(LexError::ReservedDollar { ref token, .. }) if token == "x$1"), + "expected ParseError::Lex(ReservedDollar(\"x$1\")), got {err:?}", + ); +} + +#[test] +fn dollar_in_string_in_module_parses_clean() { + // The exemption survives at the `parse` level too: `$` inside a + // string literal in a real module is fine. + let src = "(module dollar_ok\n (fn main\n (type (fn-type (params) (ret (con Str))))\n (params)\n (body \"price: $5\")))\n"; + parse(src).expect("string-internal `$` must parse clean"); +} +``` + +- [ ] **Step 2: Confirm the imports resolve against the crate's public API** + +Run: `cargo build -p ailang-surface --tests` +Expected: build succeeds (0 errors). The imports use the crate-root +re-exports `ailang_surface::{parse, LexError, ParseError}` (confirmed +present in `crates/ailang-surface/src/lib.rs:8-9`); the `lex` / `parse` +modules are private, so the module-qualified paths would NOT compile. + +- [ ] **Step 3: Run the integration test** + +Run: `cargo test -p ailang-surface --test reserved_dollar_pin` +Expected: PASS — both `dollar_binder_in_module_is_rejected_at_parse` +and `dollar_in_string_in_module_parses_clean` green (2 tests ran). + +--- + +## Task 4: No-regression sweep + +**Files:** none (verification only) + +- [ ] **Step 1: Full workspace test suite** + +Run: `cargo test --workspace` +Expected: PASS — all crates green. In particular +`crates/ailang-surface/tests/round_trip.rs` +(`parse_then_print_then_parse_is_idempotent_on_every_ail_fixture`, +`parse_is_deterministic_over_every_ail_fixture`) stays green: no +`examples/*.ail` carries an authored `$` ident, and the six +comment-only `$` files (e.g. `local_rec_let_capture.ail`) survive +because comments are stripped before a run is sliced. + +- [ ] **Step 2: CLI check — comment-`$` example survives end-to-end** + +Run: `cargo run -q -p ail -- check examples/local_rec_let_capture.ail` +Expected: exit 0 (`ok (… symbols …)`) — a real checked-in file whose +comments mention `loop$lr_0` still checks clean, proving the guard +does not reach comment text at the CLI level. + +- [ ] **Step 3: CLI check — the #43 shadowing idiom still compiles clean** + +Run: `cargo run -q -p ail -- build examples/raw_buf_int.ail -o /tmp/rawbuf_int_check` +Expected: exit 0 — `raw_buf_int.ail` is the real nested-`let`-shadow +idiom (`(let buf (new …) (let buf (app RawBuf.set buf …) …))`); its +inner `buf` → `buf$1` rename is minted at desugar, after parse, and is +never re-lexed, so the reservation does not regress it. (If `ail build` +needs different flags in this tree, fall back to +`cargo run -q -p ail -- check examples/raw_buf_int.ail` and assert +exit 0 — check alone still exercises parse → desugar → typecheck over +the shadow rename.)