From 74849160b768a850be1e687ad918d68488b8eb7d Mon Sep 17 00:00:00 2001 From: Brummel Date: Tue, 2 Jun 2026 16:32:31 +0200 Subject: [PATCH] spec: cross-model harness corpus revival (refs #68) The cma authoring-form harness (experiments/2026-05-12-cross-model-authoring) is milestone-complete but its corpus went schema-dead: the implicit-cutover made param_modes/ret_mode mandatory, so today's ail check rejects 100% of the master examples + reference solutions and the renderer crashes on load. This spec revives the corpus to the current language and gets a mock run green. Scope (corpus-first, method-form-later per #68): minimal field-completion preserving existing modes; two new author-facing examples (loop/recur, new); spec.md section 4 rewritten to own/borrow with the borrow-over-value rule surfaced under the parse-gate; spec_completeness allowlists the non-authorable Term::Intrinsic out; mock fixture migrated so harness score assertions hold. No live IONOS call. Old specs 0017/0018 left intact as closed-milestone history. --- docs/specs/0069-cma-corpus-revival.md | 261 ++++++++++++++++++++++++++ 1 file changed, 261 insertions(+) create mode 100644 docs/specs/0069-cma-corpus-revival.md diff --git a/docs/specs/0069-cma-corpus-revival.md b/docs/specs/0069-cma-corpus-revival.md new file mode 100644 index 0000000..4de394b --- /dev/null +++ b/docs/specs/0069-cma-corpus-revival.md @@ -0,0 +1,261 @@ +# Cross-model harness corpus revival — Design Spec + +**Date:** 2026-06-02 +**Status:** Draft — ready for planner +**Authors:** Brummel (orchestrator) + Claude +**Tracker:** refs #68 + +## Goal + +The cross-model authoring-form harness in +`experiments/2026-05-12-cross-model-authoring/` (shipped by milestones +cma.1-3, ms.1-2) is milestone-complete but its corpus is **schema-dead**. +The `render` and `harness` crates still build against current +`ailang-core`/`ailang-surface`, but every `master/examples/*.ail.json` +and `master/tasks/*.reference.ail.json` was authored when `param_modes` +could be omitted. The #55 implicit-cutover (`76b21c0`) deleted +`ParamMode::Implicit` and made `param_modes`/`ret_mode` mandatory on +`Type::Fn`. Consequences, all reproduced 2026-06-02: + +- `ail check master/tasks/t1_add_three.reference.ail.json` + → `Error: missing field 'param_modes'`. 100% of the corpus is rejected. +- `render` crashes loading the first example + (`missing field 'param_modes' at line 19`). +- `render/tests/spec_completeness.rs` no longer compiles: it matches on + the deleted `ParamMode::Implicit`, and its `Term` walker is + non-exhaustive against four variants that landed since May — `Loop`, + `Recur`, `New`, `Intrinsic`. + +This iteration brings the corpus to the language as it ships today and +gets a **mock** run green. It does **not** make a live IONOS call and +does **not** decide the recurring-method form (skill vs. tool) — both +are deferred per #68. + +This spec is **new**; the closed milestone specs +`docs/specs/0017-cross-model-authoring-form-test.md` and +`0018-multi-subject-codellama.md` are **not** rewritten. They correctly +describe the then-state (`.ailx`, `docs/DESIGN.md`); editing them would +falsify the history they record (honesty-rule, `0007-honesty-rule.md`). + +## What changed in the language since the corpus was written + +Mapped 2026-06-02 against the current tree: + +1. **`param_modes` + `ret_mode` mandatory.** Every `Type::Fn` in canonical + JSON now carries `param_modes` (a list of `own`/`borrow`, one per + parameter) and `ret_mode`. `ParamMode::Implicit` no longer exists. +2. **Three new author-facing Term variants** — surface-writable, appear in + ordinary `examples/`, classified author-facing in the ledger: + - `Loop` / `Recur` — bounded iteration (`design/contracts/0012-tail-calls.md`, + `docs/specs/0034-loop-recur.md`). Surface keywords `loop` / `recur`. + - `New` — kernel-extension type construction + (`design/models/0007-kernel-extensions.md`). Surface keyword `new`. +3. **One new Term variant that is NOT author-facing** — `Intrinsic`, the + `(intrinsic)` body-marker for compiler-supplied function bodies. It is + **not** in the parser's term dispatch and the typechecker rejects it + outside a kernel-tier/prelude module + (`crates/ailang-core/src/ast.rs` ~609; `design/glossary.md`). A foreign + model neither can nor should produce it. + +## Architecture + +### Repair strategy — minimal field-completion, modes preserved + +The mode of each parameter is a **semantic** property (does the callee +consume the argument, or borrow it?), not a free mechanical field. But +recon (2026-06-02) shows most corpus files **already carry their +`param_modes` explicitly** — `data_simple` (`own`), `data_with_match` +(`borrow`), `param_modes_all` (both). What is uniformly absent is +`ret_mode`; only a handful lack `param_modes` outright (the 0-param fns +need `[]`; `fn_calls_prelude.add`, `fn_with_lambda.make_adder`, +`match_literal_pattern`'s two fns need values). So the Spec goal — modes +author-set, not guessed — is already met for the files that set them. The +repair must **preserve** those, not overwrite them. + +Strategy, per `master/examples/*.ail.json` and +`master/tasks/*.reference.ail.json`: + +1. Complete the missing schema fields in place, preserving every existing + `param_modes` value verbatim: + - add `ret_mode` where absent — `own` (a function returns a fresh owned + value; `borrow`-return is forbidden by `0015`, so `own` is the only + legal choice here); + - add `param_modes` where absent — `[]` for 0-param fns; for the + value-taking fns, `own` per parameter (the natural consume default). +2. `ail check .ail.json` exit 0 is the per-file oracle: a wrong mode is + a check error, so a file that checks green has a ratified mode set. +3. `example_roundtrip` (load → print `.ail` → reparse → same canonical + bytes) is the second gate; it canonicalises on compare, so field-order + in the edited JSON is not load-bearing as long as the file loads. + +A full Form-A re-author is **not** used for the existing corpus (it would +risk flattening the deliberate `borrow` annotations to `own`). It *is* the +path for the two NEW examples below, where the `.ail` is authored fresh and +`ail parse .ail > .ail.json` produces canonical JSON. Both paths sit +on the canonical-authoring-form machinery the round-trip invariant protects +(`0009-roundtrip-invariant.md`), so neither can silently drift. + +### Mode-annotations section (spec.md §4) — rewrite + +`master/examples/param_modes_all.ail.json` currently defines `f_implicit`, +`f_own`, `f_borrow` and its doc-strings explain `ParamMode::Implicit` as the +"legacy default". `Implicit` is gone — **and** a second drift surfaced under +the parse-gate (2026-06-02): the typechecker now rejects `(borrow V)` over an +unboxed value type with `borrow-over-value` (`crates/ailang-check/src/lib.rs` +~813; unboxed = `Int`/`Bool`/`Float`/`Unit`, `linearity.rs:175`). The old +`f_borrow` borrows over `Int` — illegal today. `borrow` is only legal over a +**boxed** type (an ADT). + +So the rewrite cannot be "two Int functions". It must show the mode set *and* +the value/boxed rule: a small boxed ADT, an `own` function over an unboxed +`Int`, and a `borrow` function over the ADT. The example is verified to +check green (see plan Task 2). spec.md §4 prose presents the exhaustive rule: +every parameter carries exactly one of `own`/`borrow`; `own` is mandatory for +unboxed value types (`borrow` over them is a `borrow-over-value` error); +`borrow` is available only for boxed types; `ret_mode` is always `own`. No +mention of a defaulted/omitted mode. + +### Two new spec.md sections + examples + +The mini-spec must show a foreign author the constructs the language now +has. Add two sections and two examples (AST-source `.ail.json`, regenerated +from the `.ail` below via `ail parse`): + +**§ Iteration (`loop` / `recur`)** — example `loop_sum`: + +``` +(module loop_sum + (fn sum_to + (type (fn-type (params (own (con Int))) (ret (own (con Int))))) + (params n) + (body + (loop (acc (con Int) 0) (i (con Int) 1) + (if (app gt i n) + acc + (recur (app + acc i) (app + i 1))))))) +``` + +**§ Kernel-extension types (`new`)** — example `new_rawbuf`: + +``` +(module new_rawbuf + (fn main + (type (fn-type (params) (ret (own (con Unit))) (effects IO))) + (params) + (body + (let b (new RawBuf (con Int) 3) + (app print (app RawBuf.size b)))))) +``` + +Both are reductions of shipped, checked `examples/` fixtures +(`loop_sum_to.ail`, `new_rawbuf_size_only.ail`), so they are known to +parse, check, build, and run under today's toolchain. + +### spec_completeness.rs — the one design cut + +The completeness test asserts every `Term`/`ParamMode` variant is exercised +by at least one master example. Update it: + +- Remove the `ParamMode::Implicit` arm. +- Cover `Loop`, `Recur`, `New` via the two new examples. +- **Allowlist `Intrinsic` OUT**, with an in-code comment stating the reason: + `Intrinsic` is non-authorable (absent from the parser term dispatch; + typecheck enforces kernel-tier-only), so a foreign-author mini-spec must + not show it, and no master example can legally contain it. This mirrors + the harness's existing `OPTIMISATION_ONLY` allowlist precedent — a named, + justified exemption rather than a silent gap. + +### Mock fixture migration (harness) — corpus-decoupled, but in scope + +`harness/tests/fixtures/mock_full_run.json` is a **separate** program +store from the `master/` corpus: it holds hand-authored *simulated model +output* (string-embedded `.ail.json` programs, one per cohort×task×turn) +that the harness pipeline scores by running `ail check` + `ail build` on +each. Every embedded program is legacy-schema (`grep -c param_modes` → 0), +so under today's mandatory `param_modes` they all fail `ail check`. Two +harness tests depend on this fixture and would break: + +- `mock_full_run.rs` asserts `json,t3_main_prints,false,2` (t3 in the json + cohort goes green on **turn 2**), `ail,t1_add_three,false,INF` (runs to + the turn limit, never green), exactly 8 score rows, and `RUN_STATUS=ok`. + The `,2` row breaks under schema-death: t3's turn-2 program no longer + reaches green, so `turns_to_green` flips to `INF`. +- `budget_abort.rs` asserts `RUN_STATUS=budget_exceeded` and ≥1 + `budget_abort` row. This depends only on the token economy (the fixture's + `usage` fields, ~1200/turn vs. a 1500 budget), **not** on the check + result, so it is robust to the schema drift — but must be re-verified + green, and the `usage` fields must be left untouched. + +**Repair:** migrate the embedded `content` programs to the current schema +(`param_modes`/`ret_mode`), **except** the deliberately content-broken ones +(e.g. t3 json turn 1 is `…"defs":[]` — an intentional first-attempt miss +that must stay broken-by-content). Hard requirements pinned by the +assertions: + +- t3_main_prints / json / turn 2 → a valid current-schema program that + passes `ail check` + `ail build` + run, so the `,2` row holds. +- t1_add_three / ail → stays failing to the turn limit, so `INF` holds; its + failure should be **content-driven** (the fixture's intended miss), not a + silent schema-drift artefact — otherwise the fixture stops testing what it + documents. + +Leave the `usage` token fields byte-for-byte unchanged. This keeps the mock +an honest simulation of a model authoring against *today's* language, rather +than a fixture that passes only because everything uniformly schema-fails. + +`harness/tests/fixtures/check_schema_missing_field.stderr` (open question +raised in recon) is a **no-op**: it pins `missing field ``type```, not +`param_modes`, so it is independent of this drift and stays green. + +## Components + +- `experiments/2026-05-12-cross-model-authoring/master/examples/*.ail.json` + (13 files) — regenerated with `param_modes`/`ret_mode`. +- `.../master/tasks/*.reference.ail.json` (4 files) — regenerated. +- `.../master/examples/param_modes_all.ail.json` — rewritten (own/borrow only). +- `.../master/examples/loop_sum.ail.json`, `.../new_rawbuf.ail.json` — new. +- `.../master/spec.md` — §4 rewritten; two new sections added; their + `{example: …}` markers wired. +- `.../render/src/*` — only if the renderer needs a new section/example + hook; prefer pure data additions. +- `.../render/tests/spec_completeness.rs` — variant coverage + Intrinsic + allowlist. +- `.../rendered/json.md`, `.../rendered/ail.md` — regenerated output, + re-committed for review. +- `.../harness/tests/fixtures/mock_full_run.json` — embedded `content` + programs migrated to current schema (except the deliberately + content-broken ones); `usage` fields untouched. + +## Data flow + +1. Re-author each corpus `.ail.json` as `.ail`; `ail parse` → new `.ail.json`; + `ail check` green per file. +2. Add the two new examples + two spec.md sections; rewrite §4. +3. `cargo run --manifest-path .../render/Cargo.toml -- --master … --rendered …` + succeeds; inspect regenerated `rendered/`. +4. `cargo test --manifest-path .../render/Cargo.toml` green + (`example_roundtrip`, `spec_completeness`, `token_balance`, splitter). +5. Migrate `mock_full_run.json` embedded programs (per §Mock fixture + migration); leave `usage` untouched. +6. `cargo test --manifest-path .../harness/Cargo.toml` green (mock E2E + `mock_full_run` with the `false,2` + `INF` rows holding, + `verify_references` driving every reference through the real ail+clang + pipeline, strip_locations, budget_abort). + +## Error handling / acceptance + +- Per-file bar: `ail check .ail.json` exit 0. +- Renderer bar: clean run, no load crash. +- Suite bar: both crates' `cargo test` green. The harness `verify_references` + test is the end-to-end gate — it compiles and runs every reference solution + through the actual toolchain, so a green there means the references are not + just schema-valid but executable. +- No live IONOS call. No `harness` invocation with `--model` other than `mock`. + +## Out of scope + +- The recurring-method form (project-local skill vs. light triggered tool). +- Any live IONOS run — gated on explicit per-session consent. +- Multi-subject / token-economy re-measurement (the old `runs/` datasets are + left as historical; they are gitignored raw data, not corpus). +- Rewriting specs 0017/0018.