Brummel/AILang
1
0
Fork 0
Code Issues 29 Pull Requests Actions Packages Projects Releases Wiki Activity

spec: cross-model harness corpus revival (refs #68)

Browse Source 
The cma authoring-form harness (experiments/2026-05-12-cross-model-authoring)
is milestone-complete but its corpus went schema-dead: the implicit-cutover
made param_modes/ret_mode mandatory, so today's ail check rejects 100% of the
master examples + reference solutions and the renderer crashes on load. This
spec revives the corpus to the current language and gets a mock run green.

Scope (corpus-first, method-form-later per #68): minimal field-completion
preserving existing modes; two new author-facing examples (loop/recur, new);
spec.md section 4 rewritten to own/borrow with the borrow-over-value rule
surfaced under the parse-gate; spec_completeness allowlists the non-authorable
Term::Intrinsic out; mock fixture migrated so harness score assertions hold.
No live IONOS call. Old specs 0017/0018 left intact as closed-milestone
history.
This commit is contained in:
Brummel
2026-06-02 16:32:31 +02:00
parent 54d8f0c660
commit 74849160b7
1 changed files with 261 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/specs/0069-cma-corpus-revival.md
+261
View File
@@ -0,0 +1,261 @@
# Cross-model harness corpus revival — Design Spec
**Date:** 2026-06-02
**Status:** Draft — ready for planner
**Authors:** Brummel (orchestrator) + Claude
**Tracker:** refs #68
## Goal
The cross-model authoring-form harness in
`experiments/2026-05-12-cross-model-authoring/` (shipped by milestones
cma.1-3, ms.1-2) is milestone-complete but its corpus is **schema-dead**.
The `render` and `harness` crates still build against current
`ailang-core`/`ailang-surface`, but every `master/examples/*.ail.json`
and `master/tasks/*.reference.ail.json` was authored when `param_modes`
could be omitted. The #55 implicit-cutover (`76b21c0`) deleted
`ParamMode::Implicit` and made `param_modes`/`ret_mode` mandatory on
`Type::Fn`. Consequences, all reproduced 2026-06-02:
- `ail check master/tasks/t1_add_three.reference.ail.json`
→ `Error: missing field 'param_modes'`. 100% of the corpus is rejected.
- `render` crashes loading the first example
(`missing field 'param_modes' at line 19`).
- `render/tests/spec_completeness.rs` no longer compiles: it matches on
the deleted `ParamMode::Implicit`, and its `Term` walker is
non-exhaustive against four variants that landed since May — `Loop`,
`Recur`, `New`, `Intrinsic`.
This iteration brings the corpus to the language as it ships today and
gets a **mock** run green. It does **not** make a live IONOS call and
does **not** decide the recurring-method form (skill vs. tool) — both
are deferred per #68.
This spec is **new**; the closed milestone specs
`docs/specs/0017-cross-model-authoring-form-test.md` and
`0018-multi-subject-codellama.md` are **not** rewritten. They correctly
describe the then-state (`.ailx`, `docs/DESIGN.md`); editing them would
falsify the history they record (honesty-rule, `0007-honesty-rule.md`).
## What changed in the language since the corpus was written
Mapped 2026-06-02 against the current tree:
1. **`param_modes` + `ret_mode` mandatory.** Every `Type::Fn` in canonical
JSON now carries `param_modes` (a list of `own`/`borrow`, one per
parameter) and `ret_mode`. `ParamMode::Implicit` no longer exists.
2. **Three new author-facing Term variants** — surface-writable, appear in
ordinary `examples/`, classified author-facing in the ledger:
- `Loop` / `Recur` — bounded iteration (`design/contracts/0012-tail-calls.md`,
`docs/specs/0034-loop-recur.md`). Surface keywords `loop` / `recur`.
- `New` — kernel-extension type construction
(`design/models/0007-kernel-extensions.md`). Surface keyword `new`.
3. **One new Term variant that is NOT author-facing** — `Intrinsic`, the
`(intrinsic)` body-marker for compiler-supplied function bodies. It is
**not** in the parser's term dispatch and the typechecker rejects it
outside a kernel-tier/prelude module
(`crates/ailang-core/src/ast.rs` ~609; `design/glossary.md`). A foreign
model neither can nor should produce it.
## Architecture
### Repair strategy — minimal field-completion, modes preserved
The mode of each parameter is a **semantic** property (does the callee
consume the argument, or borrow it?), not a free mechanical field. But
recon (2026-06-02) shows most corpus files **already carry their
`param_modes` explicitly** — `data_simple` (`own`), `data_with_match`
(`borrow`), `param_modes_all` (both). What is uniformly absent is
`ret_mode`; only a handful lack `param_modes` outright (the 0-param fns
need `[]`; `fn_calls_prelude.add`, `fn_with_lambda.make_adder`,
`match_literal_pattern`'s two fns need values). So the Spec goal — modes
author-set, not guessed — is already met for the files that set them. The
repair must **preserve** those, not overwrite them.
Strategy, per `master/examples/*.ail.json` and
`master/tasks/*.reference.ail.json`:
1. Complete the missing schema fields in place, preserving every existing
`param_modes` value verbatim:
- add `ret_mode` where absent — `own` (a function returns a fresh owned
value; `borrow`-return is forbidden by `0015`, so `own` is the only
legal choice here);
- add `param_modes` where absent — `[]` for 0-param fns; for the
value-taking fns, `own` per parameter (the natural consume default).
2. `ail check <f>.ail.json` exit 0 is the per-file oracle: a wrong mode is
a check error, so a file that checks green has a ratified mode set.
3. `example_roundtrip` (load → print `.ail` → reparse → same canonical
bytes) is the second gate; it canonicalises on compare, so field-order
in the edited JSON is not load-bearing as long as the file loads.
A full Form-A re-author is **not** used for the existing corpus (it would
risk flattening the deliberate `borrow` annotations to `own`). It *is* the
path for the two NEW examples below, where the `.ail` is authored fresh and
`ail parse <f>.ail > <f>.ail.json` produces canonical JSON. Both paths sit
on the canonical-authoring-form machinery the round-trip invariant protects
(`0009-roundtrip-invariant.md`), so neither can silently drift.
### Mode-annotations section (spec.md §4) — rewrite
`master/examples/param_modes_all.ail.json` currently defines `f_implicit`,
`f_own`, `f_borrow` and its doc-strings explain `ParamMode::Implicit` as the
"legacy default". `Implicit` is gone — **and** a second drift surfaced under
the parse-gate (2026-06-02): the typechecker now rejects `(borrow V)` over an
unboxed value type with `borrow-over-value` (`crates/ailang-check/src/lib.rs`
~813; unboxed = `Int`/`Bool`/`Float`/`Unit`, `linearity.rs:175`). The old
`f_borrow` borrows over `Int` — illegal today. `borrow` is only legal over a
**boxed** type (an ADT).
So the rewrite cannot be "two Int functions". It must show the mode set *and*
the value/boxed rule: a small boxed ADT, an `own` function over an unboxed
`Int`, and a `borrow` function over the ADT. The example is verified to
check green (see plan Task 2). spec.md §4 prose presents the exhaustive rule:
every parameter carries exactly one of `own`/`borrow`; `own` is mandatory for
unboxed value types (`borrow` over them is a `borrow-over-value` error);
`borrow` is available only for boxed types; `ret_mode` is always `own`. No
mention of a defaulted/omitted mode.
### Two new spec.md sections + examples
The mini-spec must show a foreign author the constructs the language now
has. Add two sections and two examples (AST-source `.ail.json`, regenerated
from the `.ail` below via `ail parse`):
**§ Iteration (`loop` / `recur`)** — example `loop_sum`:
```
(module loop_sum
(fn sum_to
(type (fn-type (params (own (con Int))) (ret (own (con Int)))))
(params n)
(body
(loop (acc (con Int) 0) (i (con Int) 1)
(if (app gt i n)
acc
(recur (app + acc i) (app + i 1)))))))
```
**§ Kernel-extension types (`new`)** — example `new_rawbuf`:
```
(module new_rawbuf
(fn main
(type (fn-type (params) (ret (own (con Unit))) (effects IO)))
(params)
(body
(let b (new RawBuf (con Int) 3)
(app print (app RawBuf.size b))))))
```
Both are reductions of shipped, checked `examples/` fixtures
(`loop_sum_to.ail`, `new_rawbuf_size_only.ail`), so they are known to
parse, check, build, and run under today's toolchain.
### spec_completeness.rs — the one design cut
The completeness test asserts every `Term`/`ParamMode` variant is exercised
by at least one master example. Update it:
- Remove the `ParamMode::Implicit` arm.
- Cover `Loop`, `Recur`, `New` via the two new examples.
- **Allowlist `Intrinsic` OUT**, with an in-code comment stating the reason:
`Intrinsic` is non-authorable (absent from the parser term dispatch;
typecheck enforces kernel-tier-only), so a foreign-author mini-spec must
not show it, and no master example can legally contain it. This mirrors
the harness's existing `OPTIMISATION_ONLY` allowlist precedent — a named,
justified exemption rather than a silent gap.
### Mock fixture migration (harness) — corpus-decoupled, but in scope
`harness/tests/fixtures/mock_full_run.json` is a **separate** program
store from the `master/` corpus: it holds hand-authored *simulated model
output* (string-embedded `.ail.json` programs, one per cohort×task×turn)
that the harness pipeline scores by running `ail check` + `ail build` on
each. Every embedded program is legacy-schema (`grep -c param_modes` → 0),
so under today's mandatory `param_modes` they all fail `ail check`. Two
harness tests depend on this fixture and would break:
- `mock_full_run.rs` asserts `json,t3_main_prints,false,2` (t3 in the json
cohort goes green on **turn 2**), `ail,t1_add_three,false,INF` (runs to
the turn limit, never green), exactly 8 score rows, and `RUN_STATUS=ok`.
The `,2` row breaks under schema-death: t3's turn-2 program no longer
reaches green, so `turns_to_green` flips to `INF`.
- `budget_abort.rs` asserts `RUN_STATUS=budget_exceeded` and ≥1
`budget_abort` row. This depends only on the token economy (the fixture's
`usage` fields, ~1200/turn vs. a 1500 budget), **not** on the check
result, so it is robust to the schema drift — but must be re-verified
green, and the `usage` fields must be left untouched.
**Repair:** migrate the embedded `content` programs to the current schema
(`param_modes`/`ret_mode`), **except** the deliberately content-broken ones
(e.g. t3 json turn 1 is `…"defs":[]` — an intentional first-attempt miss
that must stay broken-by-content). Hard requirements pinned by the
assertions:
- t3_main_prints / json / turn 2 → a valid current-schema program that
passes `ail check` + `ail build` + run, so the `,2` row holds.
- t1_add_three / ail → stays failing to the turn limit, so `INF` holds; its
failure should be **content-driven** (the fixture's intended miss), not a
silent schema-drift artefact — otherwise the fixture stops testing what it
documents.
Leave the `usage` token fields byte-for-byte unchanged. This keeps the mock
an honest simulation of a model authoring against *today's* language, rather
than a fixture that passes only because everything uniformly schema-fails.
`harness/tests/fixtures/check_schema_missing_field.stderr` (open question
raised in recon) is a **no-op**: it pins `missing field ``type```, not
`param_modes`, so it is independent of this drift and stays green.
## Components
- `experiments/2026-05-12-cross-model-authoring/master/examples/*.ail.json`
(13 files) — regenerated with `param_modes`/`ret_mode`.
- `.../master/tasks/*.reference.ail.json` (4 files) — regenerated.
- `.../master/examples/param_modes_all.ail.json` — rewritten (own/borrow only).
- `.../master/examples/loop_sum.ail.json`, `.../new_rawbuf.ail.json` — new.
- `.../master/spec.md` — §4 rewritten; two new sections added; their
`{example: …}` markers wired.
- `.../render/src/*` — only if the renderer needs a new section/example
hook; prefer pure data additions.
- `.../render/tests/spec_completeness.rs` — variant coverage + Intrinsic
allowlist.
- `.../rendered/json.md`, `.../rendered/ail.md` — regenerated output,
re-committed for review.
- `.../harness/tests/fixtures/mock_full_run.json` — embedded `content`
programs migrated to current schema (except the deliberately
content-broken ones); `usage` fields untouched.
## Data flow
1. Re-author each corpus `.ail.json` as `.ail`; `ail parse` → new `.ail.json`;
`ail check` green per file.
2. Add the two new examples + two spec.md sections; rewrite §4.
3. `cargo run --manifest-path .../render/Cargo.toml -- --master … --rendered …`
succeeds; inspect regenerated `rendered/`.
4. `cargo test --manifest-path .../render/Cargo.toml` green
(`example_roundtrip`, `spec_completeness`, `token_balance`, splitter).
5. Migrate `mock_full_run.json` embedded programs (per §Mock fixture
migration); leave `usage` untouched.
6. `cargo test --manifest-path .../harness/Cargo.toml` green (mock E2E
`mock_full_run` with the `false,2` + `INF` rows holding,
`verify_references` driving every reference through the real ail+clang
pipeline, strip_locations, budget_abort).
## Error handling / acceptance
- Per-file bar: `ail check <f>.ail.json` exit 0.
- Renderer bar: clean run, no load crash.
- Suite bar: both crates' `cargo test` green. The harness `verify_references`
test is the end-to-end gate — it compiles and runs every reference solution
through the actual toolchain, so a green there means the references are not
just schema-valid but executable.
- No live IONOS call. No `harness` invocation with `--model` other than `mock`.
## Out of scope
- The recurring-method form (project-local skill vs. light triggered tool).
- Any live IONOS run — gated on explicit per-session consent.
- Multi-subject / token-economy re-measurement (the old `runs/` datasets are
left as historical; they are gitignored raw data, not corpus).
- Rewriting specs 0017/0018.