AILang/docs/specs/0017-cross-model-authoring-form-test.md

# Cross-model authoring-form test — Design Spec

**Date:** 2026-05-12
**Status:** Draft — awaiting user spec review
**Authors:** Brummel (orchestrator) + Claude

## Goal

DESIGN.md carries an explicit, currently unratified tension between
two of its load-bearing decisions:

- §"Decision 6: authoring surface" asserts `.ailx` is the AI
  authoring projection.
- §"Feature-acceptance criterion" cites JSON-as-canonical-authoring-
  surface as a behavioural counterexample to human-driven feature
  picks — i.e. "JSON is what the LLM author would naturally produce".

Both cannot be unconditionally true. The compiler accepts both; the
roundtrip-invariant milestone (2026-05-12) closed the bijection.
What is missing is *evidence* on which form a foreign LLM author
actually reaches for and succeeds with when given the choice
between them.

This milestone produces that evidence for **one foreign subject**
(Qwen3-Coder-Next, via IONOS) by running two blind cohorts on the
same tasks: one cohort sees only a `.ail.json` mini-spec, the other
only a `.ailx` mini-spec. The product is:

1. A reproducible experimental harness in
   `experiments/2026-05-12-cross-model-authoring/`.
2. A raw dataset under `runs/<date>-<hash>/` for the live run.
3. An empirical addendum to DESIGN.md §"Decision 6" recording the
   measured data point with explicit single-subject / single-run
   scope.

The verdict question ("ratify or retire form X across N foreign
subjects") is deliberately out of scope for this milestone; it
requires a multi-subject expansion that follows once the harness
exists and runs cleanly.

## Architecture

### Top-level directory

A new top-level directory parallels `crates/`, `bench/`, `examples/`,
`runtime/`, `docs/`:

```
experiments/2026-05-12-cross-model-authoring/
├── README.md          # purpose, how to invoke a run
├── master/            # canonical mini-spec source (form-agnostic)
│   ├── spec.md        # markdown with prose + form-only blocks + example refs
│   ├── examples/      # AST source-of-truth: .ail.json files keyed by id
│   │   ├── empty_module.ail.json
│   │   ├── fn_decl_intro.ail.json
│   │   ├── data_decl_intro.ail.json
│   │   ├── match_intro.ail.json
│   │   └── ... (one per construct introduced in spec.md)
│   └── tasks/         # task definitions consumed by the harness
│       ├── t1_add_three.task.json
│       ├── t2_length.task.json
│       ├── t3_main_prints.task.json
│       └── t4_count_zeros.task.json
├── render/            # Rust binary: master/ → rendered/
│   ├── Cargo.toml     # depends on ailang-core + ailang-surface
│   ├── src/main.rs
│   └── tests/         # roundtrip + token-balance asserts
├── harness/           # Rust binary: orchestrates the live run
│   ├── Cargo.toml
│   ├── src/main.rs
│   └── tests/         # mock-mode E2E
├── rendered/          # OUTPUT of render/; checked-in for review
│   ├── json.md        # mini-spec, JSON-cohort
│   └── ailx.md        # mini-spec, AILX-cohort
└── runs/              # one subdirectory per live run
    └── <YYYY-MM-DD-shorthash>/
        ├── RUN_STATUS         # ok | api_failure | budget_exceeded
        ├── config.json        # model id, token caps, mini-spec hashes
        ├── per_cohort/
        │   └── <cohort>/<task_id>/
        │       ├── turn_<n>_request.json
        │       ├── turn_<n>_response.json
        │       ├── turn_<n>_program.{ail.json,ailx}
        │       ├── turn_<n>_check_stderr.txt
        │       └── turn_<n>_run_stdout.txt
        ├── scores.csv         # one row per (cohort, task)
        └── summary.md         # short human-readable per-cohort aggregate
```

Both `render/` and `harness/` are Cargo crates *outside* the main
workspace. They are not added to the root `Cargo.toml` workspace
members list; they are built locally inside the experiment directory
(`cargo build --manifest-path experiments/.../render/Cargo.toml`).
This keeps the experimental code from polluting the main workspace's
build, test, and rustdoc surfaces.

### The two key fairness anchors

1. **One source of truth for the mini-spec content.** `master/spec.md`
   is the canonical mini-spec. The two rendered versions
   (`rendered/json.md`, `rendered/ailx.md`) are projections produced
   by `render/`. Any change to the spec's content is a single edit;
   drift between the two forms is structurally impossible.

2. **Roundtrip-derived examples.** Every code sample in the mini-spec
   is sourced from an `.ail.json` file in `master/examples/`. The
   renderer prints each example in the cohort's form using the
   existing `ailang_surface::print` for `.ailx`, and the existing
   canonical-JSON serialiser for `.ail.json`. Both forms come from
   the same AST, so the examples are semantically identical by
   construction.

### What this Architecture deliberately does not do

- It does **not** add `render/` or `harness/` to the workspace.
  They are experimental tooling, not part of the AILang toolchain.
- It does **not** invent a new authoring surface, AST node, or
  schema field. The master examples are stock `.ail.json` files;
  the renderer uses only published `ailang-core` / `ailang-surface`
  APIs.
- It does **not** modify `ail` or any of its subcommands. The
  harness shells out to the existing `ail parse` / `ail check` /
  `ail build` binaries verbatim.

## Components

### `master/spec.md` — the mini-spec source

A single markdown file authored by the orchestrator. Structure:

- **Form-agnostic prose** describing language semantics: module,
  fn, data, match, lambda, effects, types, mode annotations,
  cross-module refs, typeclasses (Eq/Ord/Num/Bounded), Float
  semantics, prelude API. Identical text in both rendered versions.

- **Form-only blocks** introduced by the directives
  `{form-only: json}` … `{/form-only}` and `{form-only: ailx}` …
  `{/form-only}`. Only the block matching the rendering form is
  emitted. Used for: JSON schema rules (required fields, ctor
  discriminator format, hash convention), AILX grammar rules
  (tagged head, paren discipline, lexical conventions).

- **Example markers** of the form `{example: <id>}` resolve to
  the AST stored in `master/examples/<id>.ail.json`. The renderer
  prints that AST in the current form and wraps it in a fenced
  code block.

The spec.md must be **vollumfänglich**: every AILang authoring
construct the harness's tasks (or any future task) might exercise
must be introduced. Specifically — modules, imports, `Def::Fn` /
`Def::Data` / `Def::Class` / `Def::Instance`, mandatory mode + type
annotations, effect sets, all term forms (`Var`, `App`, `Lam`,
`Match`, `Lit`, `TermCtor`, `Do`, `Seq`), all pattern forms, type
expressions (`Con`, `Var`, `Fn`, `Forall`, `Constraint`), Float
literals (with bit-hex aware note), the prelude API table
(int_to_str excluded — see roadmap heap-Str ABI dependency), and
typeclasses (class def + instance def + constraint syntax + the
four prelude classes).

Hash field handling: neither authoring form asks the model to
write a hash literal. Per Decision 2 hashes are content-addresses
computed by the toolchain on `ail parse` / `ail check`; they are
not part of the authored surface in either form. The mini-spec
teaches the model that AILang has content-addressed identity
(conceptual, Decision 2) without asking it to emit BLAKE3 itself.
This is symmetric across the two cohorts and removes hash as a
form-distinguishing factor.

### `render/` — the renderer binary

Rust binary, separate Cargo project under the experiment directory.
Dependencies: `ailang-core` (path), `ailang-surface` (path),
`serde_json`. Reads `master/spec.md` + `master/examples/*`, emits
two files: `rendered/json.md` and `rendered/ailx.md`.

Per-form rendering logic:
- For JSON: loads the example as canonical-form JSON (the on-disk
  bytes; `.ail.json` files are already in canonical key order),
  emits as a fenced `\`\`\`json` code block.
- For AILX: loads the example as canonical-form JSON, parses to
  AST via the published `ailang-core` loader, prints via the
  public `ailang-surface` printer that already drives `ail render`,
  emits as a fenced `\`\`\`ailx` code block.

Concrete function names are an implement-iter detail; the contract
is: examples are roundtrippable on both sides because they come
from the same AST, and both forms are produced by the exact
printers that the roundtrip-invariant tests already gate.

Token-balance audit: `render/tests/token_balance.rs` counts tokens
(via a stable approximation — `tiktoken_rs` with the `cl100k_base`
encoding, pinned in the renderer's `Cargo.toml`) of the form-only
blocks in each rendered file. Both totals must be within ±5% of
each other, else the test fails. The check exists so the
orchestrator notices accidentally writing a 3-paragraph .ailx
grammar section against a 1-paragraph JSON schema section.

Roundtrip verification: `render/tests/example_roundtrip.rs` loads
each example as JSON, prints it via the renderer's JSON path, re-
parses; then prints via the renderer's AILX path, re-parses with
`ailang_surface::parse`. Both re-parses must canonicalise to the
same BLAKE3 hash as the original. This is a focused local
counterpart to `ailang-surface/tests/round_trip.rs` and exists so
the examples in the mini-spec are guaranteed authentic AILang in
both forms.

### `master/tasks/<id>.task.json` — task definitions

Each task is a JSON file with this shape:

```json
{
  "id": "t1_add_three",
  "title": "Three-argument addition",
  "description": "Write a module named t1_add_three that exports a top-level fn add_three taking three Int parameters and returning their sum. Also export main : () -> () !IO that prints add_three(1,2,3) then add_three(10,20,30), each on its own line.",
  "expected_stdout": "6\n60\n",
  "reference_solution": "master/tasks/t1_add_three.reference.ail.json"
}
```

The task description is in natural language. The model is told
explicitly what to name the module, what the entry point is
(`main`), and what stdout to produce. The harness uses
`expected_stdout` for the green check; the reference solution is
the orchestrator's sanity-check (verified to pass the same harness
locally before any live run).

The four MVP tasks:
- **T1 `add_three`** — fn declaration, integer parameters, prelude
  arithmetic, `main`/`io_print_int` for output. Floor test.
- **T2 `length`** — defines `data List a where Nil | Cons a (List a)`
  locally (no cross-module imports for MVP), implements
  `length : (forall a. (List a) -> Int)` recursively, exercises ADT
  + pattern match + recursion + polymorphic top-level type signature.
- **T3 `main_prints`** — strictly an effects + module-shape test:
  `main` prints two specific integers using `io_print_int` only.
  Catches model failures at the effect-set + entry-point seam
  independent of ADT or polymorphism load.
- **T4 `count_zeros`** — implements
  `count_zeros : (List Int) -> Int` using the prelude `Eq Int`
  instance to compare against `0`. Forces the model to use the
  typeclass surface (constraint syntax, `eq` method invocation),
  which is the most recent and most spec-heavy AILang feature.
  Combined with recursion on a locally-defined `List`.

### `harness/` — the run orchestrator

Rust binary, separate Cargo project. Dependencies: `reqwest`
(blocking client), `serde_json`, `clap`, `regex`, `tokio` (only
for `reqwest` runtime). No dependency on `ailang-*` crates —
shells out to the system `ail` binary.

CLI:
```
xmodel-harness --rendered <dir> --tasks <dir> --out <runs-dir>
               --model Qwen/Qwen3-Coder-Next
               --max-turns 5
               --token-budget 500000
               [--mock <mock-file>]
```

The harness:
1. Loads the two rendered mini-specs (`rendered/json.md`,
   `rendered/ailx.md`).
2. Loads the four task definitions from `--tasks`.
3. For each (cohort, task) — eight total runs — invokes the
   per-task loop (see Data flow).
4. After every API turn, records request/response JSON, the
   program file the model produced, and the resulting check /
   run output to disk.
5. After all eight runs complete (or are abandoned due to
   budget / API failure), writes `scores.csv` + `summary.md`.

Mock mode (`--mock <file>`): instead of POSTing to IONOS, reads
canned responses from a JSON file keyed by `(cohort, task, turn)`.
Used in CI; no API key required.

The IONOS API token is read from the `IONOS_API_TOKEN` environment
variable. The harness refuses to start if the variable is unset
*and* `--mock` is not provided. This keeps tokens out of the repo.

## Data flow

### Per-task loop

Pseudo-code, runs once per (cohort, task):

```
turn = 1
messages = [
    {role: "system", content: rendered_mini_spec_for_cohort},
    {role: "user",   content: task.description},
]
while turn <= max_turns:
    response = api.post(messages, temperature=0, top_p=1)
    accumulate_token_usage(response.usage)
    program = response.choices[0].message.content

    save_program_to_disk(program, cohort, task, turn)
    save_request_response(messages, response, cohort, task, turn)

    error_msg = run_pipeline(program, cohort, task)
    if error_msg is None:
        record_success(cohort, task, turn, token_usage)
        return
    stripped = strip_locations(error_msg)
    messages.append({role: "assistant", content: program})
    messages.append({role: "user",      content: stripped})
    turn += 1
record_failure(cohort, task, max_turns, token_usage)
```

### `run_pipeline(program, cohort, task)`

Returns `None` on success, otherwise an error string (un-stripped).
The harness pipeline differs slightly per cohort because of the
form difference:

For **JSON cohort**:
1. Write `program` to `prog.ail.json`.
2. `ail check prog.ail.json`. If non-zero exit → return
   `"check: " + stderr`.
3. `ail build prog.ail.json -o prog.bin`. If non-zero exit →
   return `"build: " + stderr`. (Build failures after check
   passes are rare but possible.)
4. Run `./prog.bin` with a 5s timeout, capture stdout. If
   timed out → return `"runtime: timeout after 5s"`.
5. If stdout != task.expected_stdout → return
   `"output: expected " + repr(expected) + ", got " + repr(actual)`.
6. Return None.

For **AILX cohort**:
1. Write `program` to `prog.ailx`.
2. `ail parse prog.ailx -o prog.ail.json`. If non-zero exit →
   return `"parse: " + stderr`.
3. From step 2 onward identical to JSON cohort starting at
   step 2 (the `prog.ail.json` produced by parse is the same kind
   of file as the JSON cohort writes directly).

### `strip_locations(error_msg)`

A regex pass that removes:
- JSON-pointer fragments: `$\.[a-zA-Z0-9._\[\]]+` (e.g.
  `$.defs[0].fn.body.app.fun`).
- Line/column markers: `\bline \d+\b`, `\bcolumn \d+\b`,
  `\bat \d+:\d+\b`.
- File-path prefixes: any leading `^[^:]+:\d+:\d+:` style anchor.

What remains is the form-independent payload: error class +
message ("Type mismatch: expected Int, got Bool", "Unknown
identifier: foo", "Missing required field: params"). The intent
is *symmetric degradation*: both cohorts lose the localisation
information their compiler natively produces, neither gets the
other's form-specific cues.

The full un-stripped `stderr` is still saved to the run directory.
Stripping only affects what the *model* sees on the next turn;
reconstruction is possible post-hoc.

### Scoring

`scores.csv` columns (one row per (cohort, task)):

| column | type | meaning |
|---|---|---|
| `cohort` | `json`/`ailx` | which mini-spec |
| `task_id` | string | t1/t2/t3/t4 |
| `first_attempt_green` | bool | turn 1 reached green |
| `turns_to_green` | int or `INF` | first green turn; `INF` if never |
| `prompt_tokens` | int | sum over all turns |
| `completion_tokens` | int | sum over all turns |
| `error_classes` | comma-list | sorted unique classes seen (e.g. `parse,check,output`) |
| `final_status` | enum | `green` / `turn_limit` / `budget_abort` / `api_failure` |

`summary.md` is the orchestrator-readable aggregate: per-cohort
mean / median over the four tasks for each metric, plus a short
prose paragraph naming the most common error class per cohort.
This is what the eventual DESIGN.md addendum quotes from.

## Error handling

### API-level

- **HTTP 5xx / connection error / timeout (30s read)**: retry with
  exponential backoff (1s, 4s, 16s); after the third failed retry,
  mark the whole run as `api_failure` and stop. Partial data on
  disk is preserved; no in-place rewrites.
- **HTTP 429 (rate limit)**: respect `Retry-After` header if
  present, else back off 30s and retry up to 5 times.
- **HTTP 4xx other than 429**: hard fail; mark run as
  `api_failure`. Most likely cause is auth — token expired or
  malformed.

### Budget-level

The `--token-budget` flag caps total tokens across all eight runs.
Tokens are accumulated as the runs proceed; before each API call,
the harness checks remaining budget. If a single call would
exceed, the harness abandons the *current* run (records `final_status
= budget_abort`) and continues to the next pending task. If the
budget is exhausted entirely, the harness writes
`runs/<.>/RUN_STATUS = budget_exceeded` and stops.

Default cap: 500_000 tokens. A back-of-envelope: 4 tasks × 2
cohorts × 5 turns × ~10k system tokens (mini-spec) + ~2k I/O ≈
480k tokens worst-case for the MVP. The 500k default has a small
margin.

### Pipeline-level (per cohort/task)

Subprocess failures from `ail parse` / `ail check` / `ail build`
are part of the experimental signal; they are not "errors" in the
harness sense — they are feedback to the model. The harness only
treats a pipeline failure as a harness error if the `ail` binary
is not found at all (`ENOENT`), in which case the run is aborted
with a clear "ail binary missing" message and `RUN_STATUS =
api_failure` (mis-labelled but unambiguous — the run did not
complete).

The 5-second per-run timeout on the model's compiled binary
prevents infinite-loop programs from hanging the harness. A
timeout counts as `runtime` error class, which the model receives
as `"runtime: timeout after 5s"` on the next turn.

### Mini-spec authoring errors

If the renderer's `cargo test` fails (token-balance off, an
example fails roundtrip), the harness refuses to start a live
run — `--rendered <dir>` is validated by re-running the renderer's
tests as a pre-flight. This is a guard against "I made a tweak to
spec.md, forgot to re-render, ran the live experiment, paid for
data measured against a stale spec".

## Testing strategy

The experiment's deliverables are themselves measurement artifacts;
"testing" applies to the supporting tooling.

### `render/` tests

- `tests/example_roundtrip.rs` — every example in `master/examples/`
  is loaded, JSON-canonicalised, hashed; printed via .ailx, re-parsed,
  canonicalised, hashed; both hashes must match. This is the local
  counterpart to `ailang-surface/tests/round_trip.rs` for the
  specific subset of examples the mini-spec uses.
- `tests/token_balance.rs` — form-only block token counts within ±5%.
- `tests/spec_completeness.rs` — every AILang AST variant (per
  `ailang_core::ast`) must be referenced by at least one example in
  `master/examples/`. The test borrows the visitor logic introduced
  in `crates/ailang-core/tests/schema_coverage.rs`. This guards
  against shipping a mini-spec that silently omits a construct.

### `harness/` tests

- `tests/mock_full_run.rs` — runs the harness in `--mock` mode
  against a canned response file that simulates one (cohort, task)
  pair reaching green on turn 2 and another running to the turn
  limit. Asserts `scores.csv` is well-formed, `RUN_STATUS = ok`,
  raw artefacts present.
- `tests/strip_locations.rs` — unit-style test of the regex
  pass on a corpus of sample compiler error strings (a handful
  collected from real `ail check` / `ail parse` failures, kept
  in-tree under `harness/tests/fixtures/`).
- `tests/budget_abort.rs` — mock run with a deliberately tiny
  budget; asserts the harness stops cleanly and writes the
  expected status.

### Pre-flight

Before the live run, the orchestrator runs (manually or scripted):
1. `cargo test --manifest-path experiments/.../render/Cargo.toml` — green.
2. `cargo test --manifest-path experiments/.../harness/Cargo.toml` — green.
3. The `render` binary, producing `rendered/json.md` and
   `rendered/ailx.md`.
4. For each task, runs the reference solution through the same
   harness pipeline locally (no API call) and confirms it reaches
   green. This ensures task definitions are themselves valid.

Only then is the live `--mock`-off run launched.

## Acceptance criteria

The milestone is closed when all of the following hold:

1. The directory `experiments/2026-05-12-cross-model-authoring/`
   exists with the four sub-components (`master/`, `render/`,
   `harness/`, `runs/`) and a top-level `README.md` documenting
   how to invoke a run.

2. `master/spec.md` is a complete authoring reference covering
   every AILang construct enumerated in §Components ("vollumfänglich").
   The exhaustiveness gate is `render/tests/spec_completeness.rs`
   which asserts every AST variant is exercised by at least one
   example.

3. `render/` produces `rendered/json.md` and `rendered/ailx.md`
   from `master/spec.md`. Both rendered files are checked in. All
   `render/` tests are green.

4. `harness/` builds and passes all its tests, including the
   mock-mode full run.

5. Four task definitions exist under `master/tasks/`, each with
   a natural-language description, expected stdout, and a
   reference solution. Each reference solution is verified
   locally (no API call) to reach green through the harness's
   own pipeline.

6. A live run has been executed against IONOS (Qwen3-Coder-Next,
   temperature=0, top_p=1, max-turns=5) and recorded under
   `runs/<date>-<hash>/` with: `RUN_STATUS = ok`, full
   per-turn raw artefacts for all eight (cohort × task)
   combinations, `scores.csv`, and `summary.md`.

7. DESIGN.md §"Decision 6" gains an "Empirical addendum
   (2026-05-12)" subsection (~10–15 lines) recording: model id,
   run date, the eight rows of `scores.csv` summarised as
   per-cohort means, and an explicit scope note ("single subject,
   single deterministic run; verdict on Decision 6's universal
   claim deferred to multi-subject expansion").

8. A journal entry under `docs/journals/` links the run
   directory, summarises the data point in 5–10 lines, and
   updates `docs/journals/INDEX.md`.

9. `docs/roadmap.md` P2 entry "Cross-model authoring-form test"
   is removed (with this milestone closure as the one-line
   mirror per roadmap convention). A new P3 idea entry is added
   for "Multi-subject expansion" pointing at this run as the
   baseline.

### Out of scope for this milestone

- A second foreign LLM. Single subject only.
- A free-choice cohort. The two cohorts are mutually blind to
  the other form by design; free-choice requires a third mini-spec
  covering both forms and a different experimental design.
- Repetitions for statistical robustness (temperature > 0,
  median-of-N). MVP runs single deterministic.
- A ratify-or-retire verdict on Decision 6 itself. The data
  point is recorded; the verdict requires the multi-subject
  expansion.
- A Python or shell-script harness; both `render` and `harness`
  are Rust binaries outside the main workspace.
- Form-aware compiler diagnostics. Location stripping is the
  fairness anchor today; building real .ailx-shaped diagnostics
  in `ail check` is a separate, much larger compiler-side effort.

### Iteration decomposition (informational — planner sets the real shape)

- **cma.1**: `master/spec.md` authoring + `master/examples/` +
  `render/` (binary + tests) + checked-in `rendered/*.md`. The
  bulk of the work; the mini-spec is the load-bearing artefact.
- **cma.2**: `harness/` (binary + tests + mock-mode + pre-flight
  hooks) + the four task definitions + reference-solution
  verification.
- **cma.3**: Live run + DESIGN.md addendum + journal entry +
  roadmap edits. Boss-direct, no implement dispatch (a one-shot
  execution, one prose edit, one journal append).

Expansion iterations (`cma.4+`, separate milestone): more tasks
and/or repetitions for statistical robustness, with single-subject
(Qwen) breadth covered before any second-subject question is
reopened. The roadmap entry created at this milestone's close
(P3 "Multi-subject expansion") is the placeholder for that
follow-up.