Revive cross-model authoring-form harness + establish external-model fieldtest method #68

New Issue

2026-06-02T16:11:13+02:00

Brummel commented

2026-06-02 16:11:13 +02:00

Context

experiments/2026-05-12-cross-model-authoring/ is a milestone-complete harness (cma.1-3, ms.1-2) that measures whether a foreign LLM author reaches for .ail.json or .ail and succeeds — render crate (form-agnostic master spec -> two cohorts) + harness crate (IONOS live run, mock mode, scoring). The crates still BUILD against current core/surface, but the corpus is schema-dead: the #55 implicit-cutover (76b21c0) deleted ParamMode::Implicit and made param_modes/ret_mode mandatory, so today's ail check rejects 100% of master/examples/*.ail.json + *.reference.ail.json, render crashes on the first example, and spec_completeness.rs won't compile (refs deleted Implicit; 4 new uncovered Term variants).

Goal

A project-specific recurring method that complements the generic fieldtest skill: a specialised field test that exercises AILang against an external model (Qwen3-Coder / CodeLlama via IONOS) as author, with results routed into the same finding classes as fieldtest. Distinct axis — the generic fieldtester is Claude-as-consumer, but Claude knows AILang; a foreign model that only sees the mini-spec is the honest test of the stated identity goals (hallucination robustness, onboarding-without-context-burn). The fieldtest public-interface-only Iron Law is literal for an external model.

Scope decided 2026-06-02 (corpus-first, method-form-later)

Corpus repair (first iteration). Re-author master examples + reference solutions as Form-A .ail (so param_modes/ret_mode are author-set, not guessed) and re-canonicalise via ail parse; ail check is the oracle for per-param mode. Add examples for the author-facing constructs that landed since May — Loop, Recur, New. Update master/spec.md prose. Rebuild rendered/.
Test harness. spec_completeness.rs: drop the Implicit arm, cover Loop/Recur/New, allowlist Intrinsic OUT with justification — it is non-authorable (not in the parser term dispatch; typecheck enforces kernel-tier-only). Green = render + harness test suites (mock).
Old specs 0017/0018 are NOT rewritten — closed milestones describing the then-state (.ailx, docs/DESIGN.md); the revival gets its own fresh spec under docs/specs/.

Out of scope for the first iteration

Method form (full project-local skill vs. light triggered tool) — deferred until the infrastructure demonstrably builds + a mock run is green.
Live IONOS runs — gated: mock-default; an actual endpoint call needs explicit per-session consent, which /boss does NOT lift.

## Context `experiments/2026-05-12-cross-model-authoring/` is a milestone-complete harness (cma.1-3, ms.1-2) that measures whether a foreign LLM author reaches for `.ail.json` or `.ail` and succeeds — render crate (form-agnostic master spec -> two cohorts) + harness crate (IONOS live run, mock mode, scoring). The crates still BUILD against current core/surface, but the **corpus is schema-dead**: the #55 implicit-cutover (`76b21c0`) deleted `ParamMode::Implicit` and made `param_modes`/`ret_mode` mandatory, so today's `ail check` rejects 100% of `master/examples/*.ail.json` + `*.reference.ail.json`, `render` crashes on the first example, and `spec_completeness.rs` won't compile (refs deleted `Implicit`; 4 new uncovered Term variants). ## Goal A project-specific recurring method that complements the generic `fieldtest` skill: a specialised field test that exercises AILang against an **external** model (Qwen3-Coder / CodeLlama via IONOS) as author, with results routed into the same finding classes as fieldtest. Distinct axis — the generic fieldtester is Claude-as-consumer, but Claude knows AILang; a foreign model that only sees the mini-spec is the honest test of the stated identity goals (hallucination robustness, onboarding-without-context-burn). The fieldtest public-interface-only Iron Law is literal for an external model. ## Scope decided 2026-06-02 (corpus-first, method-form-later) 1. **Corpus repair (first iteration).** Re-author master examples + reference solutions as Form-A `.ail` (so `param_modes`/`ret_mode` are author-set, not guessed) and re-canonicalise via `ail parse`; `ail check` is the oracle for per-param mode. Add examples for the author-facing constructs that landed since May — Loop, Recur, New. Update `master/spec.md` prose. Rebuild `rendered/`. 2. **Test harness.** `spec_completeness.rs`: drop the `Implicit` arm, cover Loop/Recur/New, allowlist `Intrinsic` OUT with justification — it is non-authorable (not in the parser term dispatch; typecheck enforces kernel-tier-only). Green = render + harness test suites (mock). 3. **Old specs 0017/0018 are NOT rewritten** — closed milestones describing the then-state (`.ailx`, `docs/DESIGN.md`); the revival gets its own fresh spec under `docs/specs/`. ## Out of scope for the first iteration - **Method form** (full project-local skill vs. light triggered tool) — deferred until the infrastructure demonstrably builds + a mock run is green. - **Live IONOS runs** — gated: mock-default; an actual endpoint call needs explicit per-session consent, which `/boss` does NOT lift.

Brummel added the feature label 2026-06-02 16:11:13 +02:00

Brummel referenced this issue from a commit

2026-06-02 22:24:24 +02:00

spec: cross-model harness corpus revival (refs #68)

Brummel referenced this issue from a commit

2026-06-02 22:24:24 +02:00

plan: cma-revival.1 cross-model harness corpus revival (refs #68)

Brummel referenced this issue from a commit

2026-06-02 22:24:24 +02:00

feat(cma): revive cross-model harness corpus to current language (refs #68)

Brummel referenced this issue from a commit

2026-06-02 22:24:24 +02:00