main
24 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Brummel
|
fdff6cd613 |
experiment(cma): familiar-vs-unfamiliar — AILang is not harder for a frontier model given a scan (refs #68 #69 #70)
Follows up the SMA controls with the sharper question: does AILang cost a frontier model MORE than a language it knows from training (Rust)? Claude passing SMA is binary and cannot tell "effortless" from "barely". Measured with pass@1 over 6 samples of the same task in AILang (two-example few-shot) vs Rust, single-shot, fresh clueless agents, no tools, oracled afterward. Task 1 (Expr-evaluator — ADT + match + recursion, all in the few-shot): AILang 6/6 = Rust 6/6, no gap; all six AILang answers byte-identical. When every construct is in the few-shot, AILang is not harder. Task 2 (count-greater-than — needs a comparison, OUTSIDE the few-shot): AILang 0/6 vs Rust 6/6. But the logic was correct in all six; they fell on a guessed name (`>`, which AILang lacks). A vocabulary gap, not a reasoning gap — and the single-shot setup is unfair: in Rust Claude implicitly has its training plus the obvious ability to scan an unknown crate, which AILang was denied. The fair test (scan + compiler-loop, how Claude actually works): one Claude context, count-greater-than, given the prelude as a scanned library, then iterated on raw compiler output. Converges in two feedback rounds — gt+match-on-Bool -> (case true ..) -> match (compare h N) .. GT -> green. Decisive contrast: no-scan + terse errors -> Claude DIVERGES (invents gtPos, sub, non-existent pat-var); scan + diagnostic errors -> Claude CONVERGES. Same model; the difference is the discovery affordances, not the model. The 0/6 was the artefact of an unfair test, not an intrinsic AILang weakness — with the scan-plus-iterate workflow, the "language is harder" gap dissolves. Two genuine AILang defects the thread exposed, filed as issues: - #69: `ail builtins` is an incomplete API scan — it lists no comparison operator; gt/lt/le/ge/compare live in the prelude, which builtins does not surface. A model scanning the obvious discovery tool never finds half the comparison stdlib. - #70: `match` on Bool passes `ail check` but codegen rejects it (`match on non-ADT scrutinee (i1); MVP supports only ADTs`); with no `if`, branching on a Bool has no working surface — one must route through compare->Ordering. A check/codegen inconsistency under an "internal:" prefix. Answer, decomposed: cognitively AILang is not harder (the algorithm transfers); vocabulary-without-scan is harder but that is any unfamiliar language; with scan + iteration Claude drives AILang like a foreign crate; where it genuinely is harder is the two fixable tooling/compiler defects above. Caveat: n=6, two small tasks, one model, low sampling variance — exploratory, not a study. Evidence under experiments/2026-05-12-cross-model-authoring/familiar-vs-unfamiliar/. |
||
|
Brummel
|
9c833f65e4 |
experiment(cma): authoring-capability controls — L7 is a model ceiling, not a language ceiling (refs #68)
Every surface run in format-findings.md hit the same L7 (SMA) wall, read so far as a "genuine single-shot complexity ceiling". Three controls on the plain Form-A baseline (identical L7 prompt + ail oracle) pin down what that ceiling actually is. Control 1 — blind frontier model. Ran the full L0–L7 ladder through eight fresh, clueless Claude Opus 4.8 agents, each given verbatim the same prompt the IONOS models got, with no repo access, no compiler, no oracle (all returned tool_uses: 0). Oracled afterward exactly as the harness oracles Qwen/Llama: 8/8 green, including L7. So L7 is not a language ceiling — Form-A is authorable, SMA included, by a strong enough model. The eight modules are kept under clueless-agents/ (one m.ail per dir to satisfy module-name=stem) and re-check green; L7 also runs to the expected output. Claude generalised operators never shown in the few-shot (float +, a / absent from every example, float print). Control 2 — agentic Qwen (qwen_agentic.py). The fairer question: does the compiler-as-a-tool close the gap? Runs the real Claude-Code loop — model proposes a module, harness runs ail, the raw unedited compiler output goes back into the dialogue, up to 8 turns. Result: never solved. From turn 1 Qwen collapses into a repetition loop (~274× "(seq" until max_tokens, completion=1500 every turn); the module is truncated mid-cascade, the oracle returns the correct parse error every turn, and Qwen repeats. Tool access is not an equaliser — it lifts a model only when its reasoning already operates in the right neighbourhood. Also settles the context question: turn 1 collapses at prompt=729 (no accumulated history), so the failure is "can't", not "drowns in context". Raw doc qwen-agentic-sma.md (repetition cascades collapsed to keep the repo small; the collapse marker records the original run length). Control 3 — same SMA in Python (qwen_python_sma.py). Is the AILang failure about coding ability or the unfamiliar surface? Given the same task zero-shot in Python, Qwen returns a clean, idiomatic, correct sliding-window in 113 tokens (verified by inspection + hand-trace; the harness deliberately does not auto-execute LLM code). 113 tokens of correct Python vs a 1500-token collapse in AILang, same model and temperature. Synthesis: the L7 wall is neither an algorithm ceiling nor a "model is weak" ceiling. Qwen owns the algorithm (Python) and a frontier model owns the surface (Claude, blind, first try). Qwen's AILang collapse is specifically the unfamiliar, fully-parenthesised surface it never saw in training, and a compiler-in-the-loop does not rescue it. For an LLM-authored language: making Form-A compiler-driven does not lower the model bar — AILang either targets frontier-class authors, or its surface must drop the structure-tracking load enough for a small model to keep its place. Live IONOS limited to the consented Qwen agentic + Python runs. |
||
|
Brummel
|
ded28e2f75 |
experiment(cma): large-model control on Llama-3.1-405B — the format effect REVERSES (refs #68)
Ran the two core formats (plain Form-A, annotated parens) on Llama-3.1-405B (405B dense, ~135x Qwen3-Coder-Next's 3B active) to separate format-effect from small-model-effect. Scripts parametrized by PROBE_MODEL/PROBE_TAG (Qwen default preserved). Watch-out caught: Llama-3.1 on IONOS leaks its chat template (repeats "assistant", generates endless variants to max_tokens — an IONOS-documented quirk). First run read 0/8; that was the instrument. Fixed with a stop sequence and a balanced-first-module extractor. Result reverses the effect: Qwen3-Coder (3B): plain 6/8, annotated 7/8 Llama-405B (405B): plain 7/8, annotated 6/8 Llama solves L4 (Qwen's bracket wall) with plain parens, so that wall was a small-model effect. Annotated parens help the small model and hurt the large one (at L1 Llama invented a spurious (lit 3) wrapper under the annotated dialect). Both still fail L7 (SMA) — a genuine complexity ceiling. Synthesis: redundant explicit structure is a crutch for weak models. A frontier model tracks plain brackets fine and the unfamiliar scaffolding only adds noise. The right surface is model-relative. Synthesis in format-findings.md; raw ladders llama405-*.md. |
||
|
Brummel
|
ec6efe2cf2 |
docs(cma): add Qwen3-Coder-Next model context to format findings (refs #68)
Researched the model under test. Qwen3-Coder-Next is a 3B-activated MoE (80B total, Feb 2026) trained for AGENTIC coding (codebase + tools + RL on environment feedback), not one-shot generation of unfamiliar syntax — exactly our task, so we tested its weakest discipline. Its tool-calling strength is agentic environment interaction, not emitting deep structured AST JSON, consistent with the tool probe being worst. The format direction is likely general but the magnitude is model-specific; bigger active models on the same IONOS endpoint (Llama-3.1-405B, Llama-3.3-70B) were never tried and are the obvious next step to separate format-effect from small-model-effect. |
||
|
Brummel
|
8391209618 |
experiment(cma): tool-calling re-run at max_tokens=8000 — higher budget does not help, it degenerates (refs #68)
User noted the context window is ~128k and the bad-json failures looked like
truncation, so the max_tokens=2500 cap may have skewed the tool-calling test.
Fair point: the L4/L7 bad-json cases WERE truncation. Verified max_tokens=8000
is accepted, re-ran.
Result: 1/8, slightly worse, not better. The bad-json cases became check
failures (full AST now produced, but semantically wrong), confirming
truncation was a real but secondary factor. And the extra room exposed the
deeper problem: at L2 ("print 7", trivial) Qwen emitted 8000 tokens / ~20KB of
AST JSON and ran off the end — a repetition loop on a one-line program; at L7
it nested JSON deep enough to hit the parser's recursion limit. The verbose
AST-JSON surface is not budget-limited in a recoverable way — more room just
lets it loop or over-nest. Tool-calling stays at the bottom; the higher budget
makes it degenerate, even on trivial tasks. Updated format-findings.md.
|
||
|
Brummel
|
2b9a1083b5 |
experiment(cma): tool-calling / structured-AST tested — worst, and it confirms the rule (refs #68)
User asked whether Qwen's tool-calling could help. Verified tool-calling is
supported on IONOS (a dummy add tool returns a correct tool_call). Then tested
the promising version: submit the program as canonical AST JSON via a forced
submit_program tool — no surface syntax, no converter risk (the tool argument
IS the .ail.json, written and checked directly).
Result: 2/8, tied for worst. Two failure modes, both the model's: valid JSON
but wrong AST (arity etc.), and — at L4/L7 — bad-json: the tool arguments were
not valid JSON at all, because completion hit the 2500-token cap. AST JSON is
so verbose (every leaf is {"t":"var","name":"x"}) that a non-trivial program
overruns the budget and truncates. IONOS does not constrain decoding to the
schema, so the full structure burden stays on the model, and it is the heaviest
of all surfaces (braces+brackets+keys+quotes+commas+"t": per node).
This confirms the rule across six surfaces (annotated-parens 7/8 > plain parens
6/8 > M-expr 4-5/8 > indent 3/8 > YAML 2/8 = tool-AST 2/8): the discriminator is
redundant cue vs extra burden, not explicit vs implicit. Even tool-calling, the
model's home turf, loses because AST JSON maximises bookkeeping with no
redundancy. Lever for an LLM-authored language: redundant self-checking
structure (depth-annotated brackets). Synthesis in format-findings.md.
|
||
|
Brummel
|
7fe08139fb |
experiment(cma): M-expressions tested — middle of the pack; sharpens the finding (refs #68)
Tested McCarthy's head[a; b] notation (user request). Explicit structure (brackets + semicolons) in the function-call shape LLMs know cold. Result 4-5/8 (model variance over two runs): below plain parens, above bracket-free. Two watch-out catches kept it honest: (1) a trailing ';' before ']' — a normal trailing separator Qwen wrote — was rejected by my too-strict parser; fixed to tolerate it, round-trip preserved (without this M-expr looked unfairly bad). (2) L4 then failed with 43 '[' vs 42 ']', a genuine Qwen bracket imbalance (not truncation), the same depth-tracking failure as plain parens. Why worse than plain parens despite being explicit: swapping () for [] does nothing for the balance burden, and the semicolons ADD a second consistency requirement Qwen does not reliably meet. This sharpens the overall finding across five formats (annotated-parens 7/8 > plain parens 6/8 > M-expr 4-5/8 > indent 3/8 > YAML 2/8): the discriminator is not "explicit vs implicit" but REDUNDANT CUE vs EXTRA BURDEN. Depth-annotated parens carry nesting redundantly so the model can't lose its place (best); indentation moves tracking to an implicit channel (worst); M-expr semicolons add bookkeeping on top of bracket balance (middle). Lever for an LLM-authored language: redundant self-checking structure, not a lighter or more familiar surface that carries extra bookkeeping. Synthesis in format-findings.md. |
||
|
Brummel
|
525112e32e |
experiment(cma): bracket-free formats tested — they make Qwen WORSE, not better (refs #68)
Tested the user's bracket-free ideas (indent, YAML) against the annotated-paren result. Hypothesis was that removing the closing token removes the repetition tail. The opposite held. Instrument note (the watch-out paid off twice): a real YAML library auto-quotes and gives *,-,true special meaning, which collides with the AILang operators — Qwen wrote `- *` (multiply), a YAML alias marker, and the library crashed. That is a measurement bug, not a model failure. Rebuilt with a hand-written literal indent parser (atoms verbatim, no quoting); `- *` then round-trips fine. With the instrument fixed, the result is monotone across four formats: format 4 annotated parens 7/8 > plain Form-A parens 6/8 > yamlish indent 3/8 > YAML library 2/8. More explicit structure marking helps this model; less hurts. Qwen's weakness is structure-tracking at depth, and indentation makes it track the nesting itself — as error-prone as matching parens but without the redundant cue. The bracket-free surfaces fail EARLIER (L3/L4), before the L7 repetition ceiling is even reached. Lever for an LLM-authored language: redundant explicit structure (depth- annotated brackets), not a lighter surface. Synthesis in format-findings.md; raw ladders qwen-yaml.md (PyYAML, discarded) and qwen-yamlish.md (own parser). |
||
|
Brummel
|
682b935ba3 |
experiment(cma): annotated-paren format helps Qwen — cracks the bracket wall (refs #68)
Tests the user's hypothesis: if no human reads the surface, lift the paren- counting burden. Format 4 = annotated parens, every paren carries its nesting depth. Discipline: the format has no ail parser, so the converter is round-trip verified (parse-identity on known-good demos) before judging any model output. Result: it helps. Plain Form-A 6/8 -> format 4 7/8 on the same ablation ladder. The depth annotation cracks the bracket-balance wall that broke L4. At the hardest level (L7 SMA) Qwen produced perfectly balanced brackets through the deepest nesting AND correct SMA logic; the only remaining issue was a construct quirk (it treats seq as variadic, but AILang seq is binary). Proof the blocks were right: Qwen's exact logic with a manual binary-seq fix runs and emits the exact expected SMA output. Pushing further (format 4 + explicit seq-binary hint) fixed the seq nesting but L7 still failed on two MODEL-behaviour limits, not encoding: it simplified the logic, and fell into a closing-token repetition tail (hundreds of #0) ). The tail is paren-driven, so the natural next test is a bracket-free format (indent / keyword-delimited) with no closing token to loop on — a deliberate next step needing its own verified converter. Synthesis in format-findings.md. |
||
|
Brummel
|
da99ebaba8 |
experiment(cma): ablation finds WHY Qwen fails at Form-A — two distinct walls (refs #68)
Try-and-error ablation, minimal few-shot context, single shot per level, a ladder L0 (fn returns 42) to L7 (full SMA). Two ladders: base few-shot that only destructures, then one that also constructs (term-ctor). Finding: Qwen is NOT generally incapable. With minimal correct context it writes 6/8 tasks green, including ADTs, match, recursive fns, and Series use. The failures are two specific walls: - Wall 1 (cheap): unfamiliar constructs. It built lists with (app Cons ...) — calling the ctor like a function — because it had only seen match, never term-ctor. Adding ONE term-ctor example flipped L3 and L5 to green and removed the Cons error. The model generalises a construct from one example. - Wall 2 (hard): paren/nesting discipline at depth. The two that stay red (L4, L7) fail purely on bracketing, not knowledge. In L4 the recursive function is flawless; main closes a 4x-nested term-ctor value one paren early so the let body slides inside it — total paren count is BALANCED (43/43), the parens are just misplaced. Green/red tracks nesting depth, not feature. Examples do not fix it. This is the same weakness the full-SMA probes hit (run 2 had correct logic, died on brackets; run 3's long context tipped it into a seq-repetition loop). Takeaway: the bottleneck is the fully-parenthesised surface at depth, not the semantics. Natural next test: the .ail.json authoring form, which removes human paren-counting. Driver qwen_ablation.py; ladders in qwen-ablation-min.md and qwen-ablation-ctor.md; synthesis in qwen-ablation.md. |
||
|
Brummel
|
8b7e683cc9 |
experiment(cma): SMA probe run 3 — full grammar+spec+lib makes Qwen degenerate (refs #68)
Third run of the SMA authoring probe, this time with Qwen fully equipped: a distilled complete Form-A construct grammar (with explicit arity/paren rules), the mini-spec, and the verbatim Series library source. Hypothesis: the formal grammar clears the bracket failures of runs 1-2. It did the opposite. With ~80k prompt tokens the model degenerated into an identical pathological (seq (seq (seq ...))) tower every turn (321 open parens vs 10 close, byte-identical across all 6 turns, indentation exploding). Doubling max_tokens to confirm it was not mere truncation produced HTTP 413 (request too large) — more budget yields an even larger degenerate output, confirming degeneration. Verdict (n=1 model): Qwen3-Coder-Next wrote no working SMA in any of three configurations; more reference material made it WORSE, not better (the lean run 2 got the algorithm right, the fully-equipped run 3 looped); it never converged from ail-check feedback in any run; all failures are Form-A bracket-shape errors, never semantics. Driver + raw logs updated. |
||
|
Brummel
|
577bb6be43 |
experiment(cma): SMA authoring probe against Qwen3-Coder-Next (refs #68)
Probe whether a foreign model can write the Series SMA worked example in
AILang Form-A from the mini-spec + Series API alone. Two live runs, neither
reached green, but a clean progression:
- Run 1 (stock API doc): 6 byte-identical turns, blocked on (app new ...) —
Qwen treats the term head as an ordinary function.
- Run 2 (new clarified as a term head): clears that; ownership threading
comes out idiomatically right (nested let s = Series.push s v), but
S-expression bracketing breaks (push sees 4 args, not 2).
Cross-cutting: Qwen clears a hurdle only with per-quirk guidance and never
converges from the ail check diagnostic (same failure repeated all 6 turns
in both runs); the failures are Form-A bracket-shape errors, not semantic.
n=1 model, single session. Driver: sma_probe.py; raw logs sma-probe-run{1,2}.md.
Live IONOS run, with consent.
|
||
|
Brummel
|
29625e7262 |
feat(cma): revive cross-model harness corpus to current language (refs #68)
The cma authoring-form harness corpus had gone dead against the language as it evolved since May. The plan modelled it as merely schema-dead (missing param_modes/ret_mode); it was also drift-dead in the example BODIES. Fixed in place, with `ail check` + both test suites as the oracle: - Schema: param_modes/ret_mode completed on every fn type; existing borrow annotations preserved (data_with_match's borrow over List). - Symbol drift: `<`/`==` -> `lt`/`eq` (operator-routing); the removed `io/print_int` op -> print_str(int_to_str n) followed by a newline print, preserving the trailing newline the references' expected_stdout needs. - Ownership/ADT restructures: data_simple's reuse-as now wraps the source in a match arm (ctor must be statically visible); data_with_match's count_via_letrec + local go switched borrow->own (consume-while-borrowed under the tightened ownership analysis; head_or_zero still exercises borrow over the boxed List). - param_modes_all rewritten to own (Int) + borrow over a boxed ADT -- (borrow Int) is now a borrow-over-value error. - Two new author-facing examples (loop_sum: Loop/Recur; new_rawbuf: New) cover the Term variants that landed since May. - spec_completeness.rs: drop the deleted ParamMode::Implicit; cover Loop/Recur/New; allowlist the non-authorable Term::Intrinsic out. - spec.md section 4 rewritten to own/borrow (mandatory, no implicit) with the borrow-over-value rule; rendered/ regenerated. - mock_full_run fixture's t3 turn-2 program migrated so the harness score assertions hold; usage fields untouched. Both render/ and harness/ cargo test suites green in mock mode; no live IONOS call. Run the harness budget/reference tests with AIL_BIN pointing at target/debug/ail (ail is not on PATH in the test env). |
||
|
Brummel
|
832375f2ac |
convention: counter-prefix file naming across docs/specs/, docs/plans/, design/contracts/, design/models/
All 176 files in the four accumulating directories now use a zero-padded 4-digit counter prefix that reflects creation order (`NNNN-slug.md`). The counter is assigned per directory in strict git-log creation order; ties broken alphabetically by original name. The old `YYYY-MM-DD-` prefix on docs/specs/ and docs/plans/ files is dropped — the date is recoverable from git log and the counter carries the ordering. A file's counter is stable for the life of the file: never reassigned, never reused, never compacted. Deleted files retire their counter; subsequent files do not fill the gap. This is the property that lets cross-references stay literal — refs use the full filename including the counter (`design/contracts/0007-honesty-rule.md`) so they grep cleanly and resolve directly without a glob step. 313 cross-references updated across .md/.rs/.toml/.c/.json files (test pins, include_str! paths, design-INDEX entries, baseline notes, runtime C comments, inter-contract markdown links incl. bare basename and `../models/foo.md` forms). CLAUDE.md gets a new "File-naming convention" section spelling out the rule and rationale. skills/brainstorm/SKILL.md and skills/planner/SKILL.md updated so new spec/plan creation produces counter-prefixed names from the start. The full test suite (cargo test --workspace) passes. |
||
|
Brummel
|
4fc65ccb99 |
iter schema-camelcase-fix.1 (DONE 4/4): paramTypes/retType → param-types/ret-type — closes #30
`Term::Lam`'s two camelCase JSON tags become kebab-case, matching
the convention every other compound-key tag in the AST schema
already follows. After this iter the canonical JSON has zero
camelCase outliers.
## Schema swap (atomic, Task 2)
- `crates/ailang-core/src/ast.rs:492,494` — two
`#[serde(rename)]` strings: `"paramTypes" → "param-types"`,
`"retType" → "ret-type"`. Rust field names unchanged.
- `crates/ailang-core/src/workspace.rs:1925-1926` — in-source
JSON literal in the `ct1_validator_walks_lam_embedded_types`
test.
- `design/contracts/data-model.md:142-147` — fenced JSON block
in the canonical data-model contract. Honesty-Rule touch-point:
the contract document must describe the actual present-state
schema.
- `examples/test_loop_binder_captured_by_lambda.ail.json` and
`experiments/2026-05-12-cross-model-authoring/master/examples/fn_with_lambda.ail.json`
— the two `.ail.json` fixtures whose canonical-JSON embeds
the renamed tags.
The five files moved together within one task because `Term::Lam`
has no `#[serde(default)]` on `param_tys` / `ret_ty` — a
renamed-away key is a hard deserialise error. Splitting the swap
would have left the workspace untestable mid-step.
## RED → GREEN pin (Tasks 1, 2)
`crates/ailang-core/tests/design_schema_drift.rs` gains
`lam_serialises_with_kebab_keys`: pins that `serde_json::to_value(&Term::Lam{...})`
emits `"param-types"` / `"ret-type"` AND does not emit the old
camelCase keys. Companion: the existing
`design_md_anchors_every_term_variant` test now also asserts the
kebab anchors appear inside `data-model.md`'s `lam` fenced block.
Both confirmed RED on entry, GREEN after Task 2.
## Rustdoc honesty pass (Task 3)
Three pure-prose edits keep production rustdoc consistent with
the new schema vocabulary: `crates/ailang-core/src/ast.rs:8` (the
module-level rename enumeration), `crates/ailang-surface/src/parse.rs:81`
(prose mention of `lam`'s carry), `crates/ailang-check/src/lib.rs:1707`
(InstanceMethod routing helper rustdoc). No compile or test
consequence — Honesty-Rule maintenance.
## Plan-pseudo-vs-reality finding: under-audited hash blast radius
The brainstorm spec audited `crates/ailang-core/tests/hash_pin.rs`
exhaustively (5 pinned modules, zero `(lam ...)` occurrences,
"no hash refresh required"). It missed
`crates/ailang-surface/tests/prelude_module_hash_pin.rs` — a
separate hash-pin file in a different crate that pins `prelude.ail`,
which contains 11 `(lam ...)` forms. The prelude module hash
drifted (`6d0577ff0d4e50ac` → `562a03fc57e7e017`); the implementer
refreshed it with a Honesty-Rule provenance comment matching the
precedent set in commit `26fb345`. Form-A is unchanged — only the
canonical-JSON byte stream differs, which is exactly what the
rename targets. The spec-side learning is: schema-rename
brainstorms must walk *every* test crate for hash-pin files, not
just the home crate of the AST.
## Why this shape, and not the alternatives
- *`params-ty` / `ret-ty` (Rust-field-name vibe)* — rejected. The
AST JSON schema follows its own kebab/single-word convention,
not Rust field names. The other multi-word tag (`reuse-as`) is
kebab; abbreviating `params-ty` mixes singular+plural and has
no precedent.
- *Inline typed params as a sub-tag* (e.g. `params: [{name, type}, ...]`)
— rejected. Stronger semantic locality, but it changes schema
topology rather than tag spelling. Out of scope for a camelCase
correction; would have re-pinned far more than this milestone.
- *Bundle with #27 (arith-rename)* — rejected. #27 carries four
open brainstorm questions (mod vs rem, neg vs sub 0, Num class,
deprecation window) and is structurally a separate milestone.
Each gets one re-pin wave with its own rationale; no churn
saving from bundling.
## Honest call on the feature-acceptance gate
Clause 1 (LLM author naturally uses the new form) is the weakest
of the three. Direct empirical evidence is absent — the 2026-05-21
naming-A/B run measured a different axis. The argument is
indirect: a future LLM author generalises from the rest of the
schema ("compound keys are kebab"), and the two camelCase
outliers are precisely the sites where that generalisation
diverged from reality. Clause 2 (redundancy reduction) and
Clause 3 (no semantic surface touched) are direct.
## Forbidden touches verified untouched
`docs/plans/*.md` historic plans (which embed old tag names in
example JSON), `experiments/2026-05-12-cross-model-authoring/master/spec.md`,
`experiments/.../rendered/*.md`, and `experiments/.../runs/**` —
all left as frozen historical artefacts per Honesty-Rule
analogue (describe state at time of writing, not present state).
Verified via `git diff --name-only HEAD` (Task 4 Step 4).
## Verification
- `cargo test --workspace`: all test groups OK, 0 failed,
2 ignored (pre-existing).
- `cargo test -p ailang-surface --test round_trip`: GREEN
(Form-A unchanged invariant).
- New schema-shape pin and the extended data-model anchor walk:
GREEN after Task 2.
- Negative grep across `crates/`, `examples/`, `design/`,
`runtime/`: the only remaining `paramTypes` / `retType`
occurrences in checked-in code are the load-bearing
negative-assertion strings inside the new pin (intentional —
they are what makes the pin RED-able on regression).
- Stats: `bench/orchestrator-stats/2026-05-21-iter-schema-camelcase-fix.json`
— 4/4 tasks DONE, no re-loops, no review-loops.
closes #30
|
||
|
Brummel
|
b85d498d03 |
doc: fix Form-A ctor drift in cross-model-authoring spec — closes #28
The master spec at experiments/2026-05-12-cross-model-authoring/ master/spec.md taught two wrong Form-A keywords: - Term position: `(ctor TYPE-NAME CTOR-NAME ARG*)` — parser rejects this; the canonical keyword is `term-ctor`. The bare `(ctor ...)` is reserved for the inside of `(data ...)` definitions only. - Pattern position: `(ctor CTOR-NAME SUBPAT*)` — the canonical pattern keyword is `pat-ctor`. The JSON section was already correct (canonical schema tag IS "t": "ctor", per ast.rs:471). Only the AIL section had drifted. Empirically caught by the Qwen3-Coder naming-A/B run on 2026-05-21 (experiments/2026-05-21-naming-ab/runs/r1/), where every t4_count_zeros output produced `(ctor List Nil)` in term position and failed `ail check` 100 % across all three cohorts — a constant tax independent of the naming variable under test. Re-rendered rendered/ail.md from the patched master via xmodel-render. rendered/json.md unaffected. closes #28 |
||
|
Brummel
|
5bd148a607 |
fieldtest: naming A/B against Qwen3-Coder — refactor not justified
Three-cohort harness against IONOS-hosted Qwen3-Coder-Next, testing
whether application-head tag (`app` vs `apply` vs `call`) affects
LLM authoring success. Cohorts share lambda/constructor/typecon tag
renamings (the non-discriminator changes); only the head differs.
Result: 100 % cohort-treue — Qwen writes whatever the spec teaches,
no measurable natural preference. Pipeline pass-rate (classic 1/8,
apply 0/8, call 1/8) is cohort-independent. The four task failures
are general Form-A friction, not naming-related. Refactor not
justified per the feature-acceptance gate.
Tracked harness:
- 2026-05-12-cross-model-authoring/rename-spec.py — regex tag
rewriter for the two non-classic spec variants
- 2026-05-21-naming-ab/run.py, reprocess.py — three-cohort runner
+ post-processing with disjoint-discriminator counting and
module-name-matched temp dirs
Generated artefacts (runs/, rendered/ail-{renamed,call}.md) are
gitignored — regeneratable from rename-spec.py + run.py.
Side-effects filed against current spec/schema bugs surfaced during
the run:
- refs #28 spec teaches (ctor X) for term position; parser requires
(term-ctor X) — 100 % t4 failure across cohorts
- refs #29 io/print_str appends newline (de facto println);
spec now documents the behavior, code question still open
- refs #30 schema camelCase outlier: paramTypes/retType in Term::Lam
Spec patch in rendered/ail.md:
- replaces stale io/print_int references with io/print_str (bitrot
from the print-builtin consolidation)
- documents io/print_str's trailing-newline behavior
- corrects the prelude claim about int_to_str (IS in prelude)
|
||
|
Brummel
|
72e54f4fd3 |
iter ext-rename: .ailx → .ail across the live toolchain
The surface-form file extension changes from .ailx to .ail. AILang's
authoring surface now uses the same .ail stem as its canonical JSON
form (.ail.json), giving the language a single coherent extension
family: .ail is the LLM-authored Form A, .ail.json is the canonical
JSON-AST Form B.
Scope (touched):
- 61 example renames examples/**/*.ailx → .ail (git mv)
- 1 rename experiments/.../rendered/ailx.md → ail.md
- 35 content-edited live-toolchain files (crates/, docs/DESIGN.md,
docs/roadmap.md, docs/PROSE_ROUNDTRIP.md, skills/, bench/reference/*.c,
experiment crates under experiments/.../{render,harness,master})
- Experiment-crate cohort rename Cohort::Ailx → Cohort::Ail,
Form::Ailx → Form::Ail, per_cohort/ailx → per_cohort/ail,
{form-only: ailx} → {form-only: ail}, ```ailx → ```ail
Out of scope (deliberately untouched, to preserve honest history):
- docs/journal-archive.md (content-frozen per CLAUDE.md)
- docs/journals/, docs/specs/, docs/plans/, bench/orchestrator-stats/
- experiments/.../runs/ (frozen LLM-output artefacts; models actually
saw .ailx — renaming would falsify the experimental record)
Verification: cargo build/test --workspace green; experiment crate
cargo test green; bench/check.py + compile_check.py + cross_lang.py
all 0-regressed; negative grep for ailx|Ailx|AILX outside the
out-of-scope paths returns zero matches.
Opens immediate follow-up: roadmap.md P2 todo `ail check`/build/run
accept .ail extension — after this rename, .ail is canonical
authoring surface but the CLI still produces a misleading JSON-parse
error on `ail check foo.ail`. That's the next iter.
|
||
|
Brummel
|
65a4f0aa16 |
iter ms.2: Qwen retroactive re-run + first CodeLlama-13b-Instruct run + DESIGN.md §Decision-6 addendum extended to two subjects
Two live IONOS harness invocations with the ms.1 fix in place:
- Qwen/Qwen3-Coder-Next → runs/2026-05-12-080864/ (RUN_STATUS=ok)
- meta-llama/CodeLlama-13b-Instruct-hf
→ runs/2026-05-12-9197fd/ (RUN_STATUS=ok;
2/8 JSON cells terminated as api_failure
on turn 1 zero-token — transient IONOS
terminal error, disclosed in addendum)
Per-cell results (reached/4, prompt tokens, completion tokens):
Qwen JSON 1/4 182,378 14,972
Qwen AILX 1/4 110,575 3,474
CodeLlama JSON 0/4 116,015 2,711 [2 cells api_failure]
CodeLlama AILX 2/4 93,017 2,234
Both subjects agree on direction: AILX cohort cheaper on prompt
tokens (Qwen 61%, CodeLlama 80% of JSON-cohort spend) and on
completion tokens (Qwen 23%, CodeLlama 82%); AILX reached-green
≥ JSON reached-green for each subject. Three of four
first-attempt-green cells across the two subjects are AILX.
Failure classes split symmetric to form: JSON cohorts fail at
typecheck, AILX cohorts at parse — each form's
front-of-pipeline check.
DESIGN.md §"Decision 6 / Empirical addendum (2026-05-12)"
replaced: 2-column single-subject view → 4-column two-subject
view, framing paragraph reports direction agreement and the
api_failure / Qwen-rerun-noise anomalies, scope paragraph
updated to 'two subjects, n=1 each, deterministic; not a
verdict — a universal claim would need ≥3 subjects with
statistical robustness'.
Roadmap P3 'Multi-subject expansion' entry removed; scoped
goal met for two subjects.
Milestone token spend: prompt 501,985 + completion 23,391
= 525,376.
|
||
|
Brummel
|
614fd8bc93 |
iter ms.1: pipeline.rs format!("check: {e}") → {e:#} preserves anyhow Caused-by chain in JSON-cohort feedback
Inline pinning test in pipeline.rs constructs the same chain
shape module_name_from_json produces on serde-parse failure
(leaf wrapped with "parsing JSON in <path>" context) and
asserts both layers survive the formatter. RED→GREEN against
the property, not against production line bytes — for a
two-character fix, extracting a helper just for testability
would be over-engineering.
The strip_locations regex collapses \nCaused by: chains; the
{:#} alternate Display joins with ': ' on one line, so the
collapser is a no-op on the fixed output. AILX cohort
unaffected — only the harness-side anyhow path (module
rename pre-check) was dropping cause information.
Harness suite: 14/14 green (was 13/13 + 1 new pin).
|
||
|
Brummel
|
94d6963995 |
audit-cma: close milestone Cross-model authoring-form test
Architect drift_found, both [low]:
- INDEX backfill for commit
|
||
|
Brummel
|
e91e31aadb |
iter cma.3: live Qwen3-Coder-Next run + DESIGN.md §Decision-6 empirical addendum + milestone close
Milestone "Cross-model authoring-form test" closed. Live IONOS run against Qwen/Qwen3-Coder-Next executed at temperature=0, max-turns=5, token-budget=500000. Raw dataset under runs/2026-05-12-df7531/: RUN_STATUS=ok, scores.csv with 8 rows (4 tasks × 2 cohorts), summary.md, plus 156 per-turn raw artefacts (request/response/program/ stderr/stdout) preserved for any future analysis. Headline numbers: - AILX cohort: 2/4 green, 1/4 first-attempt, ~95k prompt + ~2.6k completion - JSON cohort: 1/4 green, 0/4 first-attempt, ~192k prompt + ~15k completion - Only first-attempt success was AILX on t3_main_prints (~5 lines of tagged s-expr vs ~23 lines of structured JSON for the JSON cohort's turn-1 attempt that needed correction). - Per spec, single subject + single deterministic run = data point, not verdict. DESIGN.md §"Decision 6: authoring surface" gains an "Empirical addendum (2026-05-12)" subsection: 6-row metric table, illustrative t3 contrast, explicit single-subject scope note pointing at the roadmap follow-up entry for multi-subject expansion. Roadmap: P2 entry "Cross-model authoring-form test" removed (this journal is the convention's one-line mirror); P3 entry "Multi-subject expansion (cross-model authoring-form follow-up)" added with the run dir named as baseline. Milestone artefacts span three iters (cma.1 master mini-spec + renderer; cma.2 harness + tasks + reference solutions; cma.3 this live run + DESIGN.md addendum + close). |
||
|
Brummel
|
fe1fb6b4f0 |
iter cma.2: harness binary + 4 tasks + reference solutions + 4 integration tests
Sibling standalone Cargo crate `harness/` under experiments/2026-05-12-cross-model-authoring/ (out-of-workspace idiom carried forward from cma.1 verbatim). Six modules: strip_locations (regex pass for form-asymmetric location info, calibrated against five real `ail check`/`ail parse` stderr captures), pipeline (subprocess wrapper for parse|check|build + 5s-timeout exec; preflight on ail+clang), ionos (blocking reqwest client + retry policy per spec), mock (canned-response loader), scoring (CSV + summary.md), tasks (definition struct + loader). main.rs ties them into the per-(cohort,task) loop with budget accounting and per-turn artefact recording. Four MVP tasks land with reference solutions that compile, build, and execute green through the actual ail+clang pipeline: t1_add_three (chained `+` + io/print_int), t2_length (polymorphic List + recursion), t3_main_prints (minimal IO module), t4_count_zeros (prelude Eq Int + branched if). Reference solutions stay in canonical AILang form — param_modes is omitted when every parameter is the Implicit default, consistent with the existing examples/ corpus. 13/13 tests green: 5 lib unit (strip_locations) + 5 integration (strip_locations against verbatim captured fixtures) + 1 verify_references (drives every reference through the real ail+clang pipeline) + 1 mock_full_run (full eight-row sweep with mixed green-on-turn-2 + turn-limit cycles) + 1 budget_abort (synthetic budget exhaustion with budget_abort rows + run_status). Two implementer-phase repairs beyond the plan, both small and surfaced in the iter journal Concerns: 1. pipeline.rs renames the program file to `<module-name>.ail.json` between parse and check because `ail check` enforces filename stem == module name (compiler contract the plan did not anticipate). 2. main.rs fills synthetic budget_abort rows for tasks the outer loop never reached so scores.csv preserves the expected eight-row shape on budget exhaustion. One plan/text mismatch carried over: README "Total 8 passed" reflects the plan's four-suite count; actual sweep produces 13. Doc-fix candidate for cma.3. cma.3 (live IONOS run + DESIGN.md addendum + roadmap edits) remains out of scope. |
||
|
Brummel
|
a8c29d130b |
iter cma.1: master mini-spec + render binary for cross-model authoring experiment
New top-level experiments/2026-05-12-cross-model-authoring/ (out of
root Cargo workspace; nested-crate standalone via empty [workspace]
table per Cargo idiom). The renderer projects one canonical
master/spec.md into two form-specific files (rendered/json.md and
rendered/ailx.md) by walking three directive forms ({form-only:
json}, {form-only: ailx}, {example: <id>}) and emitting per-example
fenced blocks sourced from 13 curated .ail.json fixtures.
Four test gates green (10/10 total):
- splitter_unit (7) — directive parser, malformed inputs panic
- spec_completeness (1) — AST-variant visitor lifted verbatim from
schema_coverage.rs; 34/34 variants covered by the curated subset
- example_roundtrip (1) — every fixture roundtrips through
ailang_surface::{print, parse} to identical canonical bytes
- token_balance (1) — form-only block totals balanced to 3.22% under
tiktoken-rs::cl100k_base (gate is ±5%)
Six of the 34 variants did not have a dedicated fixture topic in the
plan and were absorbed into the closest existing fixture per the
plan's Step 4.14 escape hatch (Let/Clone → fn_calls_prelude; LetRec
→ data_with_match; If → match_literal_pattern; ReuseAs →
data_simple; Const → floats). Surfaced as a journal note, not a
plan revision.
Two ancillary additions vs. plan, both structural Cargo necessities,
spec-compatible: render/Cargo.toml carries an empty [workspace] table
so the nested crate refuses parent-workspace discovery, and
render/.gitignore drops /target + Cargo.lock. Both surfaced inline
in the per-iter journal's Concerns section.
cma.2 (harness) and cma.3 (live IONOS run + DESIGN.md addendum +
journal + roadmap edits) remain out of scope per the spec.
|