Brummel
da99ebaba8
Try-and-error ablation, minimal few-shot context, single shot per level, a ladder L0 (fn returns 42) to L7 (full SMA). Two ladders: base few-shot that only destructures, then one that also constructs (term-ctor). Finding: Qwen is NOT generally incapable. With minimal correct context it writes 6/8 tasks green, including ADTs, match, recursive fns, and Series use. The failures are two specific walls: - Wall 1 (cheap): unfamiliar constructs. It built lists with (app Cons ...) — calling the ctor like a function — because it had only seen match, never term-ctor. Adding ONE term-ctor example flipped L3 and L5 to green and removed the Cons error. The model generalises a construct from one example. - Wall 2 (hard): paren/nesting discipline at depth. The two that stay red (L4, L7) fail purely on bracketing, not knowledge. In L4 the recursive function is flawless; main closes a 4x-nested term-ctor value one paren early so the let body slides inside it — total paren count is BALANCED (43/43), the parens are just misplaced. Green/red tracks nesting depth, not feature. Examples do not fix it. This is the same weakness the full-SMA probes hit (run 2 had correct logic, died on brackets; run 3's long context tipped it into a seq-repetition loop). Takeaway: the bottleneck is the fully-parenthesised surface at depth, not the semantics. Natural next test: the .ail.json authoring form, which removes human paren-counting. Driver qwen_ablation.py; ladders in qwen-ablation-min.md and qwen-ablation-ctor.md; synthesis in qwen-ablation.md.
Cross-model authoring-form test
Empirical measurement of whether .ail.json or .ail is the form a
foreign LLM author reaches for and succeeds with. Single subject for
v1: Qwen3-Coder-Next via IONOS. Two blind cohorts; same four tasks.
Parent spec: docs/specs/0017-cross-model-authoring-form-test.md.
Layout
master/spec.md— canonical mini-spec source (form-agnostic prose +{form-only: X}blocks +{example: id}markers).master/examples/*.ail.json— AST source-of-truth; each example prints to either form via the existing roundtrip machinery.master/tasks/*.task.json— task definitions consumed by the harness. Authored in cma.2, not cma.1.render/— standalone Cargo crate, outside the root workspace, builds the renderer binary.rendered/json.md,rendered/ail.md— projected mini-specs, checked into the repo for review.harness/— Authored in cma.2.runs/<date>-<hash>/— populated byharnessduring a live run.
Running the renderer
cargo run --manifest-path experiments/2026-05-12-cross-model-authoring/render/Cargo.toml -- \
--master experiments/2026-05-12-cross-model-authoring/master \
--rendered experiments/2026-05-12-cross-model-authoring/rendered
Running the tests
cargo test --manifest-path experiments/2026-05-12-cross-model-authoring/render/Cargo.toml
Three integration tests: example_roundtrip (each example loads,
prints to AIL, reparses to the same canonical bytes), spec_completeness
(every AST variant in ailang_core::ast is exercised by at least
one example), token_balance (form-only blocks balanced within ±5%
across the two rendered files).
Running the harness
Live mode (one full eight-run sweep, ~480k tokens budget by default):
export IONOS_API_TOKEN="<token>" # see roadmap entry for token provenance
cargo run --manifest-path experiments/2026-05-12-cross-model-authoring/harness/Cargo.toml -- \
--rendered experiments/2026-05-12-cross-model-authoring/rendered \
--tasks experiments/2026-05-12-cross-model-authoring/master/tasks \
--out experiments/2026-05-12-cross-model-authoring/runs \
--model Qwen/Qwen3-Coder-Next
The harness pre-flights ail --version and clang --version before
the first API call. Set AIL_BIN if ail is not on PATH.
Mock mode (offline; CI-friendly; bypasses the IONOS API):
cargo run --manifest-path experiments/2026-05-12-cross-model-authoring/harness/Cargo.toml -- \
--rendered experiments/2026-05-12-cross-model-authoring/rendered \
--tasks experiments/2026-05-12-cross-model-authoring/master/tasks \
--out /tmp/mock-runs \
--model mock \
--mock experiments/2026-05-12-cross-model-authoring/harness/tests/fixtures/mock_full_run.json
Tests:
cargo test --manifest-path experiments/2026-05-12-cross-model-authoring/harness/Cargo.toml
Five suites: inline --lib unit tests for strip_locations (5),
plus integration tests strip_locations against captured stderr
fixtures (5), verify_references (1, drives every reference through
the real ail+clang pipeline), mock_full_run (1, full eight-row
mock E2E), budget_abort (1). Total 13 passed.