Brummel
2b9a1083b5
User asked whether Qwen's tool-calling could help. Verified tool-calling is
supported on IONOS (a dummy add tool returns a correct tool_call). Then tested
the promising version: submit the program as canonical AST JSON via a forced
submit_program tool — no surface syntax, no converter risk (the tool argument
IS the .ail.json, written and checked directly).
Result: 2/8, tied for worst. Two failure modes, both the model's: valid JSON
but wrong AST (arity etc.), and — at L4/L7 — bad-json: the tool arguments were
not valid JSON at all, because completion hit the 2500-token cap. AST JSON is
so verbose (every leaf is {"t":"var","name":"x"}) that a non-trivial program
overruns the budget and truncates. IONOS does not constrain decoding to the
schema, so the full structure burden stays on the model, and it is the heaviest
of all surfaces (braces+brackets+keys+quotes+commas+"t": per node).
This confirms the rule across six surfaces (annotated-parens 7/8 > plain parens
6/8 > M-expr 4-5/8 > indent 3/8 > YAML 2/8 = tool-AST 2/8): the discriminator is
redundant cue vs extra burden, not explicit vs implicit. Even tool-calling, the
model's home turf, loses because AST JSON maximises bookkeeping with no
redundancy. Lever for an LLM-authored language: redundant self-checking
structure (depth-annotated brackets). Synthesis in format-findings.md.
Cross-model authoring-form test
Empirical measurement of whether .ail.json or .ail is the form a
foreign LLM author reaches for and succeeds with. Single subject for
v1: Qwen3-Coder-Next via IONOS. Two blind cohorts; same four tasks.
Parent spec: docs/specs/0017-cross-model-authoring-form-test.md.
Layout
master/spec.md— canonical mini-spec source (form-agnostic prose +{form-only: X}blocks +{example: id}markers).master/examples/*.ail.json— AST source-of-truth; each example prints to either form via the existing roundtrip machinery.master/tasks/*.task.json— task definitions consumed by the harness. Authored in cma.2, not cma.1.render/— standalone Cargo crate, outside the root workspace, builds the renderer binary.rendered/json.md,rendered/ail.md— projected mini-specs, checked into the repo for review.harness/— Authored in cma.2.runs/<date>-<hash>/— populated byharnessduring a live run.
Running the renderer
cargo run --manifest-path experiments/2026-05-12-cross-model-authoring/render/Cargo.toml -- \
--master experiments/2026-05-12-cross-model-authoring/master \
--rendered experiments/2026-05-12-cross-model-authoring/rendered
Running the tests
cargo test --manifest-path experiments/2026-05-12-cross-model-authoring/render/Cargo.toml
Three integration tests: example_roundtrip (each example loads,
prints to AIL, reparses to the same canonical bytes), spec_completeness
(every AST variant in ailang_core::ast is exercised by at least
one example), token_balance (form-only blocks balanced within ±5%
across the two rendered files).
Running the harness
Live mode (one full eight-run sweep, ~480k tokens budget by default):
export IONOS_API_TOKEN="<token>" # see roadmap entry for token provenance
cargo run --manifest-path experiments/2026-05-12-cross-model-authoring/harness/Cargo.toml -- \
--rendered experiments/2026-05-12-cross-model-authoring/rendered \
--tasks experiments/2026-05-12-cross-model-authoring/master/tasks \
--out experiments/2026-05-12-cross-model-authoring/runs \
--model Qwen/Qwen3-Coder-Next
The harness pre-flights ail --version and clang --version before
the first API call. Set AIL_BIN if ail is not on PATH.
Mock mode (offline; CI-friendly; bypasses the IONOS API):
cargo run --manifest-path experiments/2026-05-12-cross-model-authoring/harness/Cargo.toml -- \
--rendered experiments/2026-05-12-cross-model-authoring/rendered \
--tasks experiments/2026-05-12-cross-model-authoring/master/tasks \
--out /tmp/mock-runs \
--model mock \
--mock experiments/2026-05-12-cross-model-authoring/harness/tests/fixtures/mock_full_run.json
Tests:
cargo test --manifest-path experiments/2026-05-12-cross-model-authoring/harness/Cargo.toml
Five suites: inline --lib unit tests for strip_locations (5),
plus integration tests strip_locations against captured stderr
fixtures (5), verify_references (1, drives every reference through
the real ail+clang pipeline), mock_full_run (1, full eight-row
mock E2E), budget_abort (1). Total 13 passed.