Files

T

Brummel 94d6963995 audit-cma: close milestone Cross-model authoring-form test

Architect drift_found, both [low]:
- INDEX backfill for commit 90512d5 (brainstorm Step 7.5 SKILL.md
  edit + roadmap P1 todo removed without a paired INDEX mirror).
- README "Total 8 passed" understated the actual 13/13 sweep.

Both fixed inline (mechanical, ≤30 LOC) per CLAUDE.md trivial-edits
carve-out; no implementer dispatch.

Bench: all three scripts exit 0, 0 regressed, 5 unexplained
latency.explicit_at_rc improvements not coupled to milestone work
(no in-workspace crate touched). Baseline left pristine — future
audits will reveal whether the improvements hold.

All 9 spec acceptance criteria green. Milestone closed clean.

2026-05-12 12:20:30 +02:00

3.2 KiB

Raw Blame History

Cross-model authoring-form test

Empirical measurement of whether .ail.json or .ailx is the form a foreign LLM author reaches for and succeeds with. Single subject for v1: Qwen3-Coder-Next via IONOS. Two blind cohorts; same four tasks.

Parent spec: docs/specs/2026-05-12-cross-model-authoring-form-test.md.

Layout

master/spec.md — canonical mini-spec source (form-agnostic prose + {form-only: X} blocks + {example: id} markers).
master/examples/*.ail.json — AST source-of-truth; each example prints to either form via the existing roundtrip machinery.
master/tasks/*.task.json — task definitions consumed by the harness. Authored in cma.2, not cma.1.
render/ — standalone Cargo crate, outside the root workspace, builds the renderer binary.
rendered/json.md, rendered/ailx.md — projected mini-specs, checked into the repo for review.
harness/ — Authored in cma.2.
runs/<date>-<hash>/ — populated by harness during a live run.

Running the renderer

cargo run --manifest-path experiments/2026-05-12-cross-model-authoring/render/Cargo.toml -- \
    --master  experiments/2026-05-12-cross-model-authoring/master \
    --rendered experiments/2026-05-12-cross-model-authoring/rendered

Running the tests

cargo test --manifest-path experiments/2026-05-12-cross-model-authoring/render/Cargo.toml

Three integration tests: example_roundtrip (each example loads, prints to AILX, reparses to the same canonical bytes), spec_completeness (every AST variant in ailang_core::ast is exercised by at least one example), token_balance (form-only blocks balanced within ±5% across the two rendered files).

Running the harness

Live mode (one full eight-run sweep, ~480k tokens budget by default):

export IONOS_API_TOKEN="<token>"   # see roadmap entry for token provenance
cargo run --manifest-path experiments/2026-05-12-cross-model-authoring/harness/Cargo.toml -- \
  --rendered experiments/2026-05-12-cross-model-authoring/rendered \
  --tasks    experiments/2026-05-12-cross-model-authoring/master/tasks \
  --out      experiments/2026-05-12-cross-model-authoring/runs \
  --model    Qwen/Qwen3-Coder-Next

The harness pre-flights ail --version and clang --version before the first API call. Set AIL_BIN if ail is not on PATH.

Mock mode (offline; CI-friendly; bypasses the IONOS API):

cargo run --manifest-path experiments/2026-05-12-cross-model-authoring/harness/Cargo.toml -- \
  --rendered experiments/2026-05-12-cross-model-authoring/rendered \
  --tasks    experiments/2026-05-12-cross-model-authoring/master/tasks \
  --out      /tmp/mock-runs \
  --model    mock \
  --mock     experiments/2026-05-12-cross-model-authoring/harness/tests/fixtures/mock_full_run.json

Tests:

cargo test --manifest-path experiments/2026-05-12-cross-model-authoring/harness/Cargo.toml

Five suites: inline --lib unit tests for strip_locations (5), plus integration tests strip_locations against captured stderr fixtures (5), verify_references (1, drives every reference through the real ail+clang pipeline), mock_full_run (1, full eight-row mock E2E), budget_abort (1). Total 13 passed.

3.2 KiB Raw Blame History