roadmap: roundtrip-invariant milestone (P1) + cross-model authoring-form test (P2)

Surfaced while scoping an empirical test of Decision 6 (.ailx as the
AI authoring projection). The cross-model test depends on a structural
.ail.json <-> .ailx bijection being established and prominently anchored
in DESIGN.md first; otherwise measured form-preference is contaminated
by expressive-power gaps rather than authoring ergonomics. P2 entry
carries the smoke-test evidence and full IONOS-endpoint addressing
notes so the test can be picked up without redoing the connection work.
This commit is contained in:
2026-05-12 08:53:47 +02:00
parent edccc087a4
commit daf9f4fdc9
+102
View File
@@ -49,6 +49,34 @@ context. Pick the next milestone from P1.)_
has neither instance by design; polymorphic helpers at Float fire
Float-aware `NoInstance` per DESIGN.md §"Float semantics".
- [ ] **\[milestone\]** Round-trip completeness invariant — `.ail.json`
`.ailx` must be a structurally lossless bijection. For every
valid module: `ail parse` followed by `ail render` yields the
original `.ailx` (modulo formatting), and `ail render` followed by
`ail parse` yields the original `.ail.json` (canonical key order).
Decision 6 mentions round-trip-as-property in passing, and
`crates/ailang-surface/tests/round_trip.rs` checks the `.ailx`
direction over the fixture corpus, but the invariant is not
prominently anchored as a non-negotiable language property and
the symmetric direction (every `.ail.json` has a printable, re-
parseable `.ailx` form) is not workspace-wide enforced. This
milestone (a) audits the existing fixture corpus for structural
gaps in either direction, (b) closes any render / parse gaps
surfaced, (c) lifts the round-trip property to a named, prominent
invariant in DESIGN.md (likely as a dedicated subsection of
Decision 6, or as its own decision-level entry), (d) adds a
workspace-wide check (E2E or unit, in addition to the existing
surface-side fixture test) that the invariant holds over every
fixture on every build.
- context: brainstorm 2026-05-12 — surfaced while scoping the
cross-model authoring-form test (see P2 entry below). The test
requires both forms to be exactly equipotent in expressive
power; otherwise measured form-preference is contaminated by
expressive-power gaps rather than authoring ergonomics. The
invariant is also load-bearing on its own: if `.ailx` is the
"AI authoring projection" per Decision 6, programs expressible
in one form but not the other are a first-class bug class.
- [ ] **\[milestone\]** Post-22 Prelude — Show + print rewire — ship
`Show` typeclass with `Show Int/Bool/Str/Float` instances; rewire
`print` through `Show.show`. Gated on heap-`Str` ABI runtime work
@@ -61,6 +89,80 @@ context. Pick the next milestone from P1.)_
## P2 — Medium-term
- [ ] **\[milestone\]** Cross-model authoring-form test — empirically
measure whether `.ail.json` or `.ailx` is the form a foreign LLM
author reaches for and succeeds with. Setup: three configurations
per task — (i) JSON-mandated, (ii) `.ailx`-mandated, (iii) free
choice (both corpora available, model picks). Multiple subjects to
check whether form-preference is model-robust. For each (model ×
form × task): system prompt includes DESIGN.md verbatim (~28k
tokens); user prompt is the task plus the form mandate; an error-
roundtrip loop feeds `ail check` / `ail parse` errors back to the
model until green or iter-limit. Measure: first-attempt success
(yes/no), iterations to green, total tokens spent, error classes
encountered (schema violation, type mismatch, hash mismatch, syntax
error in the chosen form); for free-choice runs, which form the
model picked unprompted. The output is a JOURNAL'd dataset +
interpretation + a concrete DESIGN.md edit on Decision 6 —
either ratify "ailx is the AI authoring projection" with empirical
backing across N subjects, or retire the form if JSON wins
consistently across all subjects.
- context: brainstorm 2026-05-12 (aborted in favour of two roadmap
entries). The Decision-6 prose currently asserts `.ailx` is the
AI authoring projection while §"Feature-acceptance criterion"
cites JSON-as-canonical-authoring-surface as a behavioural
counterexample to human-driven feature picks. The tension is
real; only empirical data over multiple downstream subjects
resolves it.
- depends on: Round-trip completeness invariant (P1 above) —
without it, measured preference is confounded by expressive-
power gaps. Strongly benefits from (but is not strictly
blocked by): `ail check`/`build`/`run` accept `.ailx`
extension (P2 todo below); workspace search beyond entry-
module's directory (P2 todo below). Both will be encountered
during the test if not closed first.
- smoke-test evidence (2026-05-12): Qwen3-Coder-Next reads
DESIGN.md and produces a structurally correct AILang program
that passes `ail check` in two turns; first turn missed the
module-name-matches-file-path rule, second turn fixed it after
`ail check` error was fed back. ~28k prompt tokens, ~150
completion tokens, ~2-5 s latency per turn.
- how to address the IONOS endpoint:
- URL: `https://openai.inference.de-txl.ionos.com/v1/chat/completions`
- protocol: OpenAI-compatible chat completions API.
`/v1/models` returns the catalogue.
- auth: HTTP header `Authorization: Bearer <token>`. Store the
token outside the repo (e.g. `~/.ionos_token`, `chmod 600`).
Tokens are JWTs; the one used in the 2026-05-12 smoke test
was IONOS-issued and time-bound — refresh when re-engaging.
- model IDs to try (subset of the IONOS catalogue as of
2026-05-12, sorted by relevance to this test):
- `Qwen/Qwen3-Coder-Next` — code-specialised, smoke-tested.
- `meta-llama/Meta-Llama-3.1-405B-Instruct-FP8` — large
general instruct, useful as a contrast subject.
- `meta-llama/Llama-3.3-70B-Instruct` — mid-size instruct.
- `openai/gpt-oss-120b` — OpenAI open-weight 120B.
- `mistralai/Mistral-Small-24B-Instruct` — smaller subject,
stresses spec comprehension.
- `meta-llama/CodeLlama-13b-Instruct-hf` — older code model,
useful as a low-capability floor.
- request shape: standard `messages: [{role: "system", ...},
{role: "user", ...}, ...]`, with DESIGN.md (107 KB / ~28k
tokens) in the system message and the task + form mandate in
the first user message. Subsequent turns alternate
`assistant` / `user` with the `ail check` error verbatim in
the user message.
- response shape: `choices[0].message.content` carries the
generated program (request "no markdown fences, no
explanation" in the prompt to skip post-processing).
`usage.prompt_tokens` + `usage.completion_tokens` give per-
turn token spend.
- cost estimate: ~56k tokens per task per (model × form)
configuration including a few correction roundtrips; a
full matrix (4 tasks × 3 forms × 4 models × 1 run each) is
~2.7M tokens total. Multiply by repetition count if a
median-of-3 design is chosen.
- [ ] **\[feature\]** Operator routing through `Eq` / `Ord` — `==`,
`<` etc. resolved via the typeclass instead of the built-in
primitive comparators. No commitment; gated on bench re-baselining