roadmap: roundtrip-invariant milestone (P1) + cross-model authoring-form test (P2)
Surfaced while scoping an empirical test of Decision 6 (.ailx as the AI authoring projection). The cross-model test depends on a structural .ail.json <-> .ailx bijection being established and prominently anchored in DESIGN.md first; otherwise measured form-preference is contaminated by expressive-power gaps rather than authoring ergonomics. P2 entry carries the smoke-test evidence and full IONOS-endpoint addressing notes so the test can be picked up without redoing the connection work.
This commit is contained in:
+102
@@ -49,6 +49,34 @@ context. Pick the next milestone from P1.)_
|
||||
has neither instance by design; polymorphic helpers at Float fire
|
||||
Float-aware `NoInstance` per DESIGN.md §"Float semantics".
|
||||
|
||||
- [ ] **\[milestone\]** Round-trip completeness invariant — `.ail.json`
|
||||
↔ `.ailx` must be a structurally lossless bijection. For every
|
||||
valid module: `ail parse` followed by `ail render` yields the
|
||||
original `.ailx` (modulo formatting), and `ail render` followed by
|
||||
`ail parse` yields the original `.ail.json` (canonical key order).
|
||||
Decision 6 mentions round-trip-as-property in passing, and
|
||||
`crates/ailang-surface/tests/round_trip.rs` checks the `.ailx`
|
||||
direction over the fixture corpus, but the invariant is not
|
||||
prominently anchored as a non-negotiable language property and
|
||||
the symmetric direction (every `.ail.json` has a printable, re-
|
||||
parseable `.ailx` form) is not workspace-wide enforced. This
|
||||
milestone (a) audits the existing fixture corpus for structural
|
||||
gaps in either direction, (b) closes any render / parse gaps
|
||||
surfaced, (c) lifts the round-trip property to a named, prominent
|
||||
invariant in DESIGN.md (likely as a dedicated subsection of
|
||||
Decision 6, or as its own decision-level entry), (d) adds a
|
||||
workspace-wide check (E2E or unit, in addition to the existing
|
||||
surface-side fixture test) that the invariant holds over every
|
||||
fixture on every build.
|
||||
- context: brainstorm 2026-05-12 — surfaced while scoping the
|
||||
cross-model authoring-form test (see P2 entry below). The test
|
||||
requires both forms to be exactly equipotent in expressive
|
||||
power; otherwise measured form-preference is contaminated by
|
||||
expressive-power gaps rather than authoring ergonomics. The
|
||||
invariant is also load-bearing on its own: if `.ailx` is the
|
||||
"AI authoring projection" per Decision 6, programs expressible
|
||||
in one form but not the other are a first-class bug class.
|
||||
|
||||
- [ ] **\[milestone\]** Post-22 Prelude — Show + print rewire — ship
|
||||
`Show` typeclass with `Show Int/Bool/Str/Float` instances; rewire
|
||||
`print` through `Show.show`. Gated on heap-`Str` ABI runtime work
|
||||
@@ -61,6 +89,80 @@ context. Pick the next milestone from P1.)_
|
||||
|
||||
## P2 — Medium-term
|
||||
|
||||
- [ ] **\[milestone\]** Cross-model authoring-form test — empirically
|
||||
measure whether `.ail.json` or `.ailx` is the form a foreign LLM
|
||||
author reaches for and succeeds with. Setup: three configurations
|
||||
per task — (i) JSON-mandated, (ii) `.ailx`-mandated, (iii) free
|
||||
choice (both corpora available, model picks). Multiple subjects to
|
||||
check whether form-preference is model-robust. For each (model ×
|
||||
form × task): system prompt includes DESIGN.md verbatim (~28k
|
||||
tokens); user prompt is the task plus the form mandate; an error-
|
||||
roundtrip loop feeds `ail check` / `ail parse` errors back to the
|
||||
model until green or iter-limit. Measure: first-attempt success
|
||||
(yes/no), iterations to green, total tokens spent, error classes
|
||||
encountered (schema violation, type mismatch, hash mismatch, syntax
|
||||
error in the chosen form); for free-choice runs, which form the
|
||||
model picked unprompted. The output is a JOURNAL'd dataset +
|
||||
interpretation + a concrete DESIGN.md edit on Decision 6 —
|
||||
either ratify "ailx is the AI authoring projection" with empirical
|
||||
backing across N subjects, or retire the form if JSON wins
|
||||
consistently across all subjects.
|
||||
- context: brainstorm 2026-05-12 (aborted in favour of two roadmap
|
||||
entries). The Decision-6 prose currently asserts `.ailx` is the
|
||||
AI authoring projection while §"Feature-acceptance criterion"
|
||||
cites JSON-as-canonical-authoring-surface as a behavioural
|
||||
counterexample to human-driven feature picks. The tension is
|
||||
real; only empirical data over multiple downstream subjects
|
||||
resolves it.
|
||||
- depends on: Round-trip completeness invariant (P1 above) —
|
||||
without it, measured preference is confounded by expressive-
|
||||
power gaps. Strongly benefits from (but is not strictly
|
||||
blocked by): `ail check`/`build`/`run` accept `.ailx`
|
||||
extension (P2 todo below); workspace search beyond entry-
|
||||
module's directory (P2 todo below). Both will be encountered
|
||||
during the test if not closed first.
|
||||
- smoke-test evidence (2026-05-12): Qwen3-Coder-Next reads
|
||||
DESIGN.md and produces a structurally correct AILang program
|
||||
that passes `ail check` in two turns; first turn missed the
|
||||
module-name-matches-file-path rule, second turn fixed it after
|
||||
`ail check` error was fed back. ~28k prompt tokens, ~150
|
||||
completion tokens, ~2-5 s latency per turn.
|
||||
- how to address the IONOS endpoint:
|
||||
- URL: `https://openai.inference.de-txl.ionos.com/v1/chat/completions`
|
||||
- protocol: OpenAI-compatible chat completions API.
|
||||
`/v1/models` returns the catalogue.
|
||||
- auth: HTTP header `Authorization: Bearer <token>`. Store the
|
||||
token outside the repo (e.g. `~/.ionos_token`, `chmod 600`).
|
||||
Tokens are JWTs; the one used in the 2026-05-12 smoke test
|
||||
was IONOS-issued and time-bound — refresh when re-engaging.
|
||||
- model IDs to try (subset of the IONOS catalogue as of
|
||||
2026-05-12, sorted by relevance to this test):
|
||||
- `Qwen/Qwen3-Coder-Next` — code-specialised, smoke-tested.
|
||||
- `meta-llama/Meta-Llama-3.1-405B-Instruct-FP8` — large
|
||||
general instruct, useful as a contrast subject.
|
||||
- `meta-llama/Llama-3.3-70B-Instruct` — mid-size instruct.
|
||||
- `openai/gpt-oss-120b` — OpenAI open-weight 120B.
|
||||
- `mistralai/Mistral-Small-24B-Instruct` — smaller subject,
|
||||
stresses spec comprehension.
|
||||
- `meta-llama/CodeLlama-13b-Instruct-hf` — older code model,
|
||||
useful as a low-capability floor.
|
||||
- request shape: standard `messages: [{role: "system", ...},
|
||||
{role: "user", ...}, ...]`, with DESIGN.md (107 KB / ~28k
|
||||
tokens) in the system message and the task + form mandate in
|
||||
the first user message. Subsequent turns alternate
|
||||
`assistant` / `user` with the `ail check` error verbatim in
|
||||
the user message.
|
||||
- response shape: `choices[0].message.content` carries the
|
||||
generated program (request "no markdown fences, no
|
||||
explanation" in the prompt to skip post-processing).
|
||||
`usage.prompt_tokens` + `usage.completion_tokens` give per-
|
||||
turn token spend.
|
||||
- cost estimate: ~56k tokens per task per (model × form)
|
||||
configuration including a few correction roundtrips; a
|
||||
full matrix (4 tasks × 3 forms × 4 models × 1 run each) is
|
||||
~2.7M tokens total. Multiply by repetition count if a
|
||||
median-of-3 design is chosen.
|
||||
|
||||
- [ ] **\[feature\]** Operator routing through `Eq` / `Ord` — `==`,
|
||||
`<` etc. resolved via the typeclass instead of the built-in
|
||||
primitive comparators. No commitment; gated on bench re-baselining
|
||||
|
||||
Reference in New Issue
Block a user