diff --git a/docs/roadmap.md b/docs/roadmap.md index 84fcac6..5ed9128 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -49,6 +49,34 @@ context. Pick the next milestone from P1.)_ has neither instance by design; polymorphic helpers at Float fire Float-aware `NoInstance` per DESIGN.md §"Float semantics". +- [ ] **\[milestone\]** Round-trip completeness invariant — `.ail.json` + ↔ `.ailx` must be a structurally lossless bijection. For every + valid module: `ail parse` followed by `ail render` yields the + original `.ailx` (modulo formatting), and `ail render` followed by + `ail parse` yields the original `.ail.json` (canonical key order). + Decision 6 mentions round-trip-as-property in passing, and + `crates/ailang-surface/tests/round_trip.rs` checks the `.ailx` + direction over the fixture corpus, but the invariant is not + prominently anchored as a non-negotiable language property and + the symmetric direction (every `.ail.json` has a printable, re- + parseable `.ailx` form) is not workspace-wide enforced. This + milestone (a) audits the existing fixture corpus for structural + gaps in either direction, (b) closes any render / parse gaps + surfaced, (c) lifts the round-trip property to a named, prominent + invariant in DESIGN.md (likely as a dedicated subsection of + Decision 6, or as its own decision-level entry), (d) adds a + workspace-wide check (E2E or unit, in addition to the existing + surface-side fixture test) that the invariant holds over every + fixture on every build. + - context: brainstorm 2026-05-12 — surfaced while scoping the + cross-model authoring-form test (see P2 entry below). The test + requires both forms to be exactly equipotent in expressive + power; otherwise measured form-preference is contaminated by + expressive-power gaps rather than authoring ergonomics. The + invariant is also load-bearing on its own: if `.ailx` is the + "AI authoring projection" per Decision 6, programs expressible + in one form but not the other are a first-class bug class. + - [ ] **\[milestone\]** Post-22 Prelude — Show + print rewire — ship `Show` typeclass with `Show Int/Bool/Str/Float` instances; rewire `print` through `Show.show`. Gated on heap-`Str` ABI runtime work @@ -61,6 +89,80 @@ context. Pick the next milestone from P1.)_ ## P2 — Medium-term +- [ ] **\[milestone\]** Cross-model authoring-form test — empirically + measure whether `.ail.json` or `.ailx` is the form a foreign LLM + author reaches for and succeeds with. Setup: three configurations + per task — (i) JSON-mandated, (ii) `.ailx`-mandated, (iii) free + choice (both corpora available, model picks). Multiple subjects to + check whether form-preference is model-robust. For each (model × + form × task): system prompt includes DESIGN.md verbatim (~28k + tokens); user prompt is the task plus the form mandate; an error- + roundtrip loop feeds `ail check` / `ail parse` errors back to the + model until green or iter-limit. Measure: first-attempt success + (yes/no), iterations to green, total tokens spent, error classes + encountered (schema violation, type mismatch, hash mismatch, syntax + error in the chosen form); for free-choice runs, which form the + model picked unprompted. The output is a JOURNAL'd dataset + + interpretation + a concrete DESIGN.md edit on Decision 6 — + either ratify "ailx is the AI authoring projection" with empirical + backing across N subjects, or retire the form if JSON wins + consistently across all subjects. + - context: brainstorm 2026-05-12 (aborted in favour of two roadmap + entries). The Decision-6 prose currently asserts `.ailx` is the + AI authoring projection while §"Feature-acceptance criterion" + cites JSON-as-canonical-authoring-surface as a behavioural + counterexample to human-driven feature picks. The tension is + real; only empirical data over multiple downstream subjects + resolves it. + - depends on: Round-trip completeness invariant (P1 above) — + without it, measured preference is confounded by expressive- + power gaps. Strongly benefits from (but is not strictly + blocked by): `ail check`/`build`/`run` accept `.ailx` + extension (P2 todo below); workspace search beyond entry- + module's directory (P2 todo below). Both will be encountered + during the test if not closed first. + - smoke-test evidence (2026-05-12): Qwen3-Coder-Next reads + DESIGN.md and produces a structurally correct AILang program + that passes `ail check` in two turns; first turn missed the + module-name-matches-file-path rule, second turn fixed it after + `ail check` error was fed back. ~28k prompt tokens, ~150 + completion tokens, ~2-5 s latency per turn. + - how to address the IONOS endpoint: + - URL: `https://openai.inference.de-txl.ionos.com/v1/chat/completions` + - protocol: OpenAI-compatible chat completions API. + `/v1/models` returns the catalogue. + - auth: HTTP header `Authorization: Bearer `. Store the + token outside the repo (e.g. `~/.ionos_token`, `chmod 600`). + Tokens are JWTs; the one used in the 2026-05-12 smoke test + was IONOS-issued and time-bound — refresh when re-engaging. + - model IDs to try (subset of the IONOS catalogue as of + 2026-05-12, sorted by relevance to this test): + - `Qwen/Qwen3-Coder-Next` — code-specialised, smoke-tested. + - `meta-llama/Meta-Llama-3.1-405B-Instruct-FP8` — large + general instruct, useful as a contrast subject. + - `meta-llama/Llama-3.3-70B-Instruct` — mid-size instruct. + - `openai/gpt-oss-120b` — OpenAI open-weight 120B. + - `mistralai/Mistral-Small-24B-Instruct` — smaller subject, + stresses spec comprehension. + - `meta-llama/CodeLlama-13b-Instruct-hf` — older code model, + useful as a low-capability floor. + - request shape: standard `messages: [{role: "system", ...}, + {role: "user", ...}, ...]`, with DESIGN.md (107 KB / ~28k + tokens) in the system message and the task + form mandate in + the first user message. Subsequent turns alternate + `assistant` / `user` with the `ail check` error verbatim in + the user message. + - response shape: `choices[0].message.content` carries the + generated program (request "no markdown fences, no + explanation" in the prompt to skip post-processing). + `usage.prompt_tokens` + `usage.completion_tokens` give per- + turn token spend. + - cost estimate: ~56k tokens per task per (model × form) + configuration including a few correction roundtrips; a + full matrix (4 tasks × 3 forms × 4 models × 1 run each) is + ~2.7M tokens total. Multiply by repetition count if a + median-of-3 design is chosen. + - [ ] **\[feature\]** Operator routing through `Eq` / `Ord` — `==`, `<` etc. resolved via the typeclass instead of the built-in primitive comparators. No commitment; gated on bench re-baselining