fieldtest: naming A/B against Qwen3-Coder — refactor not justified
Three-cohort harness against IONOS-hosted Qwen3-Coder-Next, testing
whether application-head tag (`app` vs `apply` vs `call`) affects
LLM authoring success. Cohorts share lambda/constructor/typecon tag
renamings (the non-discriminator changes); only the head differs.
Result: 100 % cohort-treue — Qwen writes whatever the spec teaches,
no measurable natural preference. Pipeline pass-rate (classic 1/8,
apply 0/8, call 1/8) is cohort-independent. The four task failures
are general Form-A friction, not naming-related. Refactor not
justified per the feature-acceptance gate.
Tracked harness:
- 2026-05-12-cross-model-authoring/rename-spec.py — regex tag
rewriter for the two non-classic spec variants
- 2026-05-21-naming-ab/run.py, reprocess.py — three-cohort runner
+ post-processing with disjoint-discriminator counting and
module-name-matched temp dirs
Generated artefacts (runs/, rendered/ail-{renamed,call}.md) are
gitignored — regeneratable from rename-spec.py + run.py.
Side-effects filed against current spec/schema bugs surfaced during
the run:
- refs #28 spec teaches (ctor X) for term position; parser requires
(term-ctor X) — 100 % t4 failure across cohorts
- refs #29 io/print_str appends newline (de facto println);
spec now documents the behavior, code question still open
- refs #30 schema camelCase outlier: paramTypes/retType in Term::Lam
Spec patch in rendered/ail.md:
- replaces stale io/print_int references with io/print_str (bitrot
from the print-builtin consolidation)
- documents io/print_str's trailing-newline behavior
- corrects the prelude claim about int_to_str (IS in prelude)
This commit is contained in:
@@ -0,0 +1,2 @@
|
||||
rendered/ail-renamed.md
|
||||
rendered/ail-call.md
|
||||
@@ -0,0 +1,62 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Produce a renamed-tags variant of the rendered Form-A spec.
|
||||
|
||||
Two modes:
|
||||
- apply: full-form resolution of abbreviations
|
||||
app→apply, lam→lambda, ctor→constructor, con→typecon, tail-app→tail-apply
|
||||
- call: like apply but with call as the application head (semantic shift)
|
||||
app→call, tail-app→tail-call, rest as in apply
|
||||
|
||||
Each pattern is matched in two contexts:
|
||||
- S-expression head: `(NAME ` and `(NAME)`
|
||||
- Backtick mention: `` `NAME` ``
|
||||
|
||||
Plain-text occurrences ("application", "construction", etc.) are
|
||||
preserved by anchoring on the contexts above.
|
||||
"""
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
PAIRS_APPLY = [
|
||||
("tail-app", "tail-apply"),
|
||||
("term-ctor", "term-constructor"),
|
||||
("pat-ctor", "pat-constructor"),
|
||||
("ctor", "constructor"),
|
||||
("app", "apply"),
|
||||
("lam", "lambda"),
|
||||
("con", "typecon"),
|
||||
]
|
||||
|
||||
PAIRS_CALL = [
|
||||
("tail-app", "tail-call"),
|
||||
("term-ctor", "term-constructor"),
|
||||
("pat-ctor", "pat-constructor"),
|
||||
("ctor", "constructor"),
|
||||
("app", "call"),
|
||||
("lam", "lambda"),
|
||||
("con", "typecon"),
|
||||
]
|
||||
|
||||
MODES = {"apply": PAIRS_APPLY, "call": PAIRS_CALL}
|
||||
|
||||
def transform(src: str, pairs) -> str:
|
||||
out = src
|
||||
for old, new in pairs:
|
||||
out = re.sub(rf"\({re.escape(old)}(?=[\s)])", f"({new}", out)
|
||||
out = re.sub(rf"`{re.escape(old)}`", f"`{new}`", out)
|
||||
return out
|
||||
|
||||
def main() -> int:
|
||||
if len(sys.argv) != 4 or sys.argv[1] not in MODES:
|
||||
print(f"usage: {sys.argv[0]} <apply|call> <input.md> <output.md>", file=sys.stderr)
|
||||
return 2
|
||||
mode = sys.argv[1]
|
||||
inp = Path(sys.argv[2])
|
||||
out = Path(sys.argv[3])
|
||||
out.write_text(transform(inp.read_text(), MODES[mode]))
|
||||
print(f"wrote {out} ({out.stat().st_size} bytes, mode={mode})")
|
||||
return 0
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -349,13 +349,13 @@ arms or into code outside the `match`.
|
||||
|
||||
Every function type carries an effect row — a set of effect labels
|
||||
that the function may raise. The two currently wired effects are
|
||||
`IO` (required to call effect operations like `io/print_int`) and
|
||||
`IO` (required to call effect operations like `io/print_str`) and
|
||||
`Diverge` (used for functions that may not terminate). An empty
|
||||
effect row marks the function as pure.
|
||||
|
||||
Effect operations are reached through the `do` term, not through a
|
||||
regular function application. The `do` form names the operation
|
||||
(e.g. `io/print_int`) and lists the operation's arguments; the
|
||||
(e.g. `io/print_str`) and lists the operation's arguments; the
|
||||
typechecker resolves the operation against the prelude's effect-op
|
||||
table, checks the arguments, and accumulates the operation's
|
||||
effect label into the enclosing function's effect row.
|
||||
@@ -374,8 +374,9 @@ in canonical output.
|
||||
|
||||
Effect operation: `(do OP-NAME ARG*)` — the leading `do` head is
|
||||
the parser's signal that the next identifier is an effect op, not a
|
||||
function. Example: `(do io/print_int 42)` prints the integer 42
|
||||
and raises the `IO` effect.
|
||||
function. Example: `(do io/print_str "hi\n")` prints the string
|
||||
`hi` followed by a newline and raises the `IO` effect. To print a
|
||||
non-string value, first convert it: `(do io/print_str (app int_to_str 42))`.
|
||||
|
||||
Sequencing: `(seq LHS RHS)`. The value of `(seq X Y)` is `Y`'s
|
||||
value; `X`'s value is discarded. Chains of sequencing are written
|
||||
@@ -385,10 +386,10 @@ value.
|
||||
```ail
|
||||
(module fn_with_do_seq
|
||||
(fn main
|
||||
(doc "Print two Ints in sequence. Exercises TermDo, TermSeq, and the IO effect on a fn type. The trailing unit return is implicit in the last Do (op returns Unit).")
|
||||
(doc "Print two Ints in sequence. Exercises TermDo, TermSeq, the IO effect on a fn type, and int-to-str conversion via the prelude `int_to_str`. The trailing unit return is implicit in the last Do (op returns Unit).")
|
||||
(type (fn-type (params) (ret (con Unit)) (effects IO)))
|
||||
(params)
|
||||
(body (seq (do io/print_int 1) (do io/print_int 2)))))
|
||||
(body (seq (do io/print_str (app int_to_str 1)) (do io/print_str (app int_to_str 2))))))
|
||||
```
|
||||
|
||||
## 9. Literals
|
||||
@@ -547,14 +548,14 @@ Effect operations (reached through `do`, not `app`):
|
||||
|
||||
| Op | Signature | Effect |
|
||||
|----|----------|-------|
|
||||
| `io/print_int` | `(Int) -> Unit` | `IO` |
|
||||
| `io/print_bool` | `(Bool) -> Unit` | `IO` |
|
||||
| `io/print_str` | `(Str) -> Unit` | `IO` |
|
||||
| `io/print_float` | `(Float) -> Unit` | `IO` |
|
||||
|
||||
`int_to_str` is intentionally NOT in the prelude — it is type-
|
||||
installed in a future milestone but codegen-deferred pending the
|
||||
heap-Str ABI work. Do not call it.
|
||||
`io/print_str` is the only print operation, and it appends a
|
||||
trailing newline (the runtime calls C `puts`). To print a non-string
|
||||
value, convert it first via `int_to_str`, `float_to_str`, or
|
||||
`bool_to_str` — all three are in the prelude (see the prelude
|
||||
table above). To print two values on consecutive lines, sequence
|
||||
two `(do io/print_str ...)` calls with `seq`.
|
||||
|
||||
## 12. Content addressing
|
||||
|
||||
|
||||
Reference in New Issue
Block a user