Brummel
5bd148a607
Three-cohort harness against IONOS-hosted Qwen3-Coder-Next, testing
whether application-head tag (`app` vs `apply` vs `call`) affects
LLM authoring success. Cohorts share lambda/constructor/typecon tag
renamings (the non-discriminator changes); only the head differs.
Result: 100 % cohort-treue — Qwen writes whatever the spec teaches,
no measurable natural preference. Pipeline pass-rate (classic 1/8,
apply 0/8, call 1/8) is cohort-independent. The four task failures
are general Form-A friction, not naming-related. Refactor not
justified per the feature-acceptance gate.
Tracked harness:
- 2026-05-12-cross-model-authoring/rename-spec.py — regex tag
rewriter for the two non-classic spec variants
- 2026-05-21-naming-ab/run.py, reprocess.py — three-cohort runner
+ post-processing with disjoint-discriminator counting and
module-name-matched temp dirs
Generated artefacts (runs/, rendered/ail-{renamed,call}.md) are
gitignored — regeneratable from rename-spec.py + run.py.
Side-effects filed against current spec/schema bugs surfaced during
the run:
- refs #28 spec teaches (ctor X) for term position; parser requires
(term-ctor X) — 100 % t4 failure across cohorts
- refs #29 io/print_str appends newline (de facto println);
spec now documents the behavior, code question still open
- refs #30 schema camelCase outlier: paramTypes/retType in Term::Lam
Spec patch in rendered/ail.md:
- replaces stale io/print_int references with io/print_str (bitrot
from the print-builtin consolidation)
- documents io/print_str's trailing-newline behavior
- corrects the prelude claim about int_to_str (IS in prelude)
3 lines
45 B
Plaintext
3 lines
45 B
Plaintext
rendered/ail-renamed.md
|
|
rendered/ail-call.md
|