Brummel
fe1fb6b4f0
Sibling standalone Cargo crate `harness/` under experiments/2026-05-12-cross-model-authoring/ (out-of-workspace idiom carried forward from cma.1 verbatim). Six modules: strip_locations (regex pass for form-asymmetric location info, calibrated against five real `ail check`/`ail parse` stderr captures), pipeline (subprocess wrapper for parse|check|build + 5s-timeout exec; preflight on ail+clang), ionos (blocking reqwest client + retry policy per spec), mock (canned-response loader), scoring (CSV + summary.md), tasks (definition struct + loader). main.rs ties them into the per-(cohort,task) loop with budget accounting and per-turn artefact recording. Four MVP tasks land with reference solutions that compile, build, and execute green through the actual ail+clang pipeline: t1_add_three (chained `+` + io/print_int), t2_length (polymorphic List + recursion), t3_main_prints (minimal IO module), t4_count_zeros (prelude Eq Int + branched if). Reference solutions stay in canonical AILang form — param_modes is omitted when every parameter is the Implicit default, consistent with the existing examples/ corpus. 13/13 tests green: 5 lib unit (strip_locations) + 5 integration (strip_locations against verbatim captured fixtures) + 1 verify_references (drives every reference through the real ail+clang pipeline) + 1 mock_full_run (full eight-row sweep with mixed green-on-turn-2 + turn-limit cycles) + 1 budget_abort (synthetic budget exhaustion with budget_abort rows + run_status). Two implementer-phase repairs beyond the plan, both small and surfaced in the iter journal Concerns: 1. pipeline.rs renames the program file to `<module-name>.ail.json` between parse and check because `ail check` enforces filename stem == module name (compiler contract the plan did not anticipate). 2. main.rs fills synthetic budget_abort rows for tasks the outer loop never reached so scores.csv preserves the expected eight-row shape on budget exhaustion. One plan/text mismatch carried over: README "Total 8 passed" reflects the plan's four-suite count; actual sweep produces 13. Doc-fix candidate for cma.3. cma.3 (live IONOS run + DESIGN.md addendum + roadmap edits) remains out of scope.
126 lines
16 KiB
JSON
126 lines
16 KiB
JSON
{
|
|
"json": {
|
|
"t1_add_three": {
|
|
"1": {
|
|
"content": "{\n \"schema\": \"ailang/v0\",\n \"name\": \"t1_add_three\",\n \"imports\": [],\n \"defs\": [\n {\n \"kind\": \"fn\",\n \"name\": \"add_three\",\n \"type\": {\n \"k\": \"fn\",\n \"params\": [\n { \"k\": \"con\", \"name\": \"Int\" },\n { \"k\": \"con\", \"name\": \"Int\" },\n { \"k\": \"con\", \"name\": \"Int\" }\n ],\n \"ret\": { \"k\": \"con\", \"name\": \"Int\" },\n \"effects\": []\n },\n \"params\": [\"a\", \"b\", \"c\"],\n \"body\": {\n \"t\": \"app\",\n \"fn\": { \"t\": \"var\", \"name\": \"+\" },\n \"args\": [\n { \"t\": \"var\", \"name\": \"a\" },\n {\n \"t\": \"app\",\n \"fn\": { \"t\": \"var\", \"name\": \"+\" },\n \"args\": [\n { \"t\": \"var\", \"name\": \"b\" },\n { \"t\": \"var\", \"name\": \"c\" }\n ]\n }\n ]\n }\n },\n {\n \"kind\": \"fn\",\n \"name\": \"main\",\n \"type\": {\n \"k\": \"fn\",\n \"params\": [],\n \"ret\": { \"k\": \"con\", \"name\": \"Unit\" },\n \"effects\": [\"IO\"]\n },\n \"params\": [],\n \"body\": {\n \"t\": \"seq\",\n \"lhs\": {\n \"t\": \"do\",\n \"op\": \"io/print_int\",\n \"args\": [{\n \"t\": \"app\",\n \"fn\": { \"t\": \"var\", \"name\": \"add_three\" },\n \"args\": [\n { \"t\": \"lit\", \"lit\": { \"kind\": \"int\", \"value\": 1 } },\n { \"t\": \"lit\", \"lit\": { \"kind\": \"int\", \"value\": 2 } },\n { \"t\": \"lit\", \"lit\": { \"kind\": \"int\", \"value\": 3 } }\n ]\n }]\n },\n \"rhs\": {\n \"t\": \"do\",\n \"op\": \"io/print_int\",\n \"args\": [{\n \"t\": \"app\",\n \"fn\": { \"t\": \"var\", \"name\": \"add_three\" },\n \"args\": [\n { \"t\": \"lit\", \"lit\": { \"kind\": \"int\", \"value\": 10 } },\n { \"t\": \"lit\", \"lit\": { \"kind\": \"int\", \"value\": 20 } },\n { \"t\": \"lit\", \"lit\": { \"kind\": \"int\", \"value\": 30 } }\n ]\n }]\n }\n }\n }\n ]\n}\n",
|
|
"usage": {
|
|
"prompt_tokens": 1000,
|
|
"completion_tokens": 200,
|
|
"total_tokens": 1200
|
|
}
|
|
}
|
|
},
|
|
"t2_length": {
|
|
"1": {
|
|
"content": "{\n \"schema\": \"ailang/v0\",\n \"name\": \"t2_length\",\n \"imports\": [],\n \"defs\": [\n {\n \"kind\": \"type\",\n \"name\": \"List\",\n \"vars\": [\"a\"],\n \"ctors\": [\n { \"name\": \"Nil\", \"fields\": [] },\n {\n \"name\": \"Cons\",\n \"fields\": [\n { \"k\": \"var\", \"name\": \"a\" },\n { \"k\": \"con\", \"name\": \"List\", \"args\": [{ \"k\": \"var\", \"name\": \"a\" }] }\n ]\n }\n ]\n },\n {\n \"kind\": \"fn\",\n \"name\": \"length\",\n \"type\": {\n \"k\": \"forall\",\n \"vars\": [\"a\"],\n \"body\": {\n \"k\": \"fn\",\n \"params\": [\n { \"k\": \"con\", \"name\": \"List\", \"args\": [{ \"k\": \"var\", \"name\": \"a\" }] }\n ],\n \"ret\": { \"k\": \"con\", \"name\": \"Int\" },\n \"effects\": []\n }\n },\n \"params\": [\"xs\"],\n \"body\": {\n \"t\": \"match\",\n \"scrutinee\": { \"t\": \"var\", \"name\": \"xs\" },\n \"arms\": [\n {\n \"pat\": { \"p\": \"ctor\", \"ctor\": \"Nil\", \"fields\": [] },\n \"body\": { \"t\": \"lit\", \"lit\": { \"kind\": \"int\", \"value\": 0 } }\n },\n {\n \"pat\": {\n \"p\": \"ctor\",\n \"ctor\": \"Cons\",\n \"fields\": [\n { \"p\": \"wild\" },\n { \"p\": \"var\", \"name\": \"rest\" }\n ]\n },\n \"body\": {\n \"t\": \"app\",\n \"fn\": { \"t\": \"var\", \"name\": \"+\" },\n \"args\": [\n { \"t\": \"lit\", \"lit\": { \"kind\": \"int\", \"value\": 1 } },\n {\n \"t\": \"app\",\n \"fn\": { \"t\": \"var\", \"name\": \"length\" },\n \"args\": [{ \"t\": \"var\", \"name\": \"rest\" }]\n }\n ]\n }\n }\n ]\n }\n },\n {\n \"kind\": \"fn\",\n \"name\": \"main\",\n \"type\": {\n \"k\": \"fn\",\n \"params\": [],\n \"ret\": { \"k\": \"con\", \"name\": \"Unit\" },\n \"effects\": [\"IO\"]\n },\n \"params\": [],\n \"body\": {\n \"t\": \"seq\",\n \"lhs\": {\n \"t\": \"do\",\n \"op\": \"io/print_int\",\n \"args\": [{\n \"t\": \"app\",\n \"fn\": { \"t\": \"var\", \"name\": \"length\" },\n \"args\": [{ \"t\": \"ctor\", \"type\": \"List\", \"ctor\": \"Nil\", \"args\": [] }]\n }]\n },\n \"rhs\": {\n \"t\": \"do\",\n \"op\": \"io/print_int\",\n \"args\": [{\n \"t\": \"app\",\n \"fn\": { \"t\": \"var\", \"name\": \"length\" },\n \"args\": [{\n \"t\": \"ctor\", \"type\": \"List\", \"ctor\": \"Cons\",\n \"args\": [\n { \"t\": \"lit\", \"lit\": { \"kind\": \"int\", \"value\": 7 } },\n {\n \"t\": \"ctor\", \"type\": \"List\", \"ctor\": \"Cons\",\n \"args\": [\n { \"t\": \"lit\", \"lit\": { \"kind\": \"int\", \"value\": 8 } },\n {\n \"t\": \"ctor\", \"type\": \"List\", \"ctor\": \"Cons\",\n \"args\": [\n { \"t\": \"lit\", \"lit\": { \"kind\": \"int\", \"value\": 9 } },\n { \"t\": \"ctor\", \"type\": \"List\", \"ctor\": \"Nil\", \"args\": [] }\n ]\n }\n ]\n }\n ]\n }]\n }]\n }\n }\n }\n ]\n}\n",
|
|
"usage": {
|
|
"prompt_tokens": 1000,
|
|
"completion_tokens": 200,
|
|
"total_tokens": 1200
|
|
}
|
|
}
|
|
},
|
|
"t3_main_prints": {
|
|
"1": {
|
|
"content": "{\"schema\":\"ailang/v0\",\"name\":\"broken\",\"imports\":[],\"defs\":[]}",
|
|
"usage": {
|
|
"prompt_tokens": 1000,
|
|
"completion_tokens": 50,
|
|
"total_tokens": 1050
|
|
}
|
|
},
|
|
"2": {
|
|
"content": "{\n \"schema\": \"ailang/v0\",\n \"name\": \"t3_main_prints\",\n \"imports\": [],\n \"defs\": [\n {\n \"kind\": \"fn\",\n \"name\": \"main\",\n \"type\": {\n \"k\": \"fn\",\n \"params\": [],\n \"ret\": { \"k\": \"con\", \"name\": \"Unit\" },\n \"effects\": [\"IO\"]\n },\n \"params\": [],\n \"body\": {\n \"t\": \"seq\",\n \"lhs\": {\n \"t\": \"do\",\n \"op\": \"io/print_int\",\n \"args\": [{ \"t\": \"lit\", \"lit\": { \"kind\": \"int\", \"value\": 42 } }]\n },\n \"rhs\": {\n \"t\": \"do\",\n \"op\": \"io/print_int\",\n \"args\": [{ \"t\": \"lit\", \"lit\": { \"kind\": \"int\", \"value\": 1337 } }]\n }\n }\n }\n ]\n}\n",
|
|
"usage": {
|
|
"prompt_tokens": 1200,
|
|
"completion_tokens": 200,
|
|
"total_tokens": 1400
|
|
}
|
|
}
|
|
},
|
|
"t4_count_zeros": {
|
|
"1": {
|
|
"content": "{\n \"schema\": \"ailang/v0\",\n \"name\": \"t4_count_zeros\",\n \"imports\": [],\n \"defs\": [\n {\n \"kind\": \"type\",\n \"name\": \"List\",\n \"vars\": [\"a\"],\n \"ctors\": [\n { \"name\": \"Nil\", \"fields\": [] },\n {\n \"name\": \"Cons\",\n \"fields\": [\n { \"k\": \"var\", \"name\": \"a\" },\n { \"k\": \"con\", \"name\": \"List\", \"args\": [{ \"k\": \"var\", \"name\": \"a\" }] }\n ]\n }\n ]\n },\n {\n \"kind\": \"fn\",\n \"name\": \"count_zeros\",\n \"type\": {\n \"k\": \"fn\",\n \"params\": [\n { \"k\": \"con\", \"name\": \"List\", \"args\": [{ \"k\": \"con\", \"name\": \"Int\" }] }\n ],\n \"ret\": { \"k\": \"con\", \"name\": \"Int\" },\n \"effects\": []\n },\n \"params\": [\"xs\"],\n \"body\": {\n \"t\": \"match\",\n \"scrutinee\": { \"t\": \"var\", \"name\": \"xs\" },\n \"arms\": [\n {\n \"pat\": { \"p\": \"ctor\", \"ctor\": \"Nil\", \"fields\": [] },\n \"body\": { \"t\": \"lit\", \"lit\": { \"kind\": \"int\", \"value\": 0 } }\n },\n {\n \"pat\": {\n \"p\": \"ctor\",\n \"ctor\": \"Cons\",\n \"fields\": [\n { \"p\": \"var\", \"name\": \"h\" },\n { \"p\": \"var\", \"name\": \"rest\" }\n ]\n },\n \"body\": {\n \"t\": \"if\",\n \"cond\": {\n \"t\": \"app\",\n \"fn\": { \"t\": \"var\", \"name\": \"eq\" },\n \"args\": [\n { \"t\": \"var\", \"name\": \"h\" },\n { \"t\": \"lit\", \"lit\": { \"kind\": \"int\", \"value\": 0 } }\n ]\n },\n \"then\": {\n \"t\": \"app\",\n \"fn\": { \"t\": \"var\", \"name\": \"+\" },\n \"args\": [\n { \"t\": \"lit\", \"lit\": { \"kind\": \"int\", \"value\": 1 } },\n {\n \"t\": \"app\",\n \"fn\": { \"t\": \"var\", \"name\": \"count_zeros\" },\n \"args\": [{ \"t\": \"var\", \"name\": \"rest\" }]\n }\n ]\n },\n \"else\": {\n \"t\": \"app\",\n \"fn\": { \"t\": \"var\", \"name\": \"count_zeros\" },\n \"args\": [{ \"t\": \"var\", \"name\": \"rest\" }]\n }\n }\n }\n ]\n }\n },\n {\n \"kind\": \"fn\",\n \"name\": \"main\",\n \"type\": {\n \"k\": \"fn\",\n \"params\": [],\n \"ret\": { \"k\": \"con\", \"name\": \"Unit\" },\n \"effects\": [\"IO\"]\n },\n \"params\": [],\n \"body\": {\n \"t\": \"seq\",\n \"lhs\": {\n \"t\": \"do\",\n \"op\": \"io/print_int\",\n \"args\": [{\n \"t\": \"app\",\n \"fn\": { \"t\": \"var\", \"name\": \"count_zeros\" },\n \"args\": [{ \"t\": \"ctor\", \"type\": \"List\", \"ctor\": \"Nil\", \"args\": [] }]\n }]\n },\n \"rhs\": {\n \"t\": \"do\",\n \"op\": \"io/print_int\",\n \"args\": [{\n \"t\": \"app\",\n \"fn\": { \"t\": \"var\", \"name\": \"count_zeros\" },\n \"args\": [{\n \"t\": \"ctor\", \"type\": \"List\", \"ctor\": \"Cons\",\n \"args\": [\n { \"t\": \"lit\", \"lit\": { \"kind\": \"int\", \"value\": 0 } },\n {\n \"t\": \"ctor\", \"type\": \"List\", \"ctor\": \"Cons\",\n \"args\": [\n { \"t\": \"lit\", \"lit\": { \"kind\": \"int\", \"value\": 5 } },\n {\n \"t\": \"ctor\", \"type\": \"List\", \"ctor\": \"Cons\",\n \"args\": [\n { \"t\": \"lit\", \"lit\": { \"kind\": \"int\", \"value\": 0 } },\n {\n \"t\": \"ctor\", \"type\": \"List\", \"ctor\": \"Cons\",\n \"args\": [\n { \"t\": \"lit\", \"lit\": { \"kind\": \"int\", \"value\": 3 } },\n {\n \"t\": \"ctor\", \"type\": \"List\", \"ctor\": \"Cons\",\n \"args\": [\n { \"t\": \"lit\", \"lit\": { \"kind\": \"int\", \"value\": 0 } },\n { \"t\": \"ctor\", \"type\": \"List\", \"ctor\": \"Nil\", \"args\": [] }\n ]\n }\n ]\n }\n ]\n }\n ]\n }\n ]\n }]\n }]\n }\n }\n }\n ]\n}\n",
|
|
"usage": {
|
|
"prompt_tokens": 1000,
|
|
"completion_tokens": 200,
|
|
"total_tokens": 1200
|
|
}
|
|
}
|
|
}
|
|
},
|
|
"ailx": {
|
|
"t1_add_three": {
|
|
"1": {
|
|
"content": "(module garbage",
|
|
"usage": {
|
|
"prompt_tokens": 1000,
|
|
"completion_tokens": 50,
|
|
"total_tokens": 1050
|
|
}
|
|
},
|
|
"2": {
|
|
"content": "(module garbage",
|
|
"usage": {
|
|
"prompt_tokens": 1100,
|
|
"completion_tokens": 50,
|
|
"total_tokens": 1150
|
|
}
|
|
},
|
|
"3": {
|
|
"content": "(module garbage",
|
|
"usage": {
|
|
"prompt_tokens": 1200,
|
|
"completion_tokens": 50,
|
|
"total_tokens": 1250
|
|
}
|
|
},
|
|
"4": {
|
|
"content": "(module garbage",
|
|
"usage": {
|
|
"prompt_tokens": 1300,
|
|
"completion_tokens": 50,
|
|
"total_tokens": 1350
|
|
}
|
|
},
|
|
"5": {
|
|
"content": "(module garbage",
|
|
"usage": {
|
|
"prompt_tokens": 1400,
|
|
"completion_tokens": 50,
|
|
"total_tokens": 1450
|
|
}
|
|
}
|
|
},
|
|
"t2_length": {
|
|
"1": {
|
|
"content": "(module t2_length\n (data List (vars a)\n (ctor Nil)\n (ctor Cons a (con List a)))\n (fn length\n (type (forall (vars a) (fn-type (params (con List a)) (ret (con Int)))))\n (params xs)\n (body (match xs\n (case (pat-ctor Nil) 0)\n (case (pat-ctor Cons _ rest) (app + 1 (app length rest))))))\n (fn main\n (type (fn-type (params) (ret (con Unit)) (effects IO)))\n (params)\n (body (seq (do io/print_int (app length (term-ctor List Nil))) (do io/print_int (app length (term-ctor List Cons 7 (term-ctor List Cons 8 (term-ctor List Cons 9 (term-ctor List Nil))))))))))\n",
|
|
"usage": {
|
|
"prompt_tokens": 1000,
|
|
"completion_tokens": 200,
|
|
"total_tokens": 1200
|
|
}
|
|
}
|
|
},
|
|
"t3_main_prints": {
|
|
"1": {
|
|
"content": "(module t3_main_prints\n (fn main\n (type (fn-type (params) (ret (con Unit)) (effects IO)))\n (params)\n (body (seq (do io/print_int 42) (do io/print_int 1337)))))\n",
|
|
"usage": {
|
|
"prompt_tokens": 1000,
|
|
"completion_tokens": 200,
|
|
"total_tokens": 1200
|
|
}
|
|
}
|
|
},
|
|
"t4_count_zeros": {
|
|
"1": {
|
|
"content": "(module t4_count_zeros\n (data List (vars a)\n (ctor Nil)\n (ctor Cons a (con List a)))\n (fn count_zeros\n (type (fn-type (params (con List (con Int))) (ret (con Int))))\n (params xs)\n (body (match xs\n (case (pat-ctor Nil) 0)\n (case (pat-ctor Cons h rest) (if (app eq h 0) (app + 1 (app count_zeros rest)) (app count_zeros rest))))))\n (fn main\n (type (fn-type (params) (ret (con Unit)) (effects IO)))\n (params)\n (body (seq (do io/print_int (app count_zeros (term-ctor List Nil))) (do io/print_int (app count_zeros (term-ctor List Cons 0 (term-ctor List Cons 5 (term-ctor List Cons 0 (term-ctor List Cons 3 (term-ctor List Cons 0 (term-ctor List Nil))))))))))))\n",
|
|
"usage": {
|
|
"prompt_tokens": 1000,
|
|
"completion_tokens": 200,
|
|
"total_tokens": 1200
|
|
}
|
|
}
|
|
}
|
|
}
|
|
} |