21cfccf8464591b2c3cba0894e019ae08eb6cbc7
7 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Brummel
|
21cfccf846 |
iter bench-harness-recalibration.1 (DONE 4/4): drop 6 latency entries + recapture baselines
One-artefact iter on `bench/baseline.json`. Closes Gitea #15 + #16 as documented in the spec commit `a97aaeb`. Changes: - `note` field rewritten forward-looking: drops the retired-JOURNAL workflow reference (Journal system deleted in |
||
|
Brummel
|
28ab56af7e |
audit embedding-abi-m5: milestone close — RATIFY gc_rss trio (evidenced) + DRIFT (one doc-honesty tidy); P2 bench noise causally exonerated, baseline pristine for it
Architect: Invariant 1 PASS (zero data-server in any compiler
crate/runtime; ail-embed its own workspace root); M3 frozen-layout
SSOT unmoved (the lone rc.c change
|
||
|
Brummel
|
aea3758742 |
tidy: 21'g — resolve drift after 21'-arc close
Three drift items from ailang-architect, plus one false-positive
surfaced during verification:
1. DESIGN.md silent on closure-pair 4.14x finding (21'b).
Decision-10 ratified: "Workload scope of the 1.3x target"
paragraph scopes the retirement gate to linear/tree/poly-ADT
workloads; closure-pair carve-out documented as
representational cost (closure cell + env struct = 2 allocs
per step) until a slab/pool answer ships.
2. bench/compile_check.py corpus drift. Three fixtures added
(bench_compute_intsum, bench_compute_collatz,
bench_list_sum_explicit), re-baselined. Now 12 fixtures x
2 ops = 24 compile-time metrics.
3. baseline.json convention not codified. Note field gains
"max-of-distribution gets wider band than percentile"
convention discovered in 21'd.
4. (verification finding) bench_compute_intsum cross_lang
tolerances at 15%/12% fire on subprocess-spawn jitter for
sub-millisecond fixtures. Widened to 35% across all five
intsum metrics; convention recorded in baseline_cross_lang
note field (sub-ms fixtures need looser bands).
All three bench gates re-run sequentially after edits:
bench/check.py — 63 metrics; 63 stable
bench/compile_check.py — 24 metrics; 24 stable
bench/cross_lang.py — 25 metrics; 25 stable
Total: 112 metrics under regression coverage, all green.
288 tests passing, 3 ignored. No Rust changes.
|
||
|
Brummel
|
75f7fda788 |
bench: 21'f — explicit-mode pair, full alloc+dec vs malloc+free
Closes the apples-to-apples gap from 21'e. Adds: - examples/bench_list_sum_explicit.ailx — same algorithm and sizes as bench_list_sum, fully (borrow)/(own)/(drop-iterative) annotated so codegen emits proper inc/dec instrumentation. - bench/reference/list_sum_explicit_free.c — same algorithm with explicit free() walking the chain after sum. The full alloc+dec vs malloc+free comparison reveals two non- trivial conclusions: 1. AILang's full RC pipeline is only 26% slower than glibc malloc+free on this workload (rc/c = 1.26x). The implicit- mode comparison's 1.42x was misleading — it counted neither pipeline's free path. The fair ratio is 1.26x, materially better than the previous read. 2. RC's dec is cheaper per cell than glibc free(). AILang dec-tax: ~3 ns/cell. C free-tax: ~5.5 ns/cell. Plausible cause: ailang_rc_dec operates on a known-shape cell with a fixed-offset refcount and a static per-type drop fn — no free-list bucketing, no header introspection, no global lock. bump's advantage expresses fully: bench_list_sum_explicit.bump/c = 0.42x means AILang at bump is 2.4x faster than C malloc+free. Sets a useful upper bound on a slab/pool RC allocator's potential. The 21'-family arc — bench-regression infrastructure — is now substantively complete: 21'a (bench/check.py), 21'b (corpus widening), 21'c (compile_check.py), 21'd (pure-compute fixtures + harness hardening), 21'e (cross-language hand-C), 21'f (explicit apples-to-apples). 63 runtime metrics + 18 compile metrics + 25 cross-lang metrics under regression coverage. Any future iter that regresses any axis beyond tolerance gets caught at the next family close. Remaining queue is back to substantive language work — Family 21 (typeclasses / polymorphic ADTs at runtime / pattern-binding generalisation) is now an orchestrator-level fork that needs direct user input. |
||
|
Brummel
|
5a4a6de031 |
bench: 21'd — pure-compute fixtures + harness hardening
Closes the third corpus blind spot (heap-allocation-only) by adding two fixtures with no allocation pressure: bench_compute_ intsum (tail-recursive integer accumulator) and bench_compute_ collatz (Collatz step-counter, branchy). Surprise on intsum: 50M-iteration loop runs in 1ms wall under all three allocators. LLVM's induction-variable analysis applies the closed-form triangular-sum reduction to AILang's IR — a positive codegen finding (the IR composes with LLVM's optimizer at the same level a hand-C loop would) but it makes intsum useless as a runtime regression bench. Excluded from run.sh's fixtures array; kept in examples/ as reference and as a future cross-language comparison anchor. Collatz survives optimization (data-dependent control flow). At 56ms wall, gc/bump/rc all within 2% — the canonical "pure-compute is allocator-invariant" data point this fixture is meant to prove. If a future codegen change leaks an allocation into the inner loop, the 1.00x / 1.02x ratios diverge visibly. Two infrastructure fixes the new fixtures forced: - 6-decimal precision in run.sh's Python timing helper and median averager (was 3-decimal; sub-ms times rounded to 0.000 and crashed the ratio awk with Division durch Null). - Zero-guard in the ratio awk (defensive even with the precision bump, since LLVM-eliminated workloads can still hit zero). Latency baseline: implicit_at_rc.max_us tolerance 25% -> 30%. Three captures today (477 / 456 / 609 µs) show natural run-to-run dispersion wider than the original tolerance accounts for. Not a softening to dodge regression — the original baseline was the first capture; a fairer tolerance across natural max-of-1000- samples width is what the harness needed from the start. Baseline file: 47 -> 55 metrics. 21'e (cross-language reference, clang -O2 hand-C ratios) is the natural next dispatch. |
||
|
Brummel
|
07bff24527 |
bench: 21'b — closure-pair + HOF/poly fixtures, 47-metric baseline
Two new throughput fixtures targeting blind spots in the 21'a
corpus:
- bench_closure_chain exercises the build_pair_drop_fn codegen
path (the 18c.4 doubled-braces trigger). Each iteration of
run_loop allocates a {thunk, env} closure pair via the
let-rec-name-as-value escape route. Sizes 10k / 100k / 500k.
rc/bump = 4.14x — materially worse than the 2.91x / 2.59x of
the linear/tree fixtures, exposing that closure work pays the
RC alloc tax twice (pair + env-struct).
- bench_hof_pipeline exercises poly-ADT instantiation and
indirect dispatch via fold_with_fn over List<a>. Sizes 100k /
1M / 3M elements. Ratios essentially match bench_list_sum,
confirming the 13b static-template-plus-ctor-inline design
has zero measurable overhead at this scale.
Baseline file extends from 31 to 47 metrics. The two new fixtures
build clean under all three allocators; the rc-arm build exercises
the per-type drop fn for the closure-pair, providing a tripwire
for any future 18c.4-class IR malformedness.
JOURNAL records both surprises (4.14x closure tax, ~zero HOF/poly
overhead) and explicitly notes the dispersion observation on
explicit_at_rc.p99 — three captures today (357.5 / 294.6 / 251.5)
confirm wide run-to-run variance on that fixture. Methodology
upgrade (n>=10 captures or tighter fixture) deferred to 21'c.
bench/run.sh fixtures array updated. bench/check.py needed no
changes — its parser handles the wider table by metric name.
|
||
|
Brummel
|
a1b0ad5723 |
bench: regression harness — baseline.json + check.py
Closes the structural gap between bench/run.sh (one-shot capture into JOURNAL) and a continuous tripwire. baseline.json records 31 metrics (16 throughput, 15 latency) with per-metric one-sided tolerances tuned to absorb run-to-run noise on a quiet developer machine; check.py spawns run.sh, parses both the throughput pipe-table and the per-arm latency stanzas, diffs against the baseline, prints a per-metric report, and exits non-zero on any regression beyond tolerance. User-facing flags: --from-file, --stdin, --baseline, and --update-baseline (re-run + overwrite baseline.json after intentional improvements). Validation: captured the baseline, then re-ran end-to-end. All 31 metrics within tolerance. The harness also caught a single-capture explicit-rc p99 spike (357.5 us) that was first read as drift vs. yesterday's 18g.tidy.fu2 numbers but came in at 294.6 us on the follow-up run — exactly the kind of noise the tolerance band is there to absorb. Without 21'a we would have either chased a phantom or buried the signal; with it, single noisy runs are data points, not verdicts. Tidy-iter discipline addition (run check.py at every family close alongside the architect drift report) is recommended in JOURNAL but not enacted in this iter — that's an orchestrator-level update to CLAUDE.md. |