Experiment: full pipeline with implement-skills disabled — quality vs token tracking #35
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: Brummel/Skills#35
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Context
The
implementskills (theimplementskill + itsimplement-loop/compiler-driven-editWorkflow executors) are currently disabled by the owner. Implementation inside a cycle runs via orchestrator-dispatched phase agents instead (animplementeragent on a planner-produced plan, the orchestrator verifying build/suite/clippy itself, plus one independentquality-reviewerdiff review). The rest of the pipeline runs as shipped:specify(with an ad-hocspec-skepticbias-breaker and thegrounding-checkhard-gate),planner(withplan-recon),audit(witharchitect+ report-only bench), and the per-cyclefieldtest.What we want to achieve
Maximum quality per token on a Fable main-loop session (Fable quota is half that of the smaller models, and Fable burns more tokens per step — so all mechanical fan-out must run on sonnet/opus/haiku; Fable subagents only for the four ratified gates). The experiment measures whether the full skill pipeline justifies its consumption against the skills-less baseline established in aura cycle #315 (implement without skills + one independent diff review): that baseline shipped 3 errors, 2 of them prose-truth errors, all found only by external review — the quality bar the pipeline must beat while staying affordable. Per-cycle consumption data is persisted orchestrator-side (memory file
aura-pipeline-eckdaten) so runs stay comparable.Outcome we're steering toward: either re-enable/replace the implement-skills with something leaner, keep orchestrator-driven implementation permanently, or restructure stages (see findings below) — decided on data, not vibes.
Data point 1 — aura cycle #310 (2026-07-24, full pipeline minus implement-skills)
Scope:
aura run --tap TAP=FOLDfold-subscription selector (CLI wiring + tests + ledger amendment; a deliberately small, bounded cycle).Total ≈ 691k subagent tokens (sonnet ~336k / 49 %, opus ~208k / 30 %, fable gates ~147k / 21 %), excluding the Fable main-loop orchestration itself. Wall-clock ~3.5 h incl. builds.
Quality return — every stage caught something real:
persist_tapspresentation channel vs declared taps) plus the C25 projection-layering question — both changed the shipped cut.TapPlan::subscribeis&mut self, not builder-self) before planning.graph introspect --foldsdiscovery surface misdescribes the actual fold-registry roster — a self-description bug no inside-out review caught.Efficiency findings (to act on):
Next steps
Data point 2 — harvest-sweep (2026-07-24, aura): batch mode, NOT the A-side
Not a full-pipeline cycle (no specify/planner/grounding-check — seven settled tracker-issue bodies served as the source, decisions minuted as issue comments), so this is a third mode beside the two the experiment compares. Recorded for the economics:
~570k subagent tokens for 7 issue closures (~81k/closure) vs. data point 1's 691k for 1 closure. Takeaways: (1) batching amortizes the fixed gates (one review, one architect, one fieldtest over N closures) — the dominant lever for small settled work; (2) the independent opus diff review is not removable: it caught the only substantive error of the cycle (a wrong timestamp-semantics ratification), which came from the orchestrator+fieldtest consensus, not from the implementers; (3) sequential sonnet batches of 2-3 items with grep-pointer briefs kept each dispatch ~110k and avoided cross-build interference. Still owed for the experiment: the A-side (a comparable skills-less implementation cycle with stage accounting).
Data point 3 — aura #317 "composites data-authorable" (2026-07-24, B-side: full pipeline, implement-skills OFF)
Heavy design cycle (new op-script verb end-to-end + a mid-cycle behaviour
ratification; ~1,580 insertions, 7 commits). Same mode as data point 1
(#310): specify/planner/audit skills on, implementation via directly
dispatched implementer agents.
Total: ~1.92M subagent tokens — sonnet 1.08M (56%), opus 441k (23%),
fable gates 399k (21%). vs #310: 2.8× tokens for ~10× diff — the pipeline's
fixed gate cost amortizes over cycle size.
Findings for the efficiency review:
#310's Explore+plan-recon 250k with ~40% overlap.
could make: grounding-check #1 falsified two spec claims about current
code (one made the implementation SIMPLER — the planned enum already
existed), and the source-blind fieldtest refuted a sentence of the
cycle's OWN mid-cycle ratification (empirics vs orchestrator prose).
second-largest block after implementation; it caught a silent-invariant
defect (gang de-fuse) and a docs overclaim. Delta-re-review at 102k is
arguably the squeeze point (could be sonnet for small deltas?).
(1 productive BLOCK, 1 evidence-driven amendment). Smaller spec slices
would cut re-dispatches.
Still owed: the A-side (skills-less implementation cycle of comparable,
SMALL size — #317 was too big to be the comparator; a ripe single like
aura#273 or #225 is the right candidate).
Data point 4 — aura #271 (B-side: full pipeline, implement-skills off)
Heavy design cycle (typed construction args; 7 commits, 55 files, +1755/−196 — same class as #317). Stages: consolidated recon (sonnet 189k) → spec-skeptic (opus 85k) → grounding-check (fable 84k, PASS first attempt) → 2 implementer batches (sonnet 197k + 374k) → quality review (opus 117k, approved, 0 Important/Minor — no repair loop) → architect (fable 134k) → fieldtest (opus 126k, 0 bugs). Total ~1.30M (sonnet 58 % / opus 25 % / fable 17 %).
Read against #317 (1.92M, same cycle class): the entire 32 % saving is the absent iteration — no 2nd/3rd grounding run (~175k), no repair+re-review (~255k). The causal candidate is the early opus skeptic run BEFORE the fable gate: it caught the two spec defects (an unlisted deliberate pin-flip; missing shipped call sites) that would otherwise have been gate BLOCKs. Pattern worth keeping: cheap adversarial pass before the expensive hard gate, consolidated recon serving specify+planner.
Still owed: the A-side (skills-less implementation cycle on a comparably SMALL issue — aura #273/#225 remain the candidates).
Data point 5 — aura #328 (axis-namespace reconciliation, 2026-07-25; full pipeline, implement-skills off)
Heavy multi-route cycle (one vocabulary, four intake routes, 3 crates, ~90-literal test conversion). Stages: consolidated recon (sonnet 170k) -> opus skeptic 88k (BLOCK, the cycle's best catch: a third wrapped surface the spec missed + an equality branch that would have made the refusal ladder dead code — sank the v1 spec before the fable gate) -> fable grounding 86k (PASS first attempt on v2) -> three sonnet batches 452k/186k/97k -> opus review 161k (changes_requested: untested refusal arm + prose duplicate) -> sonnet repair 107k -> fable architect 125k (drift_found: 2 high — a fourth route silently aliasing, manifests still minting wrapped) -> sonnet tidy 271k (found a latent bug en route) -> opus fieldtest 102k (0 bugs / 0 friction). Total ~1.85M (sonnet 69%, opus 19%, fable 11%).
Contrast with data point 4 (#271, 1.30M, zero post-implement iteration): here the skeptic prevented a bad design but could not prevent post-implement iteration on route completeness — repair+tidy = 20% of spend, all of it enforced by review/architect finding routes the spec never enumerated (skeptic found route 3, architect found route 4). Lesson for the playbook: when a cycle touches a vocabulary fanned out over N verbs/routes, the skeptic brief should demand an explicit enumeration of ALL intake/emission routes of that vocabulary — that single instruction would likely have halved both loops. Batch-size lesson from #271 confirmed again: the atomic big batch (452k) dominates cost.
Data point 6 — aura #331 blueprint name op (2026-07-25, full pipeline without implement-skills)
Subagent total: ~1.78M tokens — sonnet ~1.10M (62%), opus ~369k (21%), fable gates ~314k (18%). Diff: 5 commits, ~1,200 changed lines incl. a 48-file fieldtest corpus. Every gate caught something real; final suite 99 targets green.
Efficiency verdict: the iteration tail dominated — repair + route-fix + tidy + fieldtest-bugfix = ~612k (34% of the cycle), and all four loops were the SAME defect class: one more authored-file intake route outside the gate. The skeptic correctly found the class before the gate (envelope route #2), but the spec then pinned only the two routes found so far instead of demanding a class-complete enumeration up front; review, architect, and fieldtest each then paid a full loop to find routes 3-4, 5-7, and 8. Lesson (extends the data-point-5 lesson from intake vocabularies to gate placement): when a spec introduces a gate over a ROUTE CLASS, the spec itself must carry the exhaustive route enumeration (grep-derived, all verbs), and the first implement batch must gate the class, not the named instances. A second observation: the skeptic-before-gate pattern does not pre-empt grounding-class BLOCKs (untested current-behaviour claims) — gate #1 still blocked on two of those; that is the gate doing its distinct job, not overlap.
Correction to data point 6's lesson (user review, 2026-07-25)
The user's post-cycle review caught that the DP6 lesson ("the spec must carry the class-complete route enumeration") is only half right — and its naive application actively cost tokens in #331. Held against the roadmap, the route class "every authored-file intake" splits: the sugar verbs (sweep/walkforward/mc flag surfaces, sweep --list-axes) are scheduled for retirement by the very next queue item (aura #319), so the class-completeness loops that gated them (~300k tokens: the architect-driven tidy and the fieldtest B1 fix) were consistency work on a dying surface, plus a sweep-out debt in the retirement cycle. The spec's post-skeptic boundary (op intake + the canonical envelope dispatch: register/introspect/viewer) was already the roadmap-stable set.
Corrected lesson: when a spec introduces a gate/invariant over a route class, (1) enumerate the class exhaustively in the spec (DP6 stands), AND (2) check the class boundary against the roadmap — routes on a surface the queue already retires get a documented known-until-retirement exemption instead of enforcement, unless they carry a live hazard (the run-route filesystem escape did; the read-only listing routes did not). "Class-complete on a dying surface" is its own failure mode, symmetric to the under-enumeration DP5/DP6 warn about. The closing gates (architect, fieldtest) enforce present-tense consistency by design — the roadmap check is the orchestrator's job at the fix-vs-ratify fork, not theirs.
Data point 7 — the A-side baseline (cycle Aura#273, 2026-07-25)
First A-side run (owner's order, 2026-07-25): no skill stages at all. The orchestrator read the issue thread itself, did a ~6-grep in-loop recon, minuted three derived design forks on the reference issue up front (
descriptiondeferred to data-server#4, broader refusal prose, empty-archive stderr notice), wrote one precise implementation brief, and verified the diff and all gates itself. Exactly one independent diff review.Subagent total ≈ 1.17M tokens (sonnet 71 %, opus 29 %, fable 0 %); order→commit wall clock ≈ 1 h. Measurement caveat: A numbers are summed from the agent transcripts (input + cache-creation + output; cache reads excluded), B numbers came from task-notification usage blocks — believed to be the same basis, not verified. The orchestrator's own extra work (recon, brief writing, verification, minutes) ran on the main-loop model and is not counted — it is the A-mode's hidden cost.
Quality yield: gates first-pass green (independently re-run by the orchestrator: 0 failed workspace-wide, clippy
-D warningsclean, all 5 bench fingerprints OK). The single review caught exactly 1 Important — the sidecar-only OR-branch of the new verb's known-check was untested; a real catch, a regression there would have passed every existing test — plus 1 nit. Zero prose-truth errors (contrast the skills-less #315 predecessor: 3 errors, 2 of them prose-truth, all caught only externally). No audit stage, no fieldtest: their absence is untested risk, not proven savings.Control-condition caveats: the carrier task was deliberately fork-free (that isolates the mechanics/ceremony variable — it is the design, not a flaw), its scope is smaller than the B cycles', and this is one data point.
Preparation for the skills-rework session
On record (owner's decision, 2026-07-25): the skills are to be reshaped according to this experiment's results in a dedicated session; Aura#319 (sugar retirement) is designated the reformed skillset's first hard test.
What the two series support, as the rework agenda:
Method note for future data points: A-side token extraction = per-assistant-message usage summed over the agent's
.jsonltranscript.Efficiency findings 1-3 implemented (commit
fe5ca8f)All three "Efficiency findings (to act on)" from this issue are now implemented in the plugin:
docs/pipeline.md§ Cycle recon. One fullplan-recondispatch (normally specify Step 1, via a new pre-specsources/recon_scopecarrier form) serves both the spec's concrete code shapes and the plan's file-map; planner Step 2 is reuse-first (freshness / quotability / coverage conditions) with narrowly-bounded delta dispatches (new Delta dispatch contract, <=500-token output); broad Explore-type grounding sweeps are banned in every mode,/bosstriage included. Expected effect vs. the #310 data point: one full recon instead of Explore 151k + plan-recon 99k.tier: probe— 1-2 examples on exactly that axis, ~200-word report, explicitmodel: sonnetdispatch override. Probe is a tier, not a skip (the #310 evidence — the one-flag cycle's fieldtest found a real self-description bug — is cited as the reason the signal must survive). The milestone fieldtest always runs full tier on the frontmatter model; the milestone-close gate's >=2-example floor stays unconditional.grounding-checkitself is unchanged (it remains the autonomous signature under/boss).Owner decisions recorded (asked and answered 2026-07-25):
fieldtest/SKILL.mdandagent-template.md§ model rule 2.refs #35); further per-cycle data points and the implement-skills decision (re-enable / leaner replacement / permanent orchestrator-driven) remain pending on data. The implement skills were deliberately NOT removed.Review dissent, recorded for the next data point: the pre-commit design review recommended against the sonnet override for the probe tier (rule 2 places field evidence at opus; the issue-#30 precedent has a deterministic measurement + verify gate this lacks; the fieldtester's source isolation is prose-enforced only). The owner's decision stands; the guard is the one-axis scope, the >=1-example floor, orchestrator triage of every finding — and this experiment itself: the first probe-tier data point should be checked for quality loss, and the override is a one-line rollback if it shows.
Process note: design pre-reviewed (opus plan review), diff adversarially verified by a 3-lens review workflow (consistency / operability / cross-reference); all confirmed findings were folded in before commit.
Probe-tier model override retracted (owner decision, 2026-07-25 — commit
8ad3961)The owner reviewed the dissent recorded in the previous comment and agreed with it: the
model: sonnetprobe override is retracted. The probe tier now runs on the fieldtester's frontmatter opus/xhigh, exactly like the full tier; its economy comes from scope alone (1-2 examples on the single delta axis, ~200-word report).agent-template.md§ model rule 2 again covers the fieldtester unconditionally, with a note that a probe narrows scope, never the model.For the ongoing measurement this simplifies the next data point: probe-tier savings to expect are example count and report size only, not model tier.