Build a deployment arena: a standing proving ground where source-blind agents exercise the released binary #322

New Issue

2026-07-23T20:52:28+02:00

claude commented

2026-07-23 20:52:28 +02:00

Seeded by the user during the #316 cycle close (2026-07-23), prompted by the 2026-07-22 field evidence (#314): an external agent (Gemini) holding only the release binary had to reverse-engineer execution semantics and document schemas forensically. Had a standing arena existed, those inadequacies would have surfaced long before a real external consumer tripped over them.

Idea

A standing deployment arena: a proving ground where a source-blind consumer — an external LLM agent or a scripted downstream user — exercises the released, frozen artifact (the C13 deploy edge: the binary and nothing else, no repo checkout, no design ledger) through realistic end-to-end missions, on a cadence (per release / per milestone close), with friction recorded as structured findings that feed the tracker.

What distinguishes it from what exists:

The fieldtest skill is per-cycle, runs in the dev worktree, and its persona holds the repo's public docs (design ledger, READMEs, examples). The arena's knowledge floor is strictly the artifact itself — its --help surfaces, its refusal prose, its self-description (C29). Exactly the audience #319 ruled the binary serves.
The playground (ledger term, domain invariant 12) is the execution viewer / trace explorer — a different thing entirely; the arena deliberately takes a different name to avoid colliding that vocabulary.

Open design questions (brainstorm-tier)

Arena missions: curated task corpus vs. free-roam agent objectives; how failures/friction are captured as structured findings (spec shape? tracker issues?).
Driver: which agents run the arena (external models for genuine naivety vs. in-house agents with a hard knowledge firewall) — external LLM calls fall under the per-session consent rule for external services.
Cadence and gate character: informational (report-only, like the bench harness) vs. a release gate.
Where it lives: engine repo tooling vs. a separate arena repo consuming releases (engine/project separation, invariant 9).

Related: #314 (field evidence), #319 (audience ruling), #316/C29 (the self-description surfaces the arena would exercise), #315 (CLI help rendering), #267 (agent bootstrap card — a natural arena input).

> Seeded by the user during the #316 cycle close (2026-07-23), prompted by the 2026-07-22 field evidence (#314): an external agent (Gemini) holding only the release binary had to reverse-engineer execution semantics and document schemas forensically. Had a standing arena existed, those inadequacies would have surfaced long before a real external consumer tripped over them. ## Idea A standing **deployment arena**: a proving ground where a *source-blind* consumer — an external LLM agent or a scripted downstream user — exercises the **released, frozen artifact** (the C13 deploy edge: the binary and nothing else, no repo checkout, no design ledger) through realistic end-to-end missions, on a cadence (per release / per milestone close), with friction recorded as structured findings that feed the tracker. What distinguishes it from what exists: - The **fieldtest skill** is per-cycle, runs in the dev worktree, and its persona holds the repo's public docs (design ledger, READMEs, examples). The arena's knowledge floor is strictly the artifact itself — its `--help` surfaces, its refusal prose, its self-description (C29). Exactly the audience #319 ruled the binary serves. - The **playground** (ledger term, domain invariant 12) is the execution viewer / trace explorer — a different thing entirely; the arena deliberately takes a different name to avoid colliding that vocabulary. ## Open design questions (brainstorm-tier) - Arena missions: curated task corpus vs. free-roam agent objectives; how failures/friction are captured as structured findings (spec shape? tracker issues?). - Driver: which agents run the arena (external models for genuine naivety vs. in-house agents with a hard knowledge firewall) — external LLM calls fall under the per-session consent rule for external services. - Cadence and gate character: informational (report-only, like the bench harness) vs. a release gate. - Where it lives: engine repo tooling vs. a separate arena repo consuming releases (engine/project separation, invariant 9). Related: #314 (field evidence), #319 (audience ruling), #316/C29 (the self-description surfaces the arena would exercise), #315 (CLI help rendering), #267 (agent bootstrap card — a natural arena input).

claude added the idea label 2026-07-23 20:52:28 +02:00

claude commented

2026-07-23 20:56:56 +02:00

Backlog placement (2026-07-23 triage scheme)

The 2026-07-23 backlog triage organized the open backlog into six sequenced milestones (self-description; data-authorability boundary; stderr surface honesty; safe-to-embed; trace store; measurement reachable) plus a deliberately-unbundled set. This issue joins the unbundled parked-ideas bucket (beside #263 / #274 / #279 / #149 / #159), not the Self-description milestone, on these grounds:

The arena is the verification apparatus for the self-description promise, not part of its surface work — bundling it would widen that milestone's promise past its close gate.
Preconditions before this becomes actionable: the Self-description milestone shipped (#315 help surfaces and #267 bootstrap card are the arena's natural inputs — an arena run before they exist would measure their absence, which the milestone already tracks); the #319 direction call (the arena's mission corpus exercises whichever verb surface survives it); and a user direction call on the driver question (genuinely external models fall under the external-service consent rule).
Near-term stand-in until then: the Self-description milestone's closing fieldtest runs with a binary-only knowledge floor — no repo docs, only the built artifact and its own help — per the user's direction of 2026-07-23. The arena institutionalizes exactly that test form on a standing cadence.

Status: parked idea — prioritization stays the user's call (idea label).

## Backlog placement (2026-07-23 triage scheme) The 2026-07-23 backlog triage organized the open backlog into six sequenced milestones (self-description; data-authorability boundary; stderr surface honesty; safe-to-embed; trace store; measurement reachable) plus a deliberately-unbundled set. This issue joins the unbundled **parked-ideas** bucket (beside #263 / #274 / #279 / #149 / #159), not the Self-description milestone, on these grounds: - The arena is the *verification apparatus* for the self-description promise, not part of its surface work — bundling it would widen that milestone's promise past its close gate. - Preconditions before this becomes actionable: the Self-description milestone shipped (#315 help surfaces and #267 bootstrap card are the arena's natural inputs — an arena run before they exist would measure their absence, which the milestone already tracks); the #319 direction call (the arena's mission corpus exercises whichever verb surface survives it); and a user direction call on the driver question (genuinely external models fall under the external-service consent rule). - Near-term stand-in until then: the Self-description milestone's closing fieldtest runs with a binary-only knowledge floor — no repo docs, only the built artifact and its own help — per the user's direction of 2026-07-23. The arena institutionalizes exactly that test form on a standing cadence. Status: parked idea — prioritization stays the user's call (idea label).

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Brummel/Aura#322