Generalized per-cell fault isolation: a failing campaign cell is a recorded outcome, never a global abort #272

New Issue

2026-07-14T11:04:47+02:00

claude commented

2026-07-14 11:04:47 +02:00

Problem

One cell's failure aborts the whole campaign: the execute loop in aura-campaign propagates any MemberFault globally (run_members(cell, ...)?, crates/aura-campaign/src/exec.rs). Concrete incident (recorded on #264): a 22-instrument campaign aborted ~6.7 minutes in on no data for instrument Copper in window [...], discarding ~36 already-computed cells. The same global abort fires for ANY member fault — archive data gap, network/source failure yielding no data, bind failure, a genuine panic — not just gaps. What is missing is generalized cell-fault handling, not a gap-specific guard.

Decision (user, 2026-07-14)

The user weighed three directions — (1) a coverage preflight that refuses the run, (2) run to completion and report compromised results, (3) make the run "cope" with empty windows — and decided for (2), generalized:

No specialized coverage preflight that catches only the gap case. Rejected because it special-cases one fault among many and protects computed work only by preventing computation (one instrument's hole would block the whole healthy matrix).
The run executes; a cell may fail for any reason. The failure becomes a first-class recorded per-cell outcome, and the campaign result is reported as compromised where affected.
Hard invariant: a failing cell never drags down independent cells. The rest of the matrix completes and persists. (This holds today for nothing — even the incident's 36 healthy computed cells were lost.)
The rest of the system reacts adequately: selection stages, aggregation, summary, and downstream consumers must define their behaviour in the presence of failed cells rather than assume completeness.
Direction (3) is rejected: the only mechanisms that make an empty window runnable are window clipping/shifting or data synthesis, which silently fabricate cross-instrument comparability or prices.

Behaviour sketch (for the spec)

Per-cell outcome, not error propagation: ok / failed { fault } — no-data is one fault kind among others, not a special path. A panic at the member boundary is contained: the cell records it, the campaign continues.
Aggregates declare holes: walkforward pooling, generalize's worst-case floor, and MC stages name fail/skip counts per instrument (e.g. Copper: 7/10 folds, 3 failed (no data)) — never silently pool unequal member sets.
Selection defines its reaction to a nominee/gate set containing failed cells.
Summary + exit semantics: the run completes with a visible fault report; the exit-code convention for "completed with failures" is a spec decision.
Partial-coverage honesty (sub-point): a cell whose window is only partially covered (data begins after the window start; interior gap months bridged by warm state) currently runs silently on a shifted effective evaluation period. The cell record should carry a coverage annotation so "compromised" is defined for the partial case too, not only the empty one.

Non-goals

Window clipping/shifting or data synthesis (rejected above).
A coverage preflight stage inside run. The inventory verbs from #264 (aura data list / aura data coverage) remain the authoring-time tool and are unaffected.

## Problem One cell's failure aborts the whole campaign: the execute loop in `aura-campaign` propagates any `MemberFault` globally (`run_members(cell, ...)?`, crates/aura-campaign/src/exec.rs). Concrete incident (recorded on #264): a 22-instrument campaign aborted ~6.7 minutes in on `no data for instrument Copper in window [...]`, discarding ~36 already-computed cells. The same global abort fires for ANY member fault — archive data gap, network/source failure yielding no data, bind failure, a genuine panic — not just gaps. What is missing is generalized cell-fault handling, not a gap-specific guard. ## Decision (user, 2026-07-14) The user weighed three directions — (1) a coverage preflight that refuses the run, (2) run to completion and report compromised results, (3) make the run "cope" with empty windows — and decided for (2), generalized: - **No specialized coverage preflight** that catches only the gap case. Rejected because it special-cases one fault among many and protects computed work only by preventing computation (one instrument's hole would block the whole healthy matrix). - **The run executes; a cell may fail for any reason.** The failure becomes a first-class recorded per-cell outcome, and the campaign result is reported as compromised where affected. - **Hard invariant: a failing cell never drags down independent cells.** The rest of the matrix completes and persists. (This holds today for nothing — even the incident's 36 healthy computed cells were lost.) - **The rest of the system reacts adequately**: selection stages, aggregation, summary, and downstream consumers must define their behaviour in the presence of failed cells rather than assume completeness. - Direction (3) is rejected: the only mechanisms that make an empty window runnable are window clipping/shifting or data synthesis, which silently fabricate cross-instrument comparability or prices. ## Behaviour sketch (for the spec) 1. **Per-cell outcome, not error propagation**: `ok` / `failed { fault }` — no-data is one fault kind among others, not a special path. A panic at the member boundary is contained: the cell records it, the campaign continues. 2. **Aggregates declare holes**: walkforward pooling, generalize's worst-case floor, and MC stages name fail/skip counts per instrument (e.g. `Copper: 7/10 folds, 3 failed (no data)`) — never silently pool unequal member sets. 3. **Selection defines its reaction** to a nominee/gate set containing failed cells. 4. **Summary + exit semantics**: the run completes with a visible fault report; the exit-code convention for "completed with failures" is a spec decision. 5. **Partial-coverage honesty (sub-point)**: a cell whose window is only partially covered (data begins after the window start; interior gap months bridged by warm state) currently runs silently on a shifted effective evaluation period. The cell record should carry a coverage annotation so "compromised" is defined for the partial case too, not only the empty one. ## Non-goals - Window clipping/shifting or data synthesis (rejected above). - A coverage preflight stage inside `run`. The inventory verbs from #264 (`aura data list` / `aura data coverage`) remain the authoring-time tool and are unaffected.

claude added the feature label 2026-07-14 11:04:47 +02:00

claude commented

2026-07-14 13:16:26 +02:00

Implementation-fork decisions (derived) — realizing the decided direction (2)

The decided direction (a failing cell is a recorded outcome, never a global abort) leaves several implementation forks; they resolve as follows. Basis for each: derived, rationale inline.

Fork: containment granularity → cell-level for the sweep stage, fold-level for the walk_forward stage. Rationale: the containment unit is the smallest unit whose loss does not structurally distort the surviving statistics — a member fault inside a sweep grid leaves a hole the selection semantics (argmax, and especially plateau neighborhoods) silently mis-rank, so the sweep stage fails its whole cell; walk-forward folds are independent time windows whose pooled aggregate already carries a count, so a fold fault is recorded (window_faults on the stage realization), the surviving folds pool, and the summary names the ratio ("7/10 folds, 3 failed (no data)" — the issue's own example). A cell fails only when a stage can deliver nothing at all (sweep fault, or zero surviving folds).
Fork: which faults are containable → ExecFault::Member and ExecFault::Window become per-cell recorded outcomes; everything else stays a campaign-wide refusal. Rationale: member/window faults are data-and-cell-local by construction, while the doc-shape faults fire in preflight before any cell runs and a Registry fault means persistence itself is broken — recording a cell outcome into a broken registry is self-contradictory.
Fork: record representation → additive optional fields on the persisted shapes (CellRealization.fault: Option<CellFault>, StageRealization.window_faults, both skip-serialized when absent) with a small closed CellFaultKind enum (no_data | bind | run | panic | window) plus prose detail. Rationale: existing campaign_runs.jsonl lines stay readable unchanged (serde defaults), the kind tag is a closed vocabulary aggregations can count on, and prose carries the specifics — no record-format fork, no migration.
Fork: panic containment site → catch_unwind (with AssertUnwindSafe) inside the member-run closures, recorded as fault kind panic. Rationale: the sweep dispatches members on scoped worker threads whose join re-raises panics into execute (today: whole-process abort, exit 101), so the only place a panic is still attributable to its member is inside the closure; the runner is re-used across cells afterwards, which AssertUnwindSafe makes an explicit, documented judgement (the CLI runner holds read-only state).
Fork: exit-code semantics → a run that executed but recorded ≥1 failed cell exits with a new documented code 3 ("completed with failed cells"); 0 stays clean-complete, 1 stays refused-before-running, 2 stays usage. Rationale: a compromised run must not read as clean to &&-chained scripts (rules out 0), yet it is materially different from a refusal — the run record and every healthy cell's evidence exist and are usable (rules out 1); the zero-survivor precedent ("a null result is a valid result", exit 0) does not apply because a fault is not a null result. This holds even when every cell failed: the run ran and persisted its record.
Fork: cross-cell reactions → generalize treats a failed cell exactly like a gate-emptied one (no nominee → listed in missing), with the fault distinguishable via the cell record; the single-cell walkforward/mc sugar verbs surface the recorded fault as their own message and exit 3 (their one cell failing leaves nothing to summarize); the generalize sugar's "every cell must nominate" debug assertion is retired (it was only ever true in a fault-free world).
Fork: partial-coverage annotation → a per-cell, runner-supplied coverage note (MemberRunner gains a default-None window_coverage hook the executor calls once per cell; the CLI runner derives effective bounds and in-window gap months from the existing archive_extent/list_m1_months primitives) persisted as an optional field on the cell realization. Rationale: coverage is a property of (instrument, window) — one lookup per cell, not per member — and the #264 primitives already compute exactly the needed facts; a default-None hook leaves every synthetic runner untouched.

Consequences confirmed against the tree before speccing: execute already accumulates all cell outcomes and appends the one campaign-run record after the loop, so containment localizes to run_cell/run_members/run_walk_forward_stage returning fault-annotated outcomes instead of Err; the generalize stage already declares holes (winners/missing partition) and gate-emptied truncation is the in-place precedent for "record and continue". The current global-abort behaviour is pinned by execute_member_fault_aborts_with_lowest_index (flips to record-and-continue), the two wf fault tests, and a family of ~15 CLI e2e assertions (exit 1 + "no data …" prose) that move to the new exit-3 register where the run now completes.

## Implementation-fork decisions (derived) — realizing the decided direction (2) The decided direction (a failing cell is a recorded outcome, never a global abort) leaves several implementation forks; they resolve as follows. Basis for each: derived, rationale inline. - **Fork: containment granularity** → **cell-level for the sweep stage, fold-level for the walk_forward stage.** Rationale: the containment unit is the smallest unit whose loss does not structurally distort the surviving statistics — a member fault inside a sweep grid leaves a hole the selection semantics (argmax, and especially plateau neighborhoods) silently mis-rank, so the sweep stage fails its whole cell; walk-forward folds are independent time windows whose pooled aggregate already carries a count, so a fold fault is recorded (`window_faults` on the stage realization), the surviving folds pool, and the summary names the ratio ("7/10 folds, 3 failed (no data)" — the issue's own example). A cell fails only when a stage can deliver nothing at all (sweep fault, or zero surviving folds). - **Fork: which faults are containable** → **`ExecFault::Member` and `ExecFault::Window` become per-cell recorded outcomes; everything else stays a campaign-wide refusal.** Rationale: member/window faults are data-and-cell-local by construction, while the doc-shape faults fire in preflight before any cell runs and a `Registry` fault means persistence itself is broken — recording a cell outcome into a broken registry is self-contradictory. - **Fork: record representation** → **additive optional fields on the persisted shapes** (`CellRealization.fault: Option<CellFault>`, `StageRealization.window_faults`, both skip-serialized when absent) with a small closed `CellFaultKind` enum (`no_data | bind | run | panic | window`) plus prose detail. Rationale: existing `campaign_runs.jsonl` lines stay readable unchanged (serde defaults), the kind tag is a closed vocabulary aggregations can count on, and prose carries the specifics — no record-format fork, no migration. - **Fork: panic containment site** → **`catch_unwind` (with `AssertUnwindSafe`) inside the member-run closures, recorded as fault kind `panic`.** Rationale: the sweep dispatches members on scoped worker threads whose join re-raises panics into `execute` (today: whole-process abort, exit 101), so the only place a panic is still attributable to its member is inside the closure; the runner is re-used across cells afterwards, which `AssertUnwindSafe` makes an explicit, documented judgement (the CLI runner holds read-only state). - **Fork: exit-code semantics** → **a run that executed but recorded ≥1 failed cell exits with a new documented code 3 ("completed with failed cells"); 0 stays clean-complete, 1 stays refused-before-running, 2 stays usage.** Rationale: a compromised run must not read as clean to `&&`-chained scripts (rules out 0), yet it is materially different from a refusal — the run record and every healthy cell's evidence exist and are usable (rules out 1); the zero-survivor precedent ("a null result is a valid result", exit 0) does not apply because a fault is not a null result. This holds even when every cell failed: the run ran and persisted its record. - **Fork: cross-cell reactions** → **generalize treats a failed cell exactly like a gate-emptied one (no nominee → listed in `missing`), with the fault distinguishable via the cell record; the single-cell walkforward/mc sugar verbs surface the recorded fault as their own message and exit 3 (their one cell failing leaves nothing to summarize); the generalize sugar's "every cell must nominate" debug assertion is retired** (it was only ever true in a fault-free world). - **Fork: partial-coverage annotation** → **a per-cell, runner-supplied coverage note** (`MemberRunner` gains a default-`None` `window_coverage` hook the executor calls once per cell; the CLI runner derives effective bounds and in-window gap months from the existing `archive_extent`/`list_m1_months` primitives) persisted as an optional field on the cell realization. Rationale: coverage is a property of (instrument, window) — one lookup per cell, not per member — and the #264 primitives already compute exactly the needed facts; a default-`None` hook leaves every synthetic runner untouched. Consequences confirmed against the tree before speccing: `execute` already accumulates all cell outcomes and appends the one campaign-run record after the loop, so containment localizes to `run_cell`/`run_members`/`run_walk_forward_stage` returning fault-annotated outcomes instead of `Err`; the generalize stage already declares holes (`winners`/`missing` partition) and gate-emptied truncation is the in-place precedent for "record and continue". The current global-abort behaviour is pinned by `execute_member_fault_aborts_with_lowest_index` (flips to record-and-continue), the two wf fault tests, and a family of ~15 CLI e2e assertions (exit 1 + "no data …" prose) that move to the new exit-3 register where the run now completes.

claude commented

2026-07-14 14:55:59 +02:00

Mechanism refinements settled before planning (derived)

Plan-recon surfaced three mechanism facts the spec's decided direction implies but did not spell out; recorded here so the implementing plan is unambiguous. Basis for each: derived.

Fold-closure fault partition. The walk_forward fold closure captures three fault sources into one per-window vec: IS-sweep member faults, the IS winner-selection error (always an ExecFault::Registry from optimize/optimize_deflated/optimize_plateau), and OOS member faults. Containment partitions at drain time: a Registry fault aborts the whole run (persistence/optimizer machinery is broken — it stays global, exactly as Registry is excluded from contain), while Member/Window faults become per-fold WindowFaults. The surviving folds pool; zero surviving folds fail the cell.
Placeholder folds filtered by the broker sentinel. A faulted fold yields placeholder_window_run, whose report carries manifest.broker == "faulted-member-placeholder". The stage filters placeholders out of result before walkforward_member_reports, so the persisted WalkForward family holds only real OOS reports and the pooled series / MC bootstrap operate on survivors; the filtered-out count is the window_faults len named in the summary ratio.
Exit-code 3 is uniform across every campaign entry point. The failed-cell count is threaded up (present_campaign and the walkforward/mc/generalize sugar runners return it), and every dispatch site maps "≥1 failed cell" to exit 3 — including run_sweep_sugar (which shares present_campaign): any campaign that completes with a failed cell exits 3, whether run via aura campaign run or a dissolved verb. The other campaign sub-verbs (validate/introspect/register/runs) keep 0/1. One consequence pinned for the plan: campaign_run_valid_tap_reaches_the_member_data_seam requests persist_taps: ["equity"]; under containment the run now reaches the trace tail and its no-nominee branch prints "no nominee; no traces persisted", so that test moves wholesale to the exit-3 register (its !contains("traces persisted") negation is replaced, not merely re-signed).

## Mechanism refinements settled before planning (derived) Plan-recon surfaced three mechanism facts the spec's decided direction implies but did not spell out; recorded here so the implementing plan is unambiguous. Basis for each: derived. - **Fold-closure fault partition.** The walk_forward fold closure captures three fault sources into one per-window vec: IS-sweep member faults, the IS winner-selection error (always an `ExecFault::Registry` from `optimize`/`optimize_deflated`/`optimize_plateau`), and OOS member faults. Containment partitions at drain time: a `Registry` fault aborts the whole run (persistence/optimizer machinery is broken — it stays global, exactly as `Registry` is excluded from `contain`), while `Member`/`Window` faults become per-fold `WindowFault`s. The surviving folds pool; zero surviving folds fail the cell. - **Placeholder folds filtered by the broker sentinel.** A faulted fold yields `placeholder_window_run`, whose report carries `manifest.broker == "faulted-member-placeholder"`. The stage filters placeholders out of `result` before `walkforward_member_reports`, so the persisted `WalkForward` family holds only real OOS reports and the pooled series / MC bootstrap operate on survivors; the filtered-out count is the `window_faults` len named in the summary ratio. - **Exit-code 3 is uniform across every campaign entry point.** The failed-cell count is threaded up (`present_campaign` and the walkforward/mc/generalize sugar runners return it), and every dispatch site maps "≥1 failed cell" to exit 3 — including `run_sweep_sugar` (which shares `present_campaign`): any campaign that completes with a failed cell exits 3, whether run via `aura campaign run` or a dissolved verb. The other `campaign` sub-verbs (validate/introspect/register/runs) keep 0/1. One consequence pinned for the plan: `campaign_run_valid_tap_reaches_the_member_data_seam` requests `persist_taps: ["equity"]`; under containment the run now reaches the trace tail and its no-nominee branch prints "no nominee; no traces persisted", so that test moves wholesale to the exit-3 register (its `!contains("traces persisted")` negation is replaced, not merely re-signed).

Brummel referenced this issue from a commit

2026-07-14 17:01:47 +02:00

feat(campaign,registry,cli): per-cell fault isolation — a failed cell is recorded, never a global abort

Brummel referenced this issue from a commit

2026-07-14 17:01:47 +02:00

audit: cycle-close tidy for the #256/#272 latecomer block

claude referenced this issue

2026-07-14 17:02:19 +02:00

Tracker sweep + executor robustness: net-R chain, archive inventory, std::grid, per-cell fault isolation (12 issues) #270

claude closed this issue

2026-07-14 17:13:01 +02:00

claude referenced this issue from a commit