Stop leaking per-run test sandboxes into /tmp #258

New Issue

2026-07-13T16:42:59+02:00

claude commented

2026-07-13 16:42:59 +02:00

On 2026-07-13 the host's /tmp — a 16 GiB tmpfs, so the leak consumes RAM — hit 100% usage. The occupant was the Aura test suite: over 50,000 leftover aura-* sandbox directories accumulated between 2026-07-11 and 2026-07-13.

Measured breakdown at the time (du -x --max-depth=1 /tmp, grouped by name pattern):

pattern	count	total size
`aura-235-*`	159	9.9 GiB
`aura-nodes-resolve-*`	76	4.7 GiB
`aura-campaign-exec-*`	7,887	small each
`aura-registry-fam-*`	5,967	small each
`aura-campaign-wf-*`	1,776	small each
(plus ~35k further small `aura-*` dirs)

The two heavy groups each contain a scaffolded project plus a node crate whose built target/ directory weighs ~67 MB per sandbox (e.g. aura-235-<pid>/knob-lab-nodes/target).

Mechanism. The sandbox helpers never clean up after a run. The only remove_dir_all is a pre-create wipe keyed to the current PID, which can never match a previous run's directory:

// crates/aura-cli/tests/project_nodes.rs:6-11
fn temp_cwd(tag: &str) -> std::path::PathBuf {
    let d = std::env::temp_dir().join(format!("aura-nodes-{tag}-{}", std::process::id()));
    let _ = std::fs::remove_dir_all(&d); // only ever hits this run's own path
    std::fs::create_dir_all(&d).unwrap();
    d
}

Every test-binary invocation has a fresh PID, so each run creates a new set of directories and deletes none from earlier runs; there is no Drop guard or at-exit cleanup. The same hand-rolled pattern recurs across the suite, among others:

crates/aura-cli/tests/project_sweep_campaign.rs:70 — aura-235-<pid>, the 9.9 GiB group; a OnceLock scaffold that runs cargo build on a node crate, leaving one ~67 MB sandbox per binary run
crates/aura-cli/tests/project_nodes.rs:7, project_new.rs:18, graph_construct.rs:385, cli_broken_pipe.rs:18, research_docs.rs:15
test code inside crate sources: crates/aura-cli/src/scaffold.rs:491, crates/aura-cli/src/project.rs:580 (and siblings in both files), crates/aura-cli/src/verb_sugar.rs:898, crates/aura-campaign/src/lib.rs:887, crates/aura-registry/tests/campaign_run_store_e2e.rs:21

At inspection time no process held an open handle on any leaked directory (lsof +D /tmp: 0 hits). The leftovers were removed manually on 2026-07-13, freeing ~15.5 GiB; the suite recreates them on every run, so the tmpfs will fill again.

Candidate fixes (one of):

Replace the hand-rolled env::temp_dir().join(...) helpers with RAII temp dirs (e.g. tempfile::TempDir) so sandboxes are removed on drop, panics included.
Key sandbox names deterministically (no PID) so each run's pre-create wipe reclaims the previous run's directory, bounding the leak to one copy per pattern. The OnceLock builder sandboxes would be covered by the same change.

On 2026-07-13 the host's `/tmp` — a 16 GiB tmpfs, so the leak consumes RAM — hit 100% usage. The occupant was the Aura test suite: over 50,000 leftover `aura-*` sandbox directories accumulated between 2026-07-11 and 2026-07-13. Measured breakdown at the time (`du -x --max-depth=1 /tmp`, grouped by name pattern): | pattern | count | total size | |---|---|---| | `aura-235-*` | 159 | 9.9 GiB | | `aura-nodes-resolve-*` | 76 | 4.7 GiB | | `aura-campaign-exec-*` | 7,887 | small each | | `aura-registry-fam-*` | 5,967 | small each | | `aura-campaign-wf-*` | 1,776 | small each | | (plus ~35k further small `aura-*` dirs) | | | The two heavy groups each contain a scaffolded project plus a node crate whose built `target/` directory weighs ~67 MB per sandbox (e.g. `aura-235-<pid>/knob-lab-nodes/target`). **Mechanism.** The sandbox helpers never clean up after a run. The only `remove_dir_all` is a pre-create wipe keyed to the *current* PID, which can never match a previous run's directory: ```rust // crates/aura-cli/tests/project_nodes.rs:6-11 fn temp_cwd(tag: &str) -> std::path::PathBuf { let d = std::env::temp_dir().join(format!("aura-nodes-{tag}-{}", std::process::id())); let _ = std::fs::remove_dir_all(&d); // only ever hits this run's own path std::fs::create_dir_all(&d).unwrap(); d } ``` Every test-binary invocation has a fresh PID, so each run creates a new set of directories and deletes none from earlier runs; there is no `Drop` guard or at-exit cleanup. The same hand-rolled pattern recurs across the suite, among others: - `crates/aura-cli/tests/project_sweep_campaign.rs:70` — `aura-235-<pid>`, the 9.9 GiB group; a `OnceLock` scaffold that runs `cargo build` on a node crate, leaving one ~67 MB sandbox per binary run - `crates/aura-cli/tests/project_nodes.rs:7`, `project_new.rs:18`, `graph_construct.rs:385`, `cli_broken_pipe.rs:18`, `research_docs.rs:15` - test code inside crate sources: `crates/aura-cli/src/scaffold.rs:491`, `crates/aura-cli/src/project.rs:580` (and siblings in both files), `crates/aura-cli/src/verb_sugar.rs:898`, `crates/aura-campaign/src/lib.rs:887`, `crates/aura-registry/tests/campaign_run_store_e2e.rs:21` At inspection time no process held an open handle on any leaked directory (`lsof +D /tmp`: 0 hits). The leftovers were removed manually on 2026-07-13, freeing ~15.5 GiB; the suite recreates them on every run, so the tmpfs will fill again. Candidate fixes (one of): - [ ] Replace the hand-rolled `env::temp_dir().join(...)` helpers with RAII temp dirs (e.g. `tempfile::TempDir`) so sandboxes are removed on drop, panics included. - [ ] Key sandbox names deterministically (no PID) so each run's pre-create wipe reclaims the previous run's directory, bounding the leak to one copy per pattern. The `OnceLock` builder sandboxes would be covered by the same change.

claude added the bug label 2026-07-13 16:42:59 +02:00

claude commented

2026-07-13 19:40:57 +02:00

Fix-pattern decision for the sandbox leak (decision log, recorded 2026-07-13).

The two candidate fixes in the issue body resolve as follows: the repair is the fixed, tag-keyed, PID-free sandbox name under the build tree's tmp anchor, keeping each site's existing pre-create remove_dir_all, which then genuinely reclaims the previous run's directory.

Integration-test and bench targets use env!("CARGO_TARGET_TMPDIR") — the exact remedy commit 84e1075 already established for the knob-lab scaffold in project_sweep_campaign.rs; the fix extends that ratified pattern to the remaining sites instead of introducing a second mechanism.
Src-internal unit-test helpers (scaffold.rs, project.rs, verb_sugar.rs, aura-campaign/src/lib.rs) do not get CARGO_TARGET_TMPDIR (cargo sets it only for integration/bench targets); they derive the same anchor from env!("CARGO_MANIFEST_DIR") (the workspace target/tmp).

Why this over the RAII alternative (tempfile::TempDir, the issue's first candidate): fixed reclaimable names bound the residue to at most one directory per site on disk rather than the tmpfs, are collision-free across concurrent checkouts/worktrees because target/ is per-checkout, and cover the process-lifetime OnceLock scaffolds that RAII structurally cannot (statics never drop) — while TempDir would add a new dependency and, under env::temp_dir(), still strand kill-leftovers on the tmpfs.

Load-bearing constraint for the fix: tag uniqueness per call site. With PID-free fixed names, two sites sharing a tag inside one workspace would collide; every helper's tag must stay unique within its target's namespace.

A RED regression test pinning the property at one representative site (temp_cwd in crates/aura-cli/tests/project_nodes.rs: the returned path embeds no PID and does not live under env::temp_dir()) precedes the fix in the history.

Fix-pattern decision for the sandbox leak (decision log, recorded 2026-07-13). The two candidate fixes in the issue body resolve as follows: the repair is the **fixed, tag-keyed, PID-free sandbox name under the build tree's tmp anchor**, keeping each site's existing pre-create `remove_dir_all`, which then genuinely reclaims the previous run's directory. - Integration-test and bench targets use `env!("CARGO_TARGET_TMPDIR")` — the exact remedy commit 84e1075 already established for the knob-lab scaffold in `project_sweep_campaign.rs`; the fix extends that ratified pattern to the remaining sites instead of introducing a second mechanism. - Src-internal unit-test helpers (`scaffold.rs`, `project.rs`, `verb_sugar.rs`, `aura-campaign/src/lib.rs`) do not get `CARGO_TARGET_TMPDIR` (cargo sets it only for integration/bench targets); they derive the same anchor from `env!("CARGO_MANIFEST_DIR")` (the workspace `target/tmp`). Why this over the RAII alternative (`tempfile::TempDir`, the issue's first candidate): fixed reclaimable names bound the residue to at most one directory per site **on disk** rather than the tmpfs, are collision-free across concurrent checkouts/worktrees because `target/` is per-checkout, and cover the process-lifetime `OnceLock` scaffolds that RAII structurally cannot (statics never drop) — while `TempDir` would add a new dependency and, under `env::temp_dir()`, still strand kill-leftovers on the tmpfs. Load-bearing constraint for the fix: **tag uniqueness per call site**. With PID-free fixed names, two sites sharing a tag inside one workspace would collide; every helper's tag must stay unique within its target's namespace. A RED regression test pinning the property at one representative site (`temp_cwd` in `crates/aura-cli/tests/project_nodes.rs`: the returned path embeds no PID and does not live under `env::temp_dir()`) precedes the fix in the history.

claude commented

2026-07-13 20:15:56 +02:00

Ruling on the one pattern exception the fix surfaced (decision log, recorded 2026-07-13).

The sweep applied the decided pattern (issues/258#issuecomment-3525) at every helper site except one: crates/aura-cli/tests/project_new.rs cannot anchor under CARGO_TARGET_TMPDIR, because two of its tests (new_outside_a_work_tree_initializes_git, new_outside_a_work_tree_leaves_a_resolvable_head) require a base directory genuinely detached from any git work tree, and the build tree's target/ sits inside the checkout's own work tree — the per-site prescription was empirically false there (independently verified during review).

Decision: that helper stays under env::temp_dir() with a fixed, tag-keyed, PID-free name plus a per-checkout hash discriminator (a hash of CARGO_MANIFEST_DIR in the name). Rationale: a bare fixed name under the shared /tmp would let two concurrently running checkouts/worktrees race on the same path (wipe-vs-use), which the PID key previously prevented; the checkout discriminator restores cross-checkout isolation while keeping the name deterministic per checkout, so each run still reclaims its predecessor and the residue stays bounded at one small directory per checkout.

Also recorded: pre-fix PID-keyed directories already stranded in /tmp are not swept by the fix (nothing can reclaim a foreign PID's name safely while other checkouts may be mid-run); they age out only by manual cleanup. New accumulation stops once this fix is in every actively tested checkout.

Shipped as 1ccce9a (RED regression test precedes it as 05565ee); full workspace suite green, clippy clean.

Ruling on the one pattern exception the fix surfaced (decision log, recorded 2026-07-13). The sweep applied the decided pattern (`issues/258#issuecomment-3525`) at every helper site except one: `crates/aura-cli/tests/project_new.rs` cannot anchor under `CARGO_TARGET_TMPDIR`, because two of its tests (`new_outside_a_work_tree_initializes_git`, `new_outside_a_work_tree_leaves_a_resolvable_head`) require a base directory genuinely detached from any git work tree, and the build tree's `target/` sits inside the checkout's own work tree — the per-site prescription was empirically false there (independently verified during review). Decision: that helper stays under `env::temp_dir()` with a fixed, tag-keyed, PID-free name **plus a per-checkout hash discriminator** (a hash of `CARGO_MANIFEST_DIR` in the name). Rationale: a bare fixed name under the shared `/tmp` would let two concurrently running checkouts/worktrees race on the same path (wipe-vs-use), which the PID key previously prevented; the checkout discriminator restores cross-checkout isolation while keeping the name deterministic per checkout, so each run still reclaims its predecessor and the residue stays bounded at one small directory per checkout. Also recorded: pre-fix PID-keyed directories already stranded in `/tmp` are not swept by the fix (nothing can reclaim a foreign PID's name safely while other checkouts may be mid-run); they age out only by manual cleanup. New accumulation stops once this fix is in every actively tested checkout. Shipped as `1ccce9a` (RED regression test precedes it as `05565ee`); full workspace suite green, clippy clean.

Brummel referenced this issue from a commit

2026-07-14 03:03:18 +02:00

audit: batch tidy — exhaustive bind-error prose + the carry unit note

claude referenced this issue

2026-07-14 03:03:44 +02:00

Tracker sweep + executor robustness: net-R chain, archive inventory, std::grid, per-cell fault isolation (12 issues) #270

claude referenced this issue from a commit

2026-07-14 17:13:00 +02:00

test: red for #258 — temp_cwd sandbox leaks one dir per run into /tmp

claude referenced this issue from a commit

2026-07-14 17:13:00 +02:00

fix(test): anchor all test sandboxes on reclaimable pid-free names