From d2d0d0f4fd6e1780d37dbffdd4c8a2f0c8f957de Mon Sep 17 00:00:00 2001 From: Brummel Date: Thu, 2 Jul 2026 15:26:58 +0200 Subject: [PATCH] spec: 0102 project-as-crate load boundary (boss-signed) Grounding-check PASS as the autonomous signature: 7 load-bearing current-behaviour assumptions each ratified by a named green test; greenfield surfaces (descriptor, loader, Aura.toml) excluded per contract. Design decisions and provenance: the decision-log on the milestone reference issue. refs #180 --- docs/specs/0102-project-load-boundary.md | 427 +++++++++++++++++++++++ 1 file changed, 427 insertions(+) create mode 100644 docs/specs/0102-project-load-boundary.md diff --git a/docs/specs/0102-project-load-boundary.md b/docs/specs/0102-project-load-boundary.md new file mode 100644 index 0000000..de53fa1 --- /dev/null +++ b/docs/specs/0102-project-load-boundary.md @@ -0,0 +1,427 @@ +# The project-as-crate load boundary — Design Spec + +**Date:** 2026-07-02 +**Status:** Draft — awaiting sign-off (boss run: grounding-check gate) +**Authors:** orchestrator + Claude +**Cycle:** 0102 — first implementation cycle of milestone "Project environment — the project-as-crate authoring loop" (#180; the full fork-decision log with grounds is the decision-log comment on #180) + +## Goal + +Make a research project a loadable external Rust crate: an `aura` invocation +run inside a project directory discovers the project root (`Aura.toml`), +locates and loads the project's cdylib, verifies ABI compatibility +(refuse-don't-guess), merges the project's node vocabulary with +`std_vocabulary`, and runs every blueprint verb against the merged vocabulary — +bit-deterministically, with the loaded dylib's identity recorded in the run +manifest. Outside a project directory, behaviour stays byte-identical to +today. The `aura new` scaffolder is the next cycle, not this one. + +## Architecture + +Two new surfaces and one threading change: + +1. **`aura-core::project` (new module + build.rs)** — the shared descriptor + contract both sides compile: a `#[repr(C)]` `ProjectDescriptor` with a + **C-ABI tier** (magic, descriptor version, rustc/aura-core version stamps, + namespace) that the host reads **before** trusting anything, and a + **Rust-ABI tier** (vocabulary resolver fn, enumerable type-id list) touched + only after the stamps match. An `aura_project!` macro generates the single + exported `AURA_PROJECT` static. Because the project build recompiles + aura-core under the project's own toolchain, the stamps baked into + aura-core's constants are automatically per-side-correct. +2. **`aura-cli::project` (new module)** — discovery (cargo-style walk-up to + `Aura.toml`), artifact location via `cargo metadata` (subprocess + + `serde_json`, no new crate), `libloading` load-and-hold (the `Library` is + leaked; never unloaded), the two-tier stamp gate, the vocabulary charter + checks, and the merged resolver. +3. **Per-invocation project context** — `main()` resolves an + `Option` once (None outside a project) and threads it into the + `dispatch_*` functions; the ~14 inline `&|t| std_vocabulary(t)` resolver + sites, the ~10 `default_registry()` sites, and the inline + `data_server::DEFAULT_DATA_PATH` sites read from it. Outside a project the + context is `None` and every path collapses to the current literal. + +Design constraints (decided on #180, restated where load-bearing): + +- **Stamp before trust.** The stamp cannot certify the channel it rides on; + the C tier uses only C-compatible field types (integers, ptr+len string + slices) and is validated before any Rust-ABI field is dereferenced. +- **Vocabulary charter.** Project type-ids MUST be namespace-prefixed + (`::Type`, separator `::`); std ids stay bare. The loader refuses (exit + 1): a missing/empty namespace, a listed id without the `::` prefix, a + duplicate id within the merged set, and a listed id the resolver does not + resolve (list↔resolver cross-check). The default node name strips the + namespace, so `::` never enters the param-path address space. +- **Zero-arg inheritance.** Project vocabularies resolve zero-argument + builders only, exactly as `std_vocabulary` scopes itself; construction-arg + widening remains the queued additive extension (#156). +- **Aura.toml is paths only.** `[paths] data`, `[paths] runs` — both optional. + No instrument/pip metadata (the recorded geometry sidecar stays the only + source — C15 realization, #124), no default symbol/window/broker, no dylib + path, no namespace (the descriptor is the namespace's single source; one + refinement over the #180 log's item 7, same anti-drift rationale). +- **Author builds, aura loads.** No cargo-build orchestration inside aura; + a missing artifact refuses with a build hint. Per-invocation load of the + freshest build IS the v1 "hot-reload" (C13 reading to be recorded in the + ledger, with the one-shot-process scope boundary of load-and-hold). + +## Concrete code shapes + +### The user-facing project (the acceptance-criterion evidence) + +A project crate, in its own directory outside this repo (fixture twin: +`crates/aura-cli/tests/fixtures/demo-project/`): + +```toml +# ger40-lab/Cargo.toml +[package] +name = "ger40-lab" +version = "0.1.0" +edition = "2024" + +[lib] +crate-type = ["cdylib"] + +[dependencies] +aura-core = { path = "../aura/crates/aura-core" } # git dep in real projects + +[workspace] # standalone workspace root (docs/project-layout.md note) +``` + +```toml +# ger40-lab/Aura.toml — static project context only (C17); every key optional +[paths] +data = "/mnt/tickdata/Pepperstone" # data archive root; default: data-server default +runs = "runs" # registry root, relative to the project root; default: "runs" +``` + +```rust +// ger40-lab/src/lib.rs +use aura_core::PrimitiveBuilder; +mod third_candle; // an ordinary Node + PrimitiveBuilder factory, as in aura-std + +fn vocabulary(type_id: &str) -> Option { + match type_id { + "ger40::ThirdCandle" => Some(third_candle::builder()), + _ => None, + } +} + +fn type_ids() -> &'static [&'static str] { + &["ger40::ThirdCandle"] +} + +aura_core::aura_project! { + namespace: "ger40", + vocabulary: vocabulary, + type_ids: type_ids, +} +``` + +```console +$ cd ger40-lab && cargo build +$ aura run blueprints/third-candle.json # blueprint's "type": "ger40::ThirdCandle" +$ aura sweep blueprints/third-candle.json --list-axes +$ aura graph introspect --vocabulary # lists std AND ger40::* ids +``` + +### aura-core: descriptor + stamps (new `src/project.rs`, new `build.rs`) + +```rust +// aura-core/build.rs (new; the workspace's second build.rs after aura-cli's) +fn main() { + let rustc = std::env::var("RUSTC").unwrap_or_else(|_| "rustc".into()); + let out = std::process::Command::new(rustc).arg("--version").output() + .expect("rustc --version"); + println!("cargo:rustc-env=AURA_RUSTC_VERSION={}", + String::from_utf8_lossy(&out.stdout).trim()); +} +``` + +```rust +// aura-core/src/project.rs (new) +pub const AURA_DESCRIPTOR_MAGIC: u64 = 0x4155_5241_5052_4F4A; // "AURAPROJ" +pub const AURA_DESCRIPTOR_VERSION: u32 = 1; +pub const AURA_PROJECT_SYMBOL: &[u8] = b"AURA_PROJECT\0"; + +/// Baked at aura-core compile time — i.e. per consuming build, which is what +/// makes the host/dylib comparison meaningful (each side recompiles aura-core). +pub const RUSTC_VERSION: &str = env!("AURA_RUSTC_VERSION"); +pub const CORE_VERSION: &str = env!("CARGO_PKG_VERSION"); + +/// C-ABI string slice: ptr + len over 'static UTF-8 (no NUL convention needed). +#[repr(C)] +pub struct StrSlice { pub ptr: *const u8, pub len: usize } +impl StrSlice { + pub const fn new(s: &'static str) -> Self { + Self { ptr: s.as_ptr(), len: s.len() } + } +} + +#[repr(C)] +pub struct ProjectDescriptor { + // ---- C tier: validated BEFORE any Rust-ABI field is touched ---- + pub magic: u64, + pub descriptor_version: u32, + pub rustc_version: StrSlice, + pub aura_core_version: StrSlice, + pub namespace: StrSlice, + // ---- Rust tier: only after both stamps match ---- + pub vocabulary: fn(&str) -> Option, + pub type_ids: fn() -> &'static [&'static str], +} +unsafe impl Sync for ProjectDescriptor {} + +#[macro_export] +macro_rules! aura_project { + (namespace: $ns:literal, vocabulary: $vocab:path, type_ids: $ids:path $(,)?) => { + #[unsafe(no_mangle)] + pub static AURA_PROJECT: $crate::project::ProjectDescriptor = + $crate::project::ProjectDescriptor { + magic: $crate::project::AURA_DESCRIPTOR_MAGIC, + descriptor_version: $crate::project::AURA_DESCRIPTOR_VERSION, + rustc_version: $crate::project::StrSlice::new($crate::project::RUSTC_VERSION), + aura_core_version: $crate::project::StrSlice::new($crate::project::CORE_VERSION), + namespace: $crate::project::StrSlice::new($ns), + vocabulary: $vocab, + type_ids: $ids, + }; + }; +} +``` + +### aura-core: default node name strips the namespace + +Before (`crates/aura-core/src/node.rs:143-147`): + +```rust +pub fn node_name(&self) -> String { + self.instance_name.clone() + .unwrap_or_else(|| self.name.to_ascii_lowercase()) +} +``` + +After (std ids carry no `::`, so existing behaviour is byte-identical): + +```rust +pub fn node_name(&self) -> String { + self.instance_name.clone().unwrap_or_else(|| { + let bare = self.name.rsplit("::").next().unwrap_or(self.name); + bare.to_ascii_lowercase() + }) +} +``` + +Consequence: param paths (built from node names, `blueprint.rs` `collect_params`) +never contain `::`; the member_key sanitizer (`main.rs:215-219`, +charset `[A-Za-z0-9._-]`) is reached only by names, so no new +filesystem-portability class opens. An explicit `.named("a::b")` sanitizes to +`a__b` as any out-of-charset byte does today. + +### aura-cli: loader (new `src/project.rs`) + +```rust +pub struct AuraToml { pub data: Option, pub runs: Option } + +pub struct ProjectEnv { + pub root: PathBuf, // directory containing Aura.toml + pub toml: AuraToml, + pub namespace: String, + pub dylib_path: PathBuf, + pub dylib_sha256: String, // provenance (sha2 is already a dependency) + pub commit: Option, // best-effort: git -C rev-parse HEAD (+ -dirty) + /// fn pointer copied out of the descriptor — plain `fn`, and valid for + /// 'static because the Library is leaked (load-and-hold). + resolver: fn(&str) -> Option, + type_id_list: &'static [&'static str], +} + +impl ProjectEnv { + /// The one resolver every verb consumes: project first, then std; + /// the charter makes overlap impossible, so order is not load-bearing. + pub fn resolve(&self, type_id: &str) -> Option { ... } + pub fn type_ids(&self) -> Vec<&str> { ... } // project ∪ std, for introspection +} + +/// Walk up from cwd to the nearest Aura.toml (the way cargo finds Cargo.toml). +pub fn discover() -> Option; + +/// Locate + load + verify. Every failure is a runtime refusal (exit 1). +pub fn load(root: &Path, release: bool) -> Result; +``` + +`load` steps, in order: + +1. Parse `Aura.toml` (`toml` crate — dependency admitted per-case: research-side + leaf binary, the clap precedent; never linkable into the frozen artifact). +2. `cargo metadata --format-version 1 --no-deps` in `root` (subprocess, parsed + with the existing `serde_json`): read `target_directory` + the package's lib + name → `/{debug|release}/`. + Missing file → `ProjectError::ArtifactMissing` with a `cargo build` hint. +3. `libloading::Library::new(path)`, then `Box::leak` — **load-and-hold**: the + library lives until process exit; there is no unload path. +4. Read the `AURA_PROJECT` symbol as `*const ProjectDescriptor`; validate the + C tier in order: `magic`, `descriptor_version`, then both stamps against + the host's own `aura_core::project::{RUSTC_VERSION, CORE_VERSION}`. + Any mismatch → `ProjectError::Incompatible` naming both sides' values. + **No Rust-tier field is dereferenced before this point.** +5. Charter checks: non-empty namespace; every listed id starts with + `::`; no duplicate within `type_ids() ∪ std_vocabulary_types()`; + every listed id resolves through the descriptor's own resolver fn + (list↔resolver cross-check — the introspection surface can then never + silently omit a project type). +6. Compute `dylib_sha256` over the loaded file's bytes; read the project git + commit best-effort (absent on failure, never an error). + +### aura-cli: threading (the fan-out edit) + +`main()` (`main.rs:4561`, dispatch at `4571-4581`), before dispatch: + +```rust +let project: Option = match project::discover() { + Some(root) => Some(project::load(&root, cli.release)?), // Err => exit 1 + None => None, +}; +``` + +- A new global flag on `Cli` (`main.rs:3669`): `#[arg(long, global = true)] + release: bool` — selects the artifact profile (debug is the default, cargo's + own default build). +- The ~14 `blueprint_from_json(..., &|t| std_vocabulary(t))` call sites + (`main.rs:2233 ... 4470`) and the op-script sites + (`graph_construct.rs:113/151/170/196`) take the merged resolver: + `&|t| env.resolve(t)` where `env` falls back to a std-only resolver when no + project is loaded. +- `default_registry()` (`main.rs:1513-1515`) gains the project form: inside a + project the registry root is `/` (`runs` from Aura.toml, default + `"runs"`); outside, the literal `"runs/runs.jsonl"` as today. Every + registry-family path (runs store, `families.jsonl`, `runs/traces/`, + `runs/blueprints/`) anchors at that same runs root. +- The inline `data_server::DEFAULT_DATA_PATH` sites (`main.rs:660, 677, 718, + 742`) read `[paths] data` when set, the current default otherwise. + +### aura-engine: manifest provenance (Tier-1 additive) + +After `topology_hash` (`crates/aura-engine/src/report.rs:58-59`), same +one-directional-widening idiom as `selection`/`instrument`/`topology_hash`: + +```rust +/// Identity of the loaded project dylib, when the run used one (cycle 0102). +/// Pre-0102 records read as `None`. +#[serde(default, skip_serializing_if = "Option::is_none")] +pub project: Option, +``` + +```rust +#[derive(Clone, Debug, PartialEq, serde::Serialize, serde::Deserialize)] +pub struct ProjectProvenance { + pub namespace: String, + pub dylib_sha256: String, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub commit: Option, +} +``` + +`sim_optimal_manifest` (`main.rs:166-184`) sets `project: None`; project-aware +paths stamp it post-hoc exactly as `instrument` (`main.rs:1664`) and +`topology_hash` (`main.rs:2821/2855/3554`) are stamped today. + +### Docs / ledger alignment (rides along) + +- `docs/project-layout.md:33` — drop "instrument/pip metadata" from the + Aura.toml line (geometry is the recorded sidecar, C15 realization); align + the field list to paths-only. +- `docs/design/INDEX.md` — C13 realization note: the descriptor contract, the + per-invocation-reload reading of "hot-reload", and the load-and-hold + one-shot-process scope boundary (a future long-running host must re-solve + reload; never in-process dlclose). C16/C17 Aura.toml field lists align to + paths-only. +- `docs/glossary.md` — Aura.toml entry aligns to paths-only. + +## Components + +| Component | Crate | New/changed | +|---|---|---| +| `project::ProjectDescriptor`, `StrSlice`, stamps, `aura_project!` | aura-core | new module + build.rs | +| `node_name()` namespace strip | aura-core | 1 function | +| `project::{discover, load, ProjectEnv, ProjectError}` | aura-cli | new module | +| Resolver/registry/data-path threading + `--release` global flag | aura-cli | fan-out edit | +| `RunManifest.project` + `ProjectProvenance` | aura-engine | Tier-1 field | +| Fixture project crate | `crates/aura-cli/tests/fixtures/demo-project/` | new (not a workspace member; empty `[workspace]`) | +| Docs/ledger alignment | docs/ | edits | + +New dependencies (per-case review, all confined to the `aura-cli` leaf binary, +frozen artifact untouched — the clap precedent): `libloading`, `toml`. +Explicitly NOT added: `cargo_metadata` (subprocess + existing `serde_json`), +`rustc_version` (5-line build.rs instead). + +## Data flow + +``` +cwd → discover() → Aura.toml root ──none──→ std-only resolver (today's behaviour) + │yes + ▼ +cargo metadata ─→ dylib path ─→ libloading (leak) ─→ AURA_PROJECT symbol + ▼ +C tier: magic → version → rustc stamp → core stamp (mismatch → exit 1) + ▼ +charter: ns non-empty → ids prefixed → no duplicates → list↔resolver + ▼ +ProjectEnv { merged resolver, runs root, data root, provenance } + ▼ +dispatch_* → blueprint verbs resolve project ∪ std → run → manifest stamped +``` + +## Error handling + +All project-load failures are **runtime refusals, exit 1** (C14 partition: +well-formed command, environment not usable): Aura.toml parse error, cargo +metadata failure, missing artifact (with `cargo build` hint), missing/garbage +`AURA_PROJECT` symbol, magic/descriptor-version mismatch, stamp mismatch +(message names both sides: `project dylib built with rustc 1.XX, host expects +rustc 1.YY`), charter violations (each names the offending id). Absence of an +`Aura.toml` is not an error — it selects today's std-only behaviour. A panic +inside project node code propagates (one-shot process; loud authoring-bug +signal — decided on #180, item 3). + +## Testing strategy + +1. **Unit (aura-core):** descriptor constants; `node_name` strip (bare names + byte-identical; `ns::Type` → `type`); macro expansion compiles in a + doc-test. +2. **Unit (aura-cli, no dylib needed):** walk-up discovery (nested dirs, no + Aura.toml, cwd = root); Aura.toml parsing (empty file, partial keys, + unknown keys tolerated); charter checks + stamp comparison driven through + an in-process fake descriptor; artifact-path derivation from a canned + `cargo metadata` JSON. +3. **E2E (aura-cli integration test, the load-bearing proof):** build the + fixture project (`cargo build` in `tests/fixtures/demo-project/`, path-deps + on this workspace's aura-core), then via `CARGO_BIN_EXE_aura` with cwd + inside the fixture: `aura run ` resolves + `demo::...`, runs, exits 0; run twice → byte-identical reports (C1); + manifest carries `project.namespace = "demo"` and the sha256 of the loaded + dylib; `aura graph introspect --vocabulary` lists `demo::*` beside std. +4. **Refusals (e2e):** the same blueprint outside the fixture dir → + `UnknownNodeType`, exit 2 (existing contract); fixture with the dylib + deleted → exit 1 with build hint. +5. **Regression:** full workspace suite green unchanged; all existing goldens + byte-identical (outside-a-project paths must not move); the manifest + legacy-load round-trip gains the `project: None` assertion (report.rs + 273-330 pattern). + +## Acceptance criteria + +1. A project crate authored as shown above (fixture twin) runs end-to-end + through `run`, `sweep --list-axes`, and `graph introspect --vocabulary` + from inside its directory, bit-deterministically (two runs byte-identical). +2. Every load failure named in Error handling refuses with exit 1 and a + message naming the cause; no failure path panics or loads a wrong graph. +3. Outside a project directory, the entire existing test suite and all golden + files are byte-identical to pre-cycle behaviour. +4. A project run's manifest records namespace + dylib content hash (+ commit + when available); pre-0102 registry lines load unchanged. +5. `::` cannot reach param paths through default node names (unit-pinned); + project ids without the namespace prefix cannot load (charter-pinned). +6. The Aura.toml field lists in project-layout.md, C16/C17, and the glossary + agree on paths-only; the C13 realization note records the reload reading + and the load-and-hold scope boundary.