Aura/docs/specs/0102-project-load-boundary.md

The project-as-crate load boundary — Design Spec

Date: 2026-07-02 Status: Draft — awaiting sign-off (boss run: grounding-check gate) Authors: orchestrator + Claude Cycle: 0102 — first implementation cycle of milestone "Project environment — the project-as-crate authoring loop" (#180; the full fork-decision log with grounds is the decision-log comment on #180)

Goal

Make a research project a loadable external Rust crate: an aura invocation run inside a project directory discovers the project root (Aura.toml), locates and loads the project's cdylib, verifies ABI compatibility (refuse-don't-guess), merges the project's node vocabulary with std_vocabulary, and runs every blueprint verb against the merged vocabulary — bit-deterministically, with the loaded dylib's identity recorded in the run manifest. Outside a project directory, behaviour stays byte-identical to today. The aura new scaffolder is the next cycle, not this one.

Architecture

Two new surfaces and one threading change:

1. aura-core::project (new module + build.rs) — the shared descriptor contract both sides compile: a #[repr(C)] ProjectDescriptor with a C-ABI tier (magic, descriptor version, rustc/aura-core version stamps, namespace) that the host reads before trusting anything, and a Rust-ABI tier (vocabulary resolver fn, enumerable type-id list) touched only after the stamps match. An aura_project! macro generates the single exported AURA_PROJECT static. Because the project build recompiles aura-core under the project's own toolchain, the stamps baked into aura-core's constants are automatically per-side-correct.
2. aura-cli::project (new module) — discovery (cargo-style walk-up to Aura.toml), artifact location via cargo metadata (subprocess + serde_json, no new crate), libloading load-and-hold (the Library is leaked; never unloaded), the two-tier stamp gate, the vocabulary charter checks, and the merged resolver.
3. Per-invocation project context — main() resolves an Option<ProjectEnv> once (None outside a project) and threads it into the dispatch_* functions; the ~14 inline &|t| std_vocabulary(t) resolver sites, the ~10 default_registry() sites, and the inline data_server::DEFAULT_DATA_PATH sites read from it. Outside a project the context is None and every path collapses to the current literal.

Design constraints (decided on #180, restated where load-bearing):

• Stamp before trust. The stamp cannot certify the channel it rides on; the C tier uses only C-compatible field types (integers, ptr+len string slices) and is validated before any Rust-ABI field is dereferenced.
• Vocabulary charter. Project type-ids MUST be namespace-prefixed (<ns>::Type, separator ::); std ids stay bare. The loader refuses (exit 1): a missing/empty namespace, a listed id without the <ns>:: prefix, a duplicate id within the merged set, and a listed id the resolver does not resolve (list↔resolver cross-check). The default node name strips the namespace, so :: never enters the param-path address space.
• Zero-arg inheritance. Project vocabularies resolve zero-argument builders only, exactly as std_vocabulary scopes itself; construction-arg widening remains the queued additive extension (#156).
• Aura.toml is paths only. [paths] data, [paths] runs — both optional. No instrument/pip metadata (the recorded geometry sidecar stays the only source — C15 realization, #124), no default symbol/window/broker, no dylib path, no namespace (the descriptor is the namespace's single source; one refinement over the #180 log's item 7, same anti-drift rationale).
• Author builds, aura loads. No cargo-build orchestration inside aura; a missing artifact refuses with a build hint. Per-invocation load of the freshest build IS the v1 "hot-reload" (C13 reading to be recorded in the ledger, with the one-shot-process scope boundary of load-and-hold).

Concrete code shapes

The user-facing project (the acceptance-criterion evidence)

A project crate, in its own directory outside this repo (fixture twin: crates/aura-cli/tests/fixtures/demo-project/):

# ger40-lab/Cargo.toml
[package]
name = "ger40-lab"
version = "0.1.0"
edition = "2024"

[lib]
crate-type = ["cdylib"]

[dependencies]
aura-core = { path = "../aura/crates/aura-core" }  # git dep in real projects

[workspace]  # standalone workspace root (docs/project-layout.md note)

# ger40-lab/Aura.toml — static project context only (C17); every key optional
[paths]
data = "/mnt/tickdata/Pepperstone"  # data archive root; default: data-server default
runs = "runs"                       # registry root, relative to the project root; default: "runs"

// ger40-lab/src/lib.rs
use aura_core::PrimitiveBuilder;
mod third_candle; // an ordinary Node + PrimitiveBuilder factory, as in aura-std

fn vocabulary(type_id: &str) -> Option<PrimitiveBuilder> {
    match type_id {
        "ger40::ThirdCandle" => Some(third_candle::builder()),
        _ => None,
    }
}

fn type_ids() -> &'static [&'static str] {
    &["ger40::ThirdCandle"]
}

aura_core::aura_project! {
    namespace: "ger40",
    vocabulary: vocabulary,
    type_ids: type_ids,
}

$ cd ger40-lab && cargo build
$ aura run blueprints/third-candle.json     # blueprint's "type": "ger40::ThirdCandle"
$ aura sweep blueprints/third-candle.json --list-axes
$ aura graph introspect --vocabulary        # lists std AND ger40::* ids

aura-core: descriptor + stamps (new src/project.rs, new build.rs)

// aura-core/build.rs (new; the workspace's second build.rs after aura-cli's)
fn main() {
    let rustc = std::env::var("RUSTC").unwrap_or_else(|_| "rustc".into());
    let out = std::process::Command::new(rustc).arg("--version").output()
        .expect("rustc --version");
    println!("cargo:rustc-env=AURA_RUSTC_VERSION={}",
        String::from_utf8_lossy(&out.stdout).trim());
}

// aura-core/src/project.rs (new)
pub const AURA_DESCRIPTOR_MAGIC: u64 = 0x4155_5241_5052_4F4A; // "AURAPROJ"
pub const AURA_DESCRIPTOR_VERSION: u32 = 1;
pub const AURA_PROJECT_SYMBOL: &[u8] = b"AURA_PROJECT\0";

/// Baked at aura-core compile time — i.e. per consuming build, which is what
/// makes the host/dylib comparison meaningful (each side recompiles aura-core).
pub const RUSTC_VERSION: &str = env!("AURA_RUSTC_VERSION");
pub const CORE_VERSION: &str = env!("CARGO_PKG_VERSION");

/// C-ABI string slice: ptr + len over 'static UTF-8 (no NUL convention needed).
#[repr(C)]
pub struct StrSlice { pub ptr: *const u8, pub len: usize }
impl StrSlice {
    pub const fn new(s: &'static str) -> Self {
        Self { ptr: s.as_ptr(), len: s.len() }
    }
}

#[repr(C)]
pub struct ProjectDescriptor {
    // ---- C tier: validated BEFORE any Rust-ABI field is touched ----
    pub magic: u64,
    pub descriptor_version: u32,
    pub rustc_version: StrSlice,
    pub aura_core_version: StrSlice,
    pub namespace: StrSlice,
    // ---- Rust tier: only after both stamps match ----
    pub vocabulary: fn(&str) -> Option<PrimitiveBuilder>,
    pub type_ids: fn() -> &'static [&'static str],
}
unsafe impl Sync for ProjectDescriptor {}

#[macro_export]
macro_rules! aura_project {
    (namespace: $ns:literal, vocabulary: $vocab:path, type_ids: $ids:path $(,)?) => {
        #[unsafe(no_mangle)]
        pub static AURA_PROJECT: $crate::project::ProjectDescriptor =
            $crate::project::ProjectDescriptor {
                magic: $crate::project::AURA_DESCRIPTOR_MAGIC,
                descriptor_version: $crate::project::AURA_DESCRIPTOR_VERSION,
                rustc_version: $crate::project::StrSlice::new($crate::project::RUSTC_VERSION),
                aura_core_version: $crate::project::StrSlice::new($crate::project::CORE_VERSION),
                namespace: $crate::project::StrSlice::new($ns),
                vocabulary: $vocab,
                type_ids: $ids,
            };
    };
}

aura-core: default node name strips the namespace

Before (crates/aura-core/src/node.rs:143-147):

pub fn node_name(&self) -> String {
    self.instance_name.clone()
        .unwrap_or_else(|| self.name.to_ascii_lowercase())
}

After (std ids carry no ::, so existing behaviour is byte-identical):

pub fn node_name(&self) -> String {
    self.instance_name.clone().unwrap_or_else(|| {
        let bare = self.name.rsplit("::").next().unwrap_or(self.name);
        bare.to_ascii_lowercase()
    })
}

Consequence: param paths (built from node names, blueprint.rs collect_params) never contain ::; the member_key sanitizer (main.rs:215-219, charset [A-Za-z0-9._-]) is reached only by names, so no new filesystem-portability class opens. An explicit .named("a::b") sanitizes to a__b as any out-of-charset byte does today.

aura-cli: loader (new src/project.rs)

pub struct AuraToml { pub data: Option<PathBuf>, pub runs: Option<PathBuf> }

pub struct ProjectEnv {
    pub root: PathBuf,            // directory containing Aura.toml
    pub toml: AuraToml,
    pub namespace: String,
    pub dylib_path: PathBuf,
    pub dylib_sha256: String,     // provenance (sha2 is already a dependency)
    pub commit: Option<String>,   // best-effort: git -C <root> rev-parse HEAD (+ -dirty)
    /// fn pointer copied out of the descriptor — plain `fn`, and valid for
    /// 'static because the Library is leaked (load-and-hold).
    resolver: fn(&str) -> Option<PrimitiveBuilder>,
    type_id_list: &'static [&'static str],
}

impl ProjectEnv {
    /// The one resolver every verb consumes: project first, then std;
    /// the charter makes overlap impossible, so order is not load-bearing.
    pub fn resolve(&self, type_id: &str) -> Option<PrimitiveBuilder> { ... }
    pub fn type_ids(&self) -> Vec<&str> { ... } // project ∪ std, for introspection
}

/// Walk up from cwd to the nearest Aura.toml (the way cargo finds Cargo.toml).
pub fn discover() -> Option<PathBuf>;

/// Locate + load + verify. Every failure is a runtime refusal (exit 1).
pub fn load(root: &Path, release: bool) -> Result<ProjectEnv, ProjectError>;

load steps, in order:

1. Parse Aura.toml (toml crate — dependency admitted per-case: research-side leaf binary, the clap precedent; never linkable into the frozen artifact).
2. cargo metadata --format-version 1 --no-deps in root (subprocess, parsed with the existing serde_json): read target_directory + the package's lib name → <target>/{debug|release}/<DLL_PREFIX><snake_name><DLL_SUFFIX>. Missing file → ProjectError::ArtifactMissing with a cargo build hint.
3. libloading::Library::new(path), then Box::leak — load-and-hold: the library lives until process exit; there is no unload path.
4. Read the AURA_PROJECT symbol as *const ProjectDescriptor; validate the C tier in order: magic, descriptor_version, then both stamps against the host's own aura_core::project::{RUSTC_VERSION, CORE_VERSION}. Any mismatch → ProjectError::Incompatible naming both sides' values. No Rust-tier field is dereferenced before this point.
5. Charter checks: non-empty namespace; every listed id starts with <namespace>::; no duplicate within type_ids() ∪ std_vocabulary_types(); every listed id resolves through the descriptor's own resolver fn (list↔resolver cross-check — the introspection surface can then never silently omit a project type).
6. Compute dylib_sha256 over the loaded file's bytes; read the project git commit best-effort (absent on failure, never an error).

aura-cli: threading (the fan-out edit)

main() (main.rs:4561, dispatch at 4571-4581), before dispatch:

let project: Option<ProjectEnv> = match project::discover() {
    Some(root) => Some(project::load(&root, cli.release)?), // Err => exit 1
    None => None,
};

• A new global flag on Cli (main.rs:3669): #[arg(long, global = true)] release: bool — selects the artifact profile (debug is the default, cargo's own default build).
• The ~14 blueprint_from_json(..., &|t| std_vocabulary(t)) call sites (main.rs:2233 ... 4470) and the op-script sites (graph_construct.rs:113/151/170/196) take the merged resolver: &|t| env.resolve(t) where env falls back to a std-only resolver when no project is loaded.
• default_registry() (main.rs:1513-1515) gains the project form: inside a project the registry root is <root>/<runs> (runs from Aura.toml, default "runs"); outside, the literal "runs/runs.jsonl" as today. Every registry-family path (runs store, families.jsonl, runs/traces/, runs/blueprints/) anchors at that same runs root.
• The inline data_server::DEFAULT_DATA_PATH sites (main.rs:660, 677, 718, 742) read [paths] data when set, the current default otherwise.

aura-engine: manifest provenance (Tier-1 additive)

After topology_hash (crates/aura-engine/src/report.rs:58-59), same one-directional-widening idiom as selection/instrument/topology_hash:

/// Identity of the loaded project dylib, when the run used one (cycle 0102).
/// Pre-0102 records read as `None`.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub project: Option<ProjectProvenance>,

#[derive(Clone, Debug, PartialEq, serde::Serialize, serde::Deserialize)]
pub struct ProjectProvenance {
    pub namespace: String,
    pub dylib_sha256: String,
    #[serde(default, skip_serializing_if = "Option::is_none")]
    pub commit: Option<String>,
}

sim_optimal_manifest (main.rs:166-184) sets project: None; project-aware paths stamp it post-hoc exactly as instrument (main.rs:1664) and topology_hash (main.rs:2821/2855/3554) are stamped today.

Docs / ledger alignment (rides along)

• docs/project-layout.md:33 — drop "instrument/pip metadata" from the Aura.toml line (geometry is the recorded sidecar, C15 realization); align the field list to paths-only.
• docs/design/INDEX.md — C13 realization note: the descriptor contract, the per-invocation-reload reading of "hot-reload", and the load-and-hold one-shot-process scope boundary (a future long-running host must re-solve reload; never in-process dlclose). C16/C17 Aura.toml field lists align to paths-only.
• docs/glossary.md — Aura.toml entry aligns to paths-only.

Components

Component	Crate	New/changed
project::ProjectDescriptor, StrSlice, stamps, aura_project!	aura-core	new module + build.rs
node_name() namespace strip	aura-core	1 function
project::{discover, load, ProjectEnv, ProjectError}	aura-cli	new module
Resolver/registry/data-path threading + --release global flag	aura-cli	fan-out edit
RunManifest.project + ProjectProvenance	aura-engine	Tier-1 field
Fixture project crate	crates/aura-cli/tests/fixtures/demo-project/	new (not a workspace member; empty [workspace])
Docs/ledger alignment	docs/	edits

New dependencies (per-case review, all confined to the aura-cli leaf binary, frozen artifact untouched — the clap precedent): libloading, toml. Explicitly NOT added: cargo_metadata (subprocess + existing serde_json), rustc_version (5-line build.rs instead).

Data flow

cwd → discover() → Aura.toml root ──none──→ std-only resolver (today's behaviour)
        │yes
        ▼
cargo metadata ─→ dylib path ─→ libloading (leak) ─→ AURA_PROJECT symbol
        ▼
C tier: magic → version → rustc stamp → core stamp   (mismatch → exit 1)
        ▼
charter: ns non-empty → ids prefixed → no duplicates → list↔resolver
        ▼
ProjectEnv { merged resolver, runs root, data root, provenance }
        ▼
dispatch_* → blueprint verbs resolve project ∪ std → run → manifest stamped

Error handling

All project-load failures are runtime refusals, exit 1 (C14 partition: well-formed command, environment not usable): Aura.toml parse error, cargo metadata failure, missing artifact (with cargo build hint), missing/garbage AURA_PROJECT symbol, magic/descriptor-version mismatch, stamp mismatch (message names both sides: project dylib built with rustc 1.XX, host expects rustc 1.YY), charter violations (each names the offending id). Absence of an Aura.toml is not an error — it selects today's std-only behaviour. A panic inside project node code propagates (one-shot process; loud authoring-bug signal — decided on #180, item 3).

Testing strategy

1. Unit (aura-core): descriptor constants; node_name strip (bare names byte-identical; ns::Type → type); macro expansion compiles in a doc-test.
2. Unit (aura-cli, no dylib needed): walk-up discovery (nested dirs, no Aura.toml, cwd = root); Aura.toml parsing (empty file, partial keys, unknown keys tolerated); charter checks + stamp comparison driven through an in-process fake descriptor; artifact-path derivation from a canned cargo metadata JSON.
3. E2E (aura-cli integration test, the load-bearing proof): build the fixture project (cargo build in tests/fixtures/demo-project/, path-deps on this workspace's aura-core), then via CARGO_BIN_EXE_aura with cwd inside the fixture: aura run <fixture blueprint> resolves demo::..., runs, exits 0; run twice → byte-identical reports (C1); manifest carries project.namespace = "demo" and the sha256 of the loaded dylib; aura graph introspect --vocabulary lists demo::* beside std.
4. Refusals (e2e): the same blueprint outside the fixture dir → UnknownNodeType, exit 2 (existing contract); fixture with the dylib deleted → exit 1 with build hint.
5. Regression: full workspace suite green unchanged; all existing goldens byte-identical (outside-a-project paths must not move); the manifest legacy-load round-trip gains the project: None assertion (report.rs 273-330 pattern).

Acceptance criteria

1. A project crate authored as shown above (fixture twin) runs end-to-end through run, sweep --list-axes, and graph introspect --vocabulary from inside its directory, bit-deterministically (two runs byte-identical).
2. Every load failure named in Error handling refuses with exit 1 and a message naming the cause; no failure path panics or loads a wrong graph.
3. Outside a project directory, the entire existing test suite and all golden files are byte-identical to pre-cycle behaviour.
4. A project run's manifest records namespace + dylib content hash (+ commit when available); pre-0102 registry lines load unchanged.
5. :: cannot reach param paths through default node names (unit-pinned); project ids without the namespace prefix cannot load (charter-pinned).
6. The Aura.toml field lists in project-layout.md, C16/C17, and the glossary agree on paths-only; the C13 realization note records the reload reading and the load-and-hold scope boundary.