Brummel
d2d0d0f4fd
Grounding-check PASS as the autonomous signature: 7 load-bearing current-behaviour assumptions each ratified by a named green test; greenfield surfaces (descriptor, loader, Aura.toml) excluded per contract. Design decisions and provenance: the decision-log on the milestone reference issue. refs #180
428 lines
19 KiB
Markdown
428 lines
19 KiB
Markdown
# The project-as-crate load boundary — Design Spec
|
||
|
||
**Date:** 2026-07-02
|
||
**Status:** Draft — awaiting sign-off (boss run: grounding-check gate)
|
||
**Authors:** orchestrator + Claude
|
||
**Cycle:** 0102 — first implementation cycle of milestone "Project environment — the project-as-crate authoring loop" (#180; the full fork-decision log with grounds is the decision-log comment on #180)
|
||
|
||
## Goal
|
||
|
||
Make a research project a loadable external Rust crate: an `aura` invocation
|
||
run inside a project directory discovers the project root (`Aura.toml`),
|
||
locates and loads the project's cdylib, verifies ABI compatibility
|
||
(refuse-don't-guess), merges the project's node vocabulary with
|
||
`std_vocabulary`, and runs every blueprint verb against the merged vocabulary —
|
||
bit-deterministically, with the loaded dylib's identity recorded in the run
|
||
manifest. Outside a project directory, behaviour stays byte-identical to
|
||
today. The `aura new` scaffolder is the next cycle, not this one.
|
||
|
||
## Architecture
|
||
|
||
Two new surfaces and one threading change:
|
||
|
||
1. **`aura-core::project` (new module + build.rs)** — the shared descriptor
|
||
contract both sides compile: a `#[repr(C)]` `ProjectDescriptor` with a
|
||
**C-ABI tier** (magic, descriptor version, rustc/aura-core version stamps,
|
||
namespace) that the host reads **before** trusting anything, and a
|
||
**Rust-ABI tier** (vocabulary resolver fn, enumerable type-id list) touched
|
||
only after the stamps match. An `aura_project!` macro generates the single
|
||
exported `AURA_PROJECT` static. Because the project build recompiles
|
||
aura-core under the project's own toolchain, the stamps baked into
|
||
aura-core's constants are automatically per-side-correct.
|
||
2. **`aura-cli::project` (new module)** — discovery (cargo-style walk-up to
|
||
`Aura.toml`), artifact location via `cargo metadata` (subprocess +
|
||
`serde_json`, no new crate), `libloading` load-and-hold (the `Library` is
|
||
leaked; never unloaded), the two-tier stamp gate, the vocabulary charter
|
||
checks, and the merged resolver.
|
||
3. **Per-invocation project context** — `main()` resolves an
|
||
`Option<ProjectEnv>` once (None outside a project) and threads it into the
|
||
`dispatch_*` functions; the ~14 inline `&|t| std_vocabulary(t)` resolver
|
||
sites, the ~10 `default_registry()` sites, and the inline
|
||
`data_server::DEFAULT_DATA_PATH` sites read from it. Outside a project the
|
||
context is `None` and every path collapses to the current literal.
|
||
|
||
Design constraints (decided on #180, restated where load-bearing):
|
||
|
||
- **Stamp before trust.** The stamp cannot certify the channel it rides on;
|
||
the C tier uses only C-compatible field types (integers, ptr+len string
|
||
slices) and is validated before any Rust-ABI field is dereferenced.
|
||
- **Vocabulary charter.** Project type-ids MUST be namespace-prefixed
|
||
(`<ns>::Type`, separator `::`); std ids stay bare. The loader refuses (exit
|
||
1): a missing/empty namespace, a listed id without the `<ns>::` prefix, a
|
||
duplicate id within the merged set, and a listed id the resolver does not
|
||
resolve (list↔resolver cross-check). The default node name strips the
|
||
namespace, so `::` never enters the param-path address space.
|
||
- **Zero-arg inheritance.** Project vocabularies resolve zero-argument
|
||
builders only, exactly as `std_vocabulary` scopes itself; construction-arg
|
||
widening remains the queued additive extension (#156).
|
||
- **Aura.toml is paths only.** `[paths] data`, `[paths] runs` — both optional.
|
||
No instrument/pip metadata (the recorded geometry sidecar stays the only
|
||
source — C15 realization, #124), no default symbol/window/broker, no dylib
|
||
path, no namespace (the descriptor is the namespace's single source; one
|
||
refinement over the #180 log's item 7, same anti-drift rationale).
|
||
- **Author builds, aura loads.** No cargo-build orchestration inside aura;
|
||
a missing artifact refuses with a build hint. Per-invocation load of the
|
||
freshest build IS the v1 "hot-reload" (C13 reading to be recorded in the
|
||
ledger, with the one-shot-process scope boundary of load-and-hold).
|
||
|
||
## Concrete code shapes
|
||
|
||
### The user-facing project (the acceptance-criterion evidence)
|
||
|
||
A project crate, in its own directory outside this repo (fixture twin:
|
||
`crates/aura-cli/tests/fixtures/demo-project/`):
|
||
|
||
```toml
|
||
# ger40-lab/Cargo.toml
|
||
[package]
|
||
name = "ger40-lab"
|
||
version = "0.1.0"
|
||
edition = "2024"
|
||
|
||
[lib]
|
||
crate-type = ["cdylib"]
|
||
|
||
[dependencies]
|
||
aura-core = { path = "../aura/crates/aura-core" } # git dep in real projects
|
||
|
||
[workspace] # standalone workspace root (docs/project-layout.md note)
|
||
```
|
||
|
||
```toml
|
||
# ger40-lab/Aura.toml — static project context only (C17); every key optional
|
||
[paths]
|
||
data = "/mnt/tickdata/Pepperstone" # data archive root; default: data-server default
|
||
runs = "runs" # registry root, relative to the project root; default: "runs"
|
||
```
|
||
|
||
```rust
|
||
// ger40-lab/src/lib.rs
|
||
use aura_core::PrimitiveBuilder;
|
||
mod third_candle; // an ordinary Node + PrimitiveBuilder factory, as in aura-std
|
||
|
||
fn vocabulary(type_id: &str) -> Option<PrimitiveBuilder> {
|
||
match type_id {
|
||
"ger40::ThirdCandle" => Some(third_candle::builder()),
|
||
_ => None,
|
||
}
|
||
}
|
||
|
||
fn type_ids() -> &'static [&'static str] {
|
||
&["ger40::ThirdCandle"]
|
||
}
|
||
|
||
aura_core::aura_project! {
|
||
namespace: "ger40",
|
||
vocabulary: vocabulary,
|
||
type_ids: type_ids,
|
||
}
|
||
```
|
||
|
||
```console
|
||
$ cd ger40-lab && cargo build
|
||
$ aura run blueprints/third-candle.json # blueprint's "type": "ger40::ThirdCandle"
|
||
$ aura sweep blueprints/third-candle.json --list-axes
|
||
$ aura graph introspect --vocabulary # lists std AND ger40::* ids
|
||
```
|
||
|
||
### aura-core: descriptor + stamps (new `src/project.rs`, new `build.rs`)
|
||
|
||
```rust
|
||
// aura-core/build.rs (new; the workspace's second build.rs after aura-cli's)
|
||
fn main() {
|
||
let rustc = std::env::var("RUSTC").unwrap_or_else(|_| "rustc".into());
|
||
let out = std::process::Command::new(rustc).arg("--version").output()
|
||
.expect("rustc --version");
|
||
println!("cargo:rustc-env=AURA_RUSTC_VERSION={}",
|
||
String::from_utf8_lossy(&out.stdout).trim());
|
||
}
|
||
```
|
||
|
||
```rust
|
||
// aura-core/src/project.rs (new)
|
||
pub const AURA_DESCRIPTOR_MAGIC: u64 = 0x4155_5241_5052_4F4A; // "AURAPROJ"
|
||
pub const AURA_DESCRIPTOR_VERSION: u32 = 1;
|
||
pub const AURA_PROJECT_SYMBOL: &[u8] = b"AURA_PROJECT\0";
|
||
|
||
/// Baked at aura-core compile time — i.e. per consuming build, which is what
|
||
/// makes the host/dylib comparison meaningful (each side recompiles aura-core).
|
||
pub const RUSTC_VERSION: &str = env!("AURA_RUSTC_VERSION");
|
||
pub const CORE_VERSION: &str = env!("CARGO_PKG_VERSION");
|
||
|
||
/// C-ABI string slice: ptr + len over 'static UTF-8 (no NUL convention needed).
|
||
#[repr(C)]
|
||
pub struct StrSlice { pub ptr: *const u8, pub len: usize }
|
||
impl StrSlice {
|
||
pub const fn new(s: &'static str) -> Self {
|
||
Self { ptr: s.as_ptr(), len: s.len() }
|
||
}
|
||
}
|
||
|
||
#[repr(C)]
|
||
pub struct ProjectDescriptor {
|
||
// ---- C tier: validated BEFORE any Rust-ABI field is touched ----
|
||
pub magic: u64,
|
||
pub descriptor_version: u32,
|
||
pub rustc_version: StrSlice,
|
||
pub aura_core_version: StrSlice,
|
||
pub namespace: StrSlice,
|
||
// ---- Rust tier: only after both stamps match ----
|
||
pub vocabulary: fn(&str) -> Option<PrimitiveBuilder>,
|
||
pub type_ids: fn() -> &'static [&'static str],
|
||
}
|
||
unsafe impl Sync for ProjectDescriptor {}
|
||
|
||
#[macro_export]
|
||
macro_rules! aura_project {
|
||
(namespace: $ns:literal, vocabulary: $vocab:path, type_ids: $ids:path $(,)?) => {
|
||
#[unsafe(no_mangle)]
|
||
pub static AURA_PROJECT: $crate::project::ProjectDescriptor =
|
||
$crate::project::ProjectDescriptor {
|
||
magic: $crate::project::AURA_DESCRIPTOR_MAGIC,
|
||
descriptor_version: $crate::project::AURA_DESCRIPTOR_VERSION,
|
||
rustc_version: $crate::project::StrSlice::new($crate::project::RUSTC_VERSION),
|
||
aura_core_version: $crate::project::StrSlice::new($crate::project::CORE_VERSION),
|
||
namespace: $crate::project::StrSlice::new($ns),
|
||
vocabulary: $vocab,
|
||
type_ids: $ids,
|
||
};
|
||
};
|
||
}
|
||
```
|
||
|
||
### aura-core: default node name strips the namespace
|
||
|
||
Before (`crates/aura-core/src/node.rs:143-147`):
|
||
|
||
```rust
|
||
pub fn node_name(&self) -> String {
|
||
self.instance_name.clone()
|
||
.unwrap_or_else(|| self.name.to_ascii_lowercase())
|
||
}
|
||
```
|
||
|
||
After (std ids carry no `::`, so existing behaviour is byte-identical):
|
||
|
||
```rust
|
||
pub fn node_name(&self) -> String {
|
||
self.instance_name.clone().unwrap_or_else(|| {
|
||
let bare = self.name.rsplit("::").next().unwrap_or(self.name);
|
||
bare.to_ascii_lowercase()
|
||
})
|
||
}
|
||
```
|
||
|
||
Consequence: param paths (built from node names, `blueprint.rs` `collect_params`)
|
||
never contain `::`; the member_key sanitizer (`main.rs:215-219`,
|
||
charset `[A-Za-z0-9._-]`) is reached only by names, so no new
|
||
filesystem-portability class opens. An explicit `.named("a::b")` sanitizes to
|
||
`a__b` as any out-of-charset byte does today.
|
||
|
||
### aura-cli: loader (new `src/project.rs`)
|
||
|
||
```rust
|
||
pub struct AuraToml { pub data: Option<PathBuf>, pub runs: Option<PathBuf> }
|
||
|
||
pub struct ProjectEnv {
|
||
pub root: PathBuf, // directory containing Aura.toml
|
||
pub toml: AuraToml,
|
||
pub namespace: String,
|
||
pub dylib_path: PathBuf,
|
||
pub dylib_sha256: String, // provenance (sha2 is already a dependency)
|
||
pub commit: Option<String>, // best-effort: git -C <root> rev-parse HEAD (+ -dirty)
|
||
/// fn pointer copied out of the descriptor — plain `fn`, and valid for
|
||
/// 'static because the Library is leaked (load-and-hold).
|
||
resolver: fn(&str) -> Option<PrimitiveBuilder>,
|
||
type_id_list: &'static [&'static str],
|
||
}
|
||
|
||
impl ProjectEnv {
|
||
/// The one resolver every verb consumes: project first, then std;
|
||
/// the charter makes overlap impossible, so order is not load-bearing.
|
||
pub fn resolve(&self, type_id: &str) -> Option<PrimitiveBuilder> { ... }
|
||
pub fn type_ids(&self) -> Vec<&str> { ... } // project ∪ std, for introspection
|
||
}
|
||
|
||
/// Walk up from cwd to the nearest Aura.toml (the way cargo finds Cargo.toml).
|
||
pub fn discover() -> Option<PathBuf>;
|
||
|
||
/// Locate + load + verify. Every failure is a runtime refusal (exit 1).
|
||
pub fn load(root: &Path, release: bool) -> Result<ProjectEnv, ProjectError>;
|
||
```
|
||
|
||
`load` steps, in order:
|
||
|
||
1. Parse `Aura.toml` (`toml` crate — dependency admitted per-case: research-side
|
||
leaf binary, the clap precedent; never linkable into the frozen artifact).
|
||
2. `cargo metadata --format-version 1 --no-deps` in `root` (subprocess, parsed
|
||
with the existing `serde_json`): read `target_directory` + the package's lib
|
||
name → `<target>/{debug|release}/<DLL_PREFIX><snake_name><DLL_SUFFIX>`.
|
||
Missing file → `ProjectError::ArtifactMissing` with a `cargo build` hint.
|
||
3. `libloading::Library::new(path)`, then `Box::leak` — **load-and-hold**: the
|
||
library lives until process exit; there is no unload path.
|
||
4. Read the `AURA_PROJECT` symbol as `*const ProjectDescriptor`; validate the
|
||
C tier in order: `magic`, `descriptor_version`, then both stamps against
|
||
the host's own `aura_core::project::{RUSTC_VERSION, CORE_VERSION}`.
|
||
Any mismatch → `ProjectError::Incompatible` naming both sides' values.
|
||
**No Rust-tier field is dereferenced before this point.**
|
||
5. Charter checks: non-empty namespace; every listed id starts with
|
||
`<namespace>::`; no duplicate within `type_ids() ∪ std_vocabulary_types()`;
|
||
every listed id resolves through the descriptor's own resolver fn
|
||
(list↔resolver cross-check — the introspection surface can then never
|
||
silently omit a project type).
|
||
6. Compute `dylib_sha256` over the loaded file's bytes; read the project git
|
||
commit best-effort (absent on failure, never an error).
|
||
|
||
### aura-cli: threading (the fan-out edit)
|
||
|
||
`main()` (`main.rs:4561`, dispatch at `4571-4581`), before dispatch:
|
||
|
||
```rust
|
||
let project: Option<ProjectEnv> = match project::discover() {
|
||
Some(root) => Some(project::load(&root, cli.release)?), // Err => exit 1
|
||
None => None,
|
||
};
|
||
```
|
||
|
||
- A new global flag on `Cli` (`main.rs:3669`): `#[arg(long, global = true)]
|
||
release: bool` — selects the artifact profile (debug is the default, cargo's
|
||
own default build).
|
||
- The ~14 `blueprint_from_json(..., &|t| std_vocabulary(t))` call sites
|
||
(`main.rs:2233 ... 4470`) and the op-script sites
|
||
(`graph_construct.rs:113/151/170/196`) take the merged resolver:
|
||
`&|t| env.resolve(t)` where `env` falls back to a std-only resolver when no
|
||
project is loaded.
|
||
- `default_registry()` (`main.rs:1513-1515`) gains the project form: inside a
|
||
project the registry root is `<root>/<runs>` (`runs` from Aura.toml, default
|
||
`"runs"`); outside, the literal `"runs/runs.jsonl"` as today. Every
|
||
registry-family path (runs store, `families.jsonl`, `runs/traces/`,
|
||
`runs/blueprints/`) anchors at that same runs root.
|
||
- The inline `data_server::DEFAULT_DATA_PATH` sites (`main.rs:660, 677, 718,
|
||
742`) read `[paths] data` when set, the current default otherwise.
|
||
|
||
### aura-engine: manifest provenance (Tier-1 additive)
|
||
|
||
After `topology_hash` (`crates/aura-engine/src/report.rs:58-59`), same
|
||
one-directional-widening idiom as `selection`/`instrument`/`topology_hash`:
|
||
|
||
```rust
|
||
/// Identity of the loaded project dylib, when the run used one (cycle 0102).
|
||
/// Pre-0102 records read as `None`.
|
||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||
pub project: Option<ProjectProvenance>,
|
||
```
|
||
|
||
```rust
|
||
#[derive(Clone, Debug, PartialEq, serde::Serialize, serde::Deserialize)]
|
||
pub struct ProjectProvenance {
|
||
pub namespace: String,
|
||
pub dylib_sha256: String,
|
||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||
pub commit: Option<String>,
|
||
}
|
||
```
|
||
|
||
`sim_optimal_manifest` (`main.rs:166-184`) sets `project: None`; project-aware
|
||
paths stamp it post-hoc exactly as `instrument` (`main.rs:1664`) and
|
||
`topology_hash` (`main.rs:2821/2855/3554`) are stamped today.
|
||
|
||
### Docs / ledger alignment (rides along)
|
||
|
||
- `docs/project-layout.md:33` — drop "instrument/pip metadata" from the
|
||
Aura.toml line (geometry is the recorded sidecar, C15 realization); align
|
||
the field list to paths-only.
|
||
- `docs/design/INDEX.md` — C13 realization note: the descriptor contract, the
|
||
per-invocation-reload reading of "hot-reload", and the load-and-hold
|
||
one-shot-process scope boundary (a future long-running host must re-solve
|
||
reload; never in-process dlclose). C16/C17 Aura.toml field lists align to
|
||
paths-only.
|
||
- `docs/glossary.md` — Aura.toml entry aligns to paths-only.
|
||
|
||
## Components
|
||
|
||
| Component | Crate | New/changed |
|
||
|---|---|---|
|
||
| `project::ProjectDescriptor`, `StrSlice`, stamps, `aura_project!` | aura-core | new module + build.rs |
|
||
| `node_name()` namespace strip | aura-core | 1 function |
|
||
| `project::{discover, load, ProjectEnv, ProjectError}` | aura-cli | new module |
|
||
| Resolver/registry/data-path threading + `--release` global flag | aura-cli | fan-out edit |
|
||
| `RunManifest.project` + `ProjectProvenance` | aura-engine | Tier-1 field |
|
||
| Fixture project crate | `crates/aura-cli/tests/fixtures/demo-project/` | new (not a workspace member; empty `[workspace]`) |
|
||
| Docs/ledger alignment | docs/ | edits |
|
||
|
||
New dependencies (per-case review, all confined to the `aura-cli` leaf binary,
|
||
frozen artifact untouched — the clap precedent): `libloading`, `toml`.
|
||
Explicitly NOT added: `cargo_metadata` (subprocess + existing `serde_json`),
|
||
`rustc_version` (5-line build.rs instead).
|
||
|
||
## Data flow
|
||
|
||
```
|
||
cwd → discover() → Aura.toml root ──none──→ std-only resolver (today's behaviour)
|
||
│yes
|
||
▼
|
||
cargo metadata ─→ dylib path ─→ libloading (leak) ─→ AURA_PROJECT symbol
|
||
▼
|
||
C tier: magic → version → rustc stamp → core stamp (mismatch → exit 1)
|
||
▼
|
||
charter: ns non-empty → ids prefixed → no duplicates → list↔resolver
|
||
▼
|
||
ProjectEnv { merged resolver, runs root, data root, provenance }
|
||
▼
|
||
dispatch_* → blueprint verbs resolve project ∪ std → run → manifest stamped
|
||
```
|
||
|
||
## Error handling
|
||
|
||
All project-load failures are **runtime refusals, exit 1** (C14 partition:
|
||
well-formed command, environment not usable): Aura.toml parse error, cargo
|
||
metadata failure, missing artifact (with `cargo build` hint), missing/garbage
|
||
`AURA_PROJECT` symbol, magic/descriptor-version mismatch, stamp mismatch
|
||
(message names both sides: `project dylib built with rustc 1.XX, host expects
|
||
rustc 1.YY`), charter violations (each names the offending id). Absence of an
|
||
`Aura.toml` is not an error — it selects today's std-only behaviour. A panic
|
||
inside project node code propagates (one-shot process; loud authoring-bug
|
||
signal — decided on #180, item 3).
|
||
|
||
## Testing strategy
|
||
|
||
1. **Unit (aura-core):** descriptor constants; `node_name` strip (bare names
|
||
byte-identical; `ns::Type` → `type`); macro expansion compiles in a
|
||
doc-test.
|
||
2. **Unit (aura-cli, no dylib needed):** walk-up discovery (nested dirs, no
|
||
Aura.toml, cwd = root); Aura.toml parsing (empty file, partial keys,
|
||
unknown keys tolerated); charter checks + stamp comparison driven through
|
||
an in-process fake descriptor; artifact-path derivation from a canned
|
||
`cargo metadata` JSON.
|
||
3. **E2E (aura-cli integration test, the load-bearing proof):** build the
|
||
fixture project (`cargo build` in `tests/fixtures/demo-project/`, path-deps
|
||
on this workspace's aura-core), then via `CARGO_BIN_EXE_aura` with cwd
|
||
inside the fixture: `aura run <fixture blueprint>` resolves
|
||
`demo::...`, runs, exits 0; run twice → byte-identical reports (C1);
|
||
manifest carries `project.namespace = "demo"` and the sha256 of the loaded
|
||
dylib; `aura graph introspect --vocabulary` lists `demo::*` beside std.
|
||
4. **Refusals (e2e):** the same blueprint outside the fixture dir →
|
||
`UnknownNodeType`, exit 2 (existing contract); fixture with the dylib
|
||
deleted → exit 1 with build hint.
|
||
5. **Regression:** full workspace suite green unchanged; all existing goldens
|
||
byte-identical (outside-a-project paths must not move); the manifest
|
||
legacy-load round-trip gains the `project: None` assertion (report.rs
|
||
273-330 pattern).
|
||
|
||
## Acceptance criteria
|
||
|
||
1. A project crate authored as shown above (fixture twin) runs end-to-end
|
||
through `run`, `sweep --list-axes`, and `graph introspect --vocabulary`
|
||
from inside its directory, bit-deterministically (two runs byte-identical).
|
||
2. Every load failure named in Error handling refuses with exit 1 and a
|
||
message naming the cause; no failure path panics or loads a wrong graph.
|
||
3. Outside a project directory, the entire existing test suite and all golden
|
||
files are byte-identical to pre-cycle behaviour.
|
||
4. A project run's manifest records namespace + dylib content hash (+ commit
|
||
when available); pre-0102 registry lines load unchanged.
|
||
5. `::` cannot reach param paths through default node names (unit-pinned);
|
||
project ids without the namespace prefix cannot load (charter-pinned).
|
||
6. The Aura.toml field lists in project-layout.md, C16/C17, and the glossary
|
||
agree on paths-only; the C13 realization note records the reload reading
|
||
and the load-and-hold scope boundary.
|