Brummel/Aura
1
0
Fork 0
Code Issues 31 Pull Requests Actions Packages Projects Releases Wiki Activity

audit: cycle 0102 tidy (drift-clean after two ledger-mirror alignments)

Browse Source 
Architect review of d7c935d..HEAD. What holds: C13/C16 separation
verified by diff (descriptor in aura-core::project, loader entirely in
aura-cli::project, aura-engine gains only the additive provenance
field); the C-tier-before-Rust-tier discipline is enforced in load()
(magic -> version -> stamps, fn pointers touched only after Ok);
C17/C24/invariant-9 hold (paths-only Aura.toml, injected merged
resolver, charter at the one seam); outside-a-project byte-identity
confirmed by the full green suite incl. every golden-bearing test.

Regression: cargo build --workspace rc=0; cargo test --workspace 862
passed / 0 failed (cli_run 133, project_load 7 e2e); clippy
--workspace --all-targets -D warnings rc=0; cargo doc --workspace
--no-deps 0 warnings. No baselines exist to move, so nothing to
ratify.

Drift resolved in this commit: two stale project-as-crate ledger
mirrors — the C24 status paragraph still calling the layer
sequencing-coupled/unbuilt, and the #109 open-thread bullet naming it
an unbuilt half — rewritten to record the landed load boundary (the
aura new scaffolder and the experiment-builder API stay open).

Carried holds (plan-prescribed bytes, cosmetic): the unreachable
rsplit fallback in node_name, the unreachable! build closure in one
aura-core test.

Ephemeral cycle artifacts removed per convention: spec 0102, plan 0102.

refs #180
This commit is contained in:
Brummel
2026-07-02 18:20:23 +02:00
parent 4928e289f7
commit b672a37903
3 changed files with 13 additions and 1884 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/design/INDEX.md
+13 -8
View File
@@ -1983,10 +1983,13 @@ the project-as-crate layer lands (#159, paired with #157's data-authoring surfac
(deliberate; fails clean as `UnknownNodeType`, never a silent wrong graph): recording
sinks (capture an `mpsc::Sender` — runtime identity, not param-generic data, C19) and
construction-arg builders (`LinComb` / `CostSum` / `SimBroker` / `Session` —
structural-axis args, a C20 concern), additively addable later (#156). Still
sequencing-coupled to the **project-as-crate authoring layer** (`aura new` /
`Aura.toml` / `cdylib` loading, C16/C17) and the **composable-orchestration** thread
(#109): topology-as-data is the substrate both stand on.
structural-axis args, a C20 concern), additively addable later (#156). The
**project-as-crate load boundary landed in cycle 0102** (`Aura.toml` discovery +
`cdylib` loading + merged project ∪ std vocabulary — see the C13 realization
note); of the two layers this paragraph used to name as sequencing-coupled,
what remains open is the `aura new` scaffolder (the milestone's next cycle) and
the **composable-orchestration** thread (#109): topology-as-data is the
substrate both stand on.
**Op-script grammar (`aura graph build`, the #157 wire surface).** The construction
op-script is a JSON **array of ops**, each object internally tagged by `"op"`,
@@ -2080,12 +2083,14 @@ artifact, and any run is deterministic once instantiated" (C1).
schema was settled paths-only in cycle 0102 (data archive root, runs dir —
see the C13 realization note); the load boundary itself shipped there.
- **The analysis meta-level — composable orchestration + project-as-crate authoring
(tracked: #109).** aura's differentiator (C21) has two unbuilt halves: (1) the
(tracked: #109).** aura's differentiator (C21) had two unbuilt halves: (1) the
orchestration axes (sweep / Monte-Carlo / walk-forward) becoming *composable tools*
a user wires into an analysis workflow, rather than today's hard-wired CLI verbs
(`sweep_family` / `walkforward_family` in `aura-cli`); and (2) the project-as-crate
authoring layer above (the `aura new` / experiment-builder API / `Aura.toml` thread
+ cdylib loading, C16/C17). On the meta-level, authoring a strategy is *wiring
(`sweep_family` / `walkforward_family` in `aura-cli`) — still open; and (2) the
project-as-crate authoring layer above — its **load boundary landed in cycle 0102**
(`Aura.toml` discovery + cdylib loading + merged vocabulary, C13 realization note;
the `aura new` scaffolder and the experiment-builder API remain open, C16/C17).
On the meta-level, authoring a strategy is *wiring
existing nodes* (C9 composition) + *defining the analysis framework* (C21), not
writing new nodes; the user wants it driven via the CLI, later the interactive
server / playground (C14/C22). **Resolved (2026-06-29) → C24.** The fork was
docs/plans/0102-project-load-boundary.md
-1449
View File
File diff suppressed because it is too large Load Diff
docs/specs/0102-project-load-boundary.md
-427
View File
@@ -1,427 +0,0 @@
# The project-as-crate load boundary — Design Spec
**Date:** 2026-07-02
**Status:** Draft — awaiting sign-off (boss run: grounding-check gate)
**Authors:** orchestrator + Claude
**Cycle:** 0102 — first implementation cycle of milestone "Project environment — the project-as-crate authoring loop" (#180; the full fork-decision log with grounds is the decision-log comment on #180)
## Goal
Make a research project a loadable external Rust crate: an `aura` invocation
run inside a project directory discovers the project root (`Aura.toml`),
locates and loads the project's cdylib, verifies ABI compatibility
(refuse-don't-guess), merges the project's node vocabulary with
`std_vocabulary`, and runs every blueprint verb against the merged vocabulary —
bit-deterministically, with the loaded dylib's identity recorded in the run
manifest. Outside a project directory, behaviour stays byte-identical to
today. The `aura new` scaffolder is the next cycle, not this one.
## Architecture
Two new surfaces and one threading change:
1. **`aura-core::project` (new module + build.rs)** — the shared descriptor
contract both sides compile: a `#[repr(C)]` `ProjectDescriptor` with a
**C-ABI tier** (magic, descriptor version, rustc/aura-core version stamps,
namespace) that the host reads **before** trusting anything, and a
**Rust-ABI tier** (vocabulary resolver fn, enumerable type-id list) touched
only after the stamps match. An `aura_project!` macro generates the single
exported `AURA_PROJECT` static. Because the project build recompiles
aura-core under the project's own toolchain, the stamps baked into
aura-core's constants are automatically per-side-correct.
2. **`aura-cli::project` (new module)** — discovery (cargo-style walk-up to
`Aura.toml`), artifact location via `cargo metadata` (subprocess +
`serde_json`, no new crate), `libloading` load-and-hold (the `Library` is
leaked; never unloaded), the two-tier stamp gate, the vocabulary charter
checks, and the merged resolver.
3. **Per-invocation project context** — `main()` resolves an
`Option<ProjectEnv>` once (None outside a project) and threads it into the
`dispatch_*` functions; the ~14 inline `&|t| std_vocabulary(t)` resolver
sites, the ~10 `default_registry()` sites, and the inline
`data_server::DEFAULT_DATA_PATH` sites read from it. Outside a project the
context is `None` and every path collapses to the current literal.
Design constraints (decided on #180, restated where load-bearing):
- **Stamp before trust.** The stamp cannot certify the channel it rides on;
the C tier uses only C-compatible field types (integers, ptr+len string
slices) and is validated before any Rust-ABI field is dereferenced.
- **Vocabulary charter.** Project type-ids MUST be namespace-prefixed
(`<ns>::Type`, separator `::`); std ids stay bare. The loader refuses (exit
1): a missing/empty namespace, a listed id without the `<ns>::` prefix, a
duplicate id within the merged set, and a listed id the resolver does not
resolve (list↔resolver cross-check). The default node name strips the
namespace, so `::` never enters the param-path address space.
- **Zero-arg inheritance.** Project vocabularies resolve zero-argument
builders only, exactly as `std_vocabulary` scopes itself; construction-arg
widening remains the queued additive extension (#156).
- **Aura.toml is paths only.** `[paths] data`, `[paths] runs` — both optional.
No instrument/pip metadata (the recorded geometry sidecar stays the only
source — C15 realization, #124), no default symbol/window/broker, no dylib
path, no namespace (the descriptor is the namespace's single source; one
refinement over the #180 log's item 7, same anti-drift rationale).
- **Author builds, aura loads.** No cargo-build orchestration inside aura;
a missing artifact refuses with a build hint. Per-invocation load of the
freshest build IS the v1 "hot-reload" (C13 reading to be recorded in the
ledger, with the one-shot-process scope boundary of load-and-hold).
## Concrete code shapes
### The user-facing project (the acceptance-criterion evidence)
A project crate, in its own directory outside this repo (fixture twin:
`crates/aura-cli/tests/fixtures/demo-project/`):
```toml
# ger40-lab/Cargo.toml
[package]
name = "ger40-lab"
version = "0.1.0"
edition = "2024"
[lib]
crate-type = ["cdylib"]
[dependencies]
aura-core = { path = "../aura/crates/aura-core" } # git dep in real projects
[workspace] # standalone workspace root (docs/project-layout.md note)
```
```toml
# ger40-lab/Aura.toml — static project context only (C17); every key optional
[paths]
data = "/mnt/tickdata/Pepperstone" # data archive root; default: data-server default
runs = "runs" # registry root, relative to the project root; default: "runs"
```
```rust
// ger40-lab/src/lib.rs
use aura_core::PrimitiveBuilder;
mod third_candle; // an ordinary Node + PrimitiveBuilder factory, as in aura-std
fn vocabulary(type_id: &str) -> Option<PrimitiveBuilder> {
match type_id {
"ger40::ThirdCandle" => Some(third_candle::builder()),
_ => None,
}
}
fn type_ids() -> &'static [&'static str] {
&["ger40::ThirdCandle"]
}
aura_core::aura_project! {
namespace: "ger40",
vocabulary: vocabulary,
type_ids: type_ids,
}
```
```console
$ cd ger40-lab && cargo build
$ aura run blueprints/third-candle.json # blueprint's "type": "ger40::ThirdCandle"
$ aura sweep blueprints/third-candle.json --list-axes
$ aura graph introspect --vocabulary # lists std AND ger40::* ids
```
### aura-core: descriptor + stamps (new `src/project.rs`, new `build.rs`)
```rust
// aura-core/build.rs (new; the workspace's second build.rs after aura-cli's)
fn main() {
let rustc = std::env::var("RUSTC").unwrap_or_else(|_| "rustc".into());
let out = std::process::Command::new(rustc).arg("--version").output()
.expect("rustc --version");
println!("cargo:rustc-env=AURA_RUSTC_VERSION={}",
String::from_utf8_lossy(&out.stdout).trim());
}
```
```rust
// aura-core/src/project.rs (new)
pub const AURA_DESCRIPTOR_MAGIC: u64 = 0x4155_5241_5052_4F4A; // "AURAPROJ"
pub const AURA_DESCRIPTOR_VERSION: u32 = 1;
pub const AURA_PROJECT_SYMBOL: &[u8] = b"AURA_PROJECT\0";
/// Baked at aura-core compile time — i.e. per consuming build, which is what
/// makes the host/dylib comparison meaningful (each side recompiles aura-core).
pub const RUSTC_VERSION: &str = env!("AURA_RUSTC_VERSION");
pub const CORE_VERSION: &str = env!("CARGO_PKG_VERSION");
/// C-ABI string slice: ptr + len over 'static UTF-8 (no NUL convention needed).
#[repr(C)]
pub struct StrSlice { pub ptr: *const u8, pub len: usize }
impl StrSlice {
pub const fn new(s: &'static str) -> Self {
Self { ptr: s.as_ptr(), len: s.len() }
}
}
#[repr(C)]
pub struct ProjectDescriptor {
// ---- C tier: validated BEFORE any Rust-ABI field is touched ----
pub magic: u64,
pub descriptor_version: u32,
pub rustc_version: StrSlice,
pub aura_core_version: StrSlice,
pub namespace: StrSlice,
// ---- Rust tier: only after both stamps match ----
pub vocabulary: fn(&str) -> Option<PrimitiveBuilder>,
pub type_ids: fn() -> &'static [&'static str],
}
unsafe impl Sync for ProjectDescriptor {}
#[macro_export]
macro_rules! aura_project {
(namespace: $ns:literal, vocabulary: $vocab:path, type_ids: $ids:path $(,)?) => {
#[unsafe(no_mangle)]
pub static AURA_PROJECT: $crate::project::ProjectDescriptor =
$crate::project::ProjectDescriptor {
magic: $crate::project::AURA_DESCRIPTOR_MAGIC,
descriptor_version: $crate::project::AURA_DESCRIPTOR_VERSION,
rustc_version: $crate::project::StrSlice::new($crate::project::RUSTC_VERSION),
aura_core_version: $crate::project::StrSlice::new($crate::project::CORE_VERSION),
namespace: $crate::project::StrSlice::new($ns),
vocabulary: $vocab,
type_ids: $ids,
};
};
}
```
### aura-core: default node name strips the namespace
Before (`crates/aura-core/src/node.rs:143-147`):
```rust
pub fn node_name(&self) -> String {
self.instance_name.clone()
.unwrap_or_else(|| self.name.to_ascii_lowercase())
}
```
After (std ids carry no `::`, so existing behaviour is byte-identical):
```rust
pub fn node_name(&self) -> String {
self.instance_name.clone().unwrap_or_else(|| {
let bare = self.name.rsplit("::").next().unwrap_or(self.name);
bare.to_ascii_lowercase()
})
}
```
Consequence: param paths (built from node names, `blueprint.rs` `collect_params`)
never contain `::`; the member_key sanitizer (`main.rs:215-219`,
charset `[A-Za-z0-9._-]`) is reached only by names, so no new
filesystem-portability class opens. An explicit `.named("a::b")` sanitizes to
`a__b` as any out-of-charset byte does today.
### aura-cli: loader (new `src/project.rs`)
```rust
pub struct AuraToml { pub data: Option<PathBuf>, pub runs: Option<PathBuf> }
pub struct ProjectEnv {
pub root: PathBuf, // directory containing Aura.toml
pub toml: AuraToml,
pub namespace: String,
pub dylib_path: PathBuf,
pub dylib_sha256: String, // provenance (sha2 is already a dependency)
pub commit: Option<String>, // best-effort: git -C <root> rev-parse HEAD (+ -dirty)
/// fn pointer copied out of the descriptor — plain `fn`, and valid for
/// 'static because the Library is leaked (load-and-hold).
resolver: fn(&str) -> Option<PrimitiveBuilder>,
type_id_list: &'static [&'static str],
}
impl ProjectEnv {
/// The one resolver every verb consumes: project first, then std;
/// the charter makes overlap impossible, so order is not load-bearing.
pub fn resolve(&self, type_id: &str) -> Option<PrimitiveBuilder> { ... }
pub fn type_ids(&self) -> Vec<&str> { ... } // project ∪ std, for introspection
}
/// Walk up from cwd to the nearest Aura.toml (the way cargo finds Cargo.toml).
pub fn discover() -> Option<PathBuf>;
/// Locate + load + verify. Every failure is a runtime refusal (exit 1).
pub fn load(root: &Path, release: bool) -> Result<ProjectEnv, ProjectError>;
```
`load` steps, in order:
1. Parse `Aura.toml` (`toml` crate — dependency admitted per-case: research-side
leaf binary, the clap precedent; never linkable into the frozen artifact).
2. `cargo metadata --format-version 1 --no-deps` in `root` (subprocess, parsed
with the existing `serde_json`): read `target_directory` + the package's lib
name → `<target>/{debug|release}/<DLL_PREFIX><snake_name><DLL_SUFFIX>`.
Missing file → `ProjectError::ArtifactMissing` with a `cargo build` hint.
3. `libloading::Library::new(path)`, then `Box::leak` — **load-and-hold**: the
library lives until process exit; there is no unload path.
4. Read the `AURA_PROJECT` symbol as `*const ProjectDescriptor`; validate the
C tier in order: `magic`, `descriptor_version`, then both stamps against
the host's own `aura_core::project::{RUSTC_VERSION, CORE_VERSION}`.
Any mismatch → `ProjectError::Incompatible` naming both sides' values.
**No Rust-tier field is dereferenced before this point.**
5. Charter checks: non-empty namespace; every listed id starts with
`<namespace>::`; no duplicate within `type_ids() ∪ std_vocabulary_types()`;
every listed id resolves through the descriptor's own resolver fn
(list↔resolver cross-check — the introspection surface can then never
silently omit a project type).
6. Compute `dylib_sha256` over the loaded file's bytes; read the project git
commit best-effort (absent on failure, never an error).
### aura-cli: threading (the fan-out edit)
`main()` (`main.rs:4561`, dispatch at `4571-4581`), before dispatch:
```rust
let project: Option<ProjectEnv> = match project::discover() {
Some(root) => Some(project::load(&root, cli.release)?), // Err => exit 1
None => None,
};
```
- A new global flag on `Cli` (`main.rs:3669`): `#[arg(long, global = true)]
release: bool` — selects the artifact profile (debug is the default, cargo's
own default build).
- The ~14 `blueprint_from_json(..., &|t| std_vocabulary(t))` call sites
(`main.rs:2233 ... 4470`) and the op-script sites
(`graph_construct.rs:113/151/170/196`) take the merged resolver:
`&|t| env.resolve(t)` where `env` falls back to a std-only resolver when no
project is loaded.
- `default_registry()` (`main.rs:1513-1515`) gains the project form: inside a
project the registry root is `<root>/<runs>` (`runs` from Aura.toml, default
`"runs"`); outside, the literal `"runs/runs.jsonl"` as today. Every
registry-family path (runs store, `families.jsonl`, `runs/traces/`,
`runs/blueprints/`) anchors at that same runs root.
- The inline `data_server::DEFAULT_DATA_PATH` sites (`main.rs:660, 677, 718,
742`) read `[paths] data` when set, the current default otherwise.
### aura-engine: manifest provenance (Tier-1 additive)
After `topology_hash` (`crates/aura-engine/src/report.rs:58-59`), same
one-directional-widening idiom as `selection`/`instrument`/`topology_hash`:
```rust
/// Identity of the loaded project dylib, when the run used one (cycle 0102).
/// Pre-0102 records read as `None`.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub project: Option<ProjectProvenance>,
```
```rust
#[derive(Clone, Debug, PartialEq, serde::Serialize, serde::Deserialize)]
pub struct ProjectProvenance {
pub namespace: String,
pub dylib_sha256: String,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub commit: Option<String>,
}
```
`sim_optimal_manifest` (`main.rs:166-184`) sets `project: None`; project-aware
paths stamp it post-hoc exactly as `instrument` (`main.rs:1664`) and
`topology_hash` (`main.rs:2821/2855/3554`) are stamped today.
### Docs / ledger alignment (rides along)
- `docs/project-layout.md:33` — drop "instrument/pip metadata" from the
Aura.toml line (geometry is the recorded sidecar, C15 realization); align
the field list to paths-only.
- `docs/design/INDEX.md` — C13 realization note: the descriptor contract, the
per-invocation-reload reading of "hot-reload", and the load-and-hold
one-shot-process scope boundary (a future long-running host must re-solve
reload; never in-process dlclose). C16/C17 Aura.toml field lists align to
paths-only.
- `docs/glossary.md` — Aura.toml entry aligns to paths-only.
## Components
| Component | Crate | New/changed |
|---|---|---|
| `project::ProjectDescriptor`, `StrSlice`, stamps, `aura_project!` | aura-core | new module + build.rs |
| `node_name()` namespace strip | aura-core | 1 function |
| `project::{discover, load, ProjectEnv, ProjectError}` | aura-cli | new module |
| Resolver/registry/data-path threading + `--release` global flag | aura-cli | fan-out edit |
| `RunManifest.project` + `ProjectProvenance` | aura-engine | Tier-1 field |
| Fixture project crate | `crates/aura-cli/tests/fixtures/demo-project/` | new (not a workspace member; empty `[workspace]`) |
| Docs/ledger alignment | docs/ | edits |
New dependencies (per-case review, all confined to the `aura-cli` leaf binary,
frozen artifact untouched — the clap precedent): `libloading`, `toml`.
Explicitly NOT added: `cargo_metadata` (subprocess + existing `serde_json`),
`rustc_version` (5-line build.rs instead).
## Data flow
```
cwd → discover() → Aura.toml root ──none──→ std-only resolver (today's behaviour)
│yes
▼
cargo metadata ─→ dylib path ─→ libloading (leak) ─→ AURA_PROJECT symbol
▼
C tier: magic → version → rustc stamp → core stamp (mismatch → exit 1)
▼
charter: ns non-empty → ids prefixed → no duplicates → list↔resolver
▼
ProjectEnv { merged resolver, runs root, data root, provenance }
▼
dispatch_* → blueprint verbs resolve project ∪ std → run → manifest stamped
```
## Error handling
All project-load failures are **runtime refusals, exit 1** (C14 partition:
well-formed command, environment not usable): Aura.toml parse error, cargo
metadata failure, missing artifact (with `cargo build` hint), missing/garbage
`AURA_PROJECT` symbol, magic/descriptor-version mismatch, stamp mismatch
(message names both sides: `project dylib built with rustc 1.XX, host expects
rustc 1.YY`), charter violations (each names the offending id). Absence of an
`Aura.toml` is not an error — it selects today's std-only behaviour. A panic
inside project node code propagates (one-shot process; loud authoring-bug
signal — decided on #180, item 3).
## Testing strategy
1. **Unit (aura-core):** descriptor constants; `node_name` strip (bare names
byte-identical; `ns::Type` → `type`); macro expansion compiles in a
doc-test.
2. **Unit (aura-cli, no dylib needed):** walk-up discovery (nested dirs, no
Aura.toml, cwd = root); Aura.toml parsing (empty file, partial keys,
unknown keys tolerated); charter checks + stamp comparison driven through
an in-process fake descriptor; artifact-path derivation from a canned
`cargo metadata` JSON.
3. **E2E (aura-cli integration test, the load-bearing proof):** build the
fixture project (`cargo build` in `tests/fixtures/demo-project/`, path-deps
on this workspace's aura-core), then via `CARGO_BIN_EXE_aura` with cwd
inside the fixture: `aura run <fixture blueprint>` resolves
`demo::...`, runs, exits 0; run twice → byte-identical reports (C1);
manifest carries `project.namespace = "demo"` and the sha256 of the loaded
dylib; `aura graph introspect --vocabulary` lists `demo::*` beside std.
4. **Refusals (e2e):** the same blueprint outside the fixture dir →
`UnknownNodeType`, exit 2 (existing contract); fixture with the dylib
deleted → exit 1 with build hint.
5. **Regression:** full workspace suite green unchanged; all existing goldens
byte-identical (outside-a-project paths must not move); the manifest
legacy-load round-trip gains the `project: None` assertion (report.rs
273-330 pattern).
## Acceptance criteria
1. A project crate authored as shown above (fixture twin) runs end-to-end
through `run`, `sweep --list-axes`, and `graph introspect --vocabulary`
from inside its directory, bit-deterministically (two runs byte-identical).
2. Every load failure named in Error handling refuses with exit 1 and a
message naming the cause; no failure path panics or loads a wrong graph.
3. Outside a project directory, the entire existing test suite and all golden
files are byte-identical to pre-cycle behaviour.
4. A project run's manifest records namespace + dylib content hash (+ commit
when available); pre-0102 registry lines load unchanged.
5. `::` cannot reach param paths through default node names (unit-pinned);
project ids without the namespace prefix cannot load (charter-pinned).
6. The Aura.toml field lists in project-layout.md, C16/C17, and the glossary
agree on paths-only; the C13 realization note records the reload reading
and the load-and-hold scope boundary.