Bind ingestion sources to the graph by role key, not list position #275
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Current state
Ingestion sources are bound to the graph purely by list position, at one runtime seam and two construction seams. A source is not a graph node:
FlatGraphcarries a fourth fieldsources: Vec<SourceSpec>parallel tonodes(crates/aura-engine/src/harness.rs:210-215), where eachSourceSpec { kind, targets: Vec<Target{node, slot}> }(harness.rs:39-52) names, by raw index, which node input slots one source position fans into.The positional coupling lives at three points (verified against the tree):
Harness::run(sources_in: Vec<Box<dyn Source>>)matches producersources_in[s]to wiringself.sources[s]by index, guarded only by an arity assert (harness.rs:371-378). The per-cycle push forwards the popped value intonodes[t.node].inputs[t.slot]by raw index (harness.rs:411-416). No name, no id.Role { name, targets, source: Option<ScalarKind> }(crates/aura-engine/src/blueprint.rs:127-134) lowers to exactly oneSourceSpecin role-declaration order (blueprint.rs:338). The role name is dropped; it never reaches theFlatGraph.column_for_rolemaps a role name to anM1Field(crates/aura-cli/src/binding.rs:52-60), consumed twice in the same canonical order — to declare thesource_roles inwrap_r(crates/aura-cli/src/main.rs:1540) and to open the columns.aura_ingest::open_columnsopens oneM1FieldSourceper field in that order, so "source index i = field i".The single invariant holding the two positional halves aligned is the canonical
M1Fielddeclaration order. A drift among role declaration, column-open order, andSourceSpecorder silently mis-wires a run — column X feeds role Y with no panic.Proposal
Replace the positional bind with a keyed by-name supply, on the binding axis only. The merge and the DAG stay untouched.
SourceSpeccarries a stablerolekey (the role name survives as a load-bearing key, not only a debug symbol) and an explicitorder: u32tie-break rank, decoupled from theVecposition.Harness::runtakes a keyed supply (role -> Box<dyn Source>) and resolves eachSourceSpecby key; a missing or misspelled role is a named error, not silent drift or a contextless arity panic.spec.orderinstead of theVecindex.The CLI drift surface (the canonical-order triple-alignment) disappears: the name carries the role->source assignment.
Why not "sources as input-less nodes"
The reframe considered first — make each source a native input-less
FlatGraphnode, dissolving the ingestion seam entirely — is incoherent with the engine model, on three verified grounds:fires()iterates a node's firing slice and returnsfalseon an empty one (harness.rs:461-491). This is exactly whyConstcarries a discarded dummy "clock" input — its own doc-comment: "a zero-input node never evaluates in the total-push engine" (crates/aura-std/src/const_node.rs:4-6).Harness::runpicks the global-min(timestamp, source_index)and only then istsfixed and handed to everyeval(harness.rs:387-408). A source node cannot be walked in a topological order whose cursor its own emission is required to define.So the seam stays; only the binding to it changes.
Guardrails any implementation must keep
s, never where the merge runs.orderrank must be stable and reconstructible; this is the one part that is not free, and it is what makes the change a ledger decision rather than a refactor — the tie-break moves from implicitVecposition to an explicit rank.FlatGraph; sources remain externally supplied torun, so disjoint sims stay concurrently runnable over a shared read-only archive.aura-ingestboundary; the type-erased per-cycle eval is unchanged.Impact on strategy development
eval, interior input roles (source: None) — is unchanged.us500.close, not positional slots), and loud on error (a mis-wire becomes a named start-time failure instead of a silently wrong backtest).sentimentrole bound to a recorded stream), without re-taring the positional alignment — the same additive-growth shape #124 anticipates for the DataServer metadata seam.Open (to be decided before implementation)
role/orderlive onSourceSpecand howorderis assigned (role-declaration order at lower time is the natural default, but must survive as an explicit field, not an implicit list index).runseam (role -> Box<dyn Source>map vs an ordered keyed list) and how a duplicate / missing / extra key is reported.ResolvedBindingcollapses to a keyed supply directly, retiringcolumns()'s positionalVec<M1Field>.Provenance: requested 2026-07-15 following an analysis of the ingestion path and the "sources as native nodes" question.
Design reconciliation (specify)
Spec:
source-role-key-binding. The "Open (to be decided before implementation)"forks in the issue body are resolved below as derived decisions (a groundable
rationale, not a user preference call), recorded before the spec is written.
Fork: layer of the by-name key. The keyed supply resolves above an
unchanged
Harness::run, rather than re-typing the primitiverunsignature.A pure resolver turns a keyed
Vec<(role, Box<dyn Source>)>supply into thepositional
Vec<Box<dyn Source>>the existingrunconsumes, inSourceSpecorder.
Basis: derived — C7/C23 make the raw-index positional supply the honest
primitive where no role name exists; hand-built engine graphs wire
SourceSpec { kind, targets }by rawTarget { node, slot }(crates/aura-engine/src/harness.rs:49). By-name binding is meaningful only
where a
Role.nameexists — the blueprint lowering that currently drops it(crates/aura-engine/src/blueprint.rs:338). Layering the resolver above
runtargets the drift exactly there, leaves the k-way merge byte-identical (the
merge loop is untouched), and gives the user-facing path a named
Resulterror while a raw-index arity mismatch stays an internal panic — a user
data-binding error and an engine wiring bug are different failure classes.
Fork:
rolerepresentation onSourceSpec.role: Option<String>—Some(name)for a spec lowered from a boundRole(carried fromRole.name),Nonefor a hand-built raw-index spec.Basis: derived — this realizes "the name survives where it exists"; a mandatory
Stringwould force""/ synthetic identifiers onto raw-index primitivesthat conceptually have no role.
Fork: explicit
order: u32, or theSourceSpecVec index as the rank.No separate
orderfield. The tie-break rank is theSourceSpecVec index;the resolver produces the positional supply in
SourceSpecorder, so themerge's existing source-index tie-break resolves on source declaration order,
independent of how the caller orders the keyed supply.
Basis: derived — C4 already defines the tie-break as "ties break by source
declaration order" (docs/design/INDEX.md:157), which the
SourceSpecVec indexalready encodes. An
orderfield equal to that index is a redundant secondsource of truth that can only drift from it. The coupling the issue targets is
between the caller's supply order and the wiring — dissolved by resolving into
SourceSpecorder, which needs no new field. Byte-identical today (declarationorder == the historical supply index). This resolves the issue body's own note
that "the explicit order rank … is the one part that is not free": the rank was
already present as the
SourceSpecVec order; only the by-name supply is new.Fork: CLI
ResolvedBinding/columns().columns()stays the open-list(which archive columns to open). The drift fix pairs each opened source with its
role name — a
Vec<(String, Box<dyn Source>)>supply resolved by name — retiringthe reliance on
open_columnsandwrap_rsharing one canonical order(crates/aura-cli/src/binding.rs:9-11; crates/aura-cli/src/main.rs:1539, 1729).
Basis: derived — the by-name match at the run boundary makes the three-point
canonical-order alignment structurally unnecessary;
columns()as the columnopener is orthogonal to it.
Ledger placement. C4's guarantee is unchanged — the tie-break stays "by
source declaration order", which this design keeps. A realization note records
that the supply is resolved by name into declaration order, so supply ordering
is no longer load-bearing. A scoped refinement of the Construction-layer C23
statement (docs/design/INDEX.md:1653-1656) records that a
SourceSpec.roleisload-bearing for source binding, while every other flat-graph name (edges,
ports, composite boundaries) stays a non-load-bearing raw-index symbol.
The k-way merge and the per-cycle event loop are not touched; C1 determinism is
preserved by construction (the run loop is byte-identical, verified downstream by
the currently-green multi-source run tests).
Spec auto-signed (specify, /boss)
The spec
source-role-key-bindingwas signed autonomously: the Step-5grounding-check returned PASS — an independent fresh-context agent's verdict that
every load-bearing assumption the spec rests on traces to a currently-green test.
Ratified:
SourceSpecis{ kind, targets }today (harness.rs:49); the blueprintlowering drops
role.nametoday (blueprint.rs:338);runpairs sources to specs byindex and the merge picks by
(timestamp, source index); the four.sources-equalitytwins are green and both sides declare matching root role names (so the role-carrying
equality stays green); and the CLI opens columns and declares roles in one shared
canonical order. No human signed; under autonomous mode the PASS is the signature.
Capability signed: binding a harness's ingestion feeds to the graph by role name
rather than by supply list position — a mis-bind becomes a named start-time error
instead of a silently wrong run, and the k-way merge / event loop are untouched
(C1 byte-identical).
The spec is a git-ignored working file (never committed). The work continues on the
run branch and awaits review before any merge to main.
Scope decisions (planner)
Two decomposition boundaries the file-structure recon surfaced, decided as derived
scope calls:
The full production supply path migrates to by-name. Not only the single-run
entry (
run_signal_r), but the sweep/campaign path (run_blueprint_memberandits
run_sourcescallers;campaign_run) route through the newrun_bound. Theshared supply helpers (
resolve_run_data,open_real_source,run_sources)return a keyed
Vec<(String, Box<dyn Source>)>— each opened column paired withits role name (
binding.entries()andbinding.columns()are the same canonicalorder, so the zip is sound).
Basis: derived — the silent-mis-wire class #275 removes is most costly on the
campaign / sweep path (many cells per run; the per-cell-fault incident behind
#272 is the reference). Migrating only the single-run entry would ship the fix
where a mis-wire is cheapest and skip it where it is dearest.
The 28 fieldtest
SourceSpecliterals are updated to the raw (no-role) form.They sit in separate packages the workspace build does not compile, but they are
in-tree path-dependencies on
aura-engine; a path-dep package that no longercompiles against the current engine is in-tree breakage, not a frozen snapshot.
The update is mechanical (
SourceSpec::raw(kind, targets)), isolated in its owntask, and verified by building the affected fieldtest packages.
Basis: derived — tree coherence for in-tree path-deps; the alternative (leave
them) is silent bit-rot of the tracked fieldtest corpus.
The raw-index engine / composites / ingest tests keep the positional
run(Vec)primitive (their specs carry no role); only the ResolvedBinding-bearing production
sites move to
run_bound.Scope correction (implement) — fieldtest sweep withdrawn
The earlier scope decision to sweep the 28 fieldtest
SourceSpecliterals rested ona premise that turned out false. During implementation the fieldtest packages were
found to already fail to build against the current engine API, for reasons wholly
unrelated to
SourceSpec:Harness::bootstrapnow takes a singleFlatGraphargument (the fieldtests still call the old 3-arg
(nodes, sources, edges)form),aura_core::InputSpecis gone,Recorder::new/SimBroker::newsignatures drifted,RMetricslost fields these tests reference, andScalarhelpers were renamed. Thiswas verified to pre-date this change (stashing the
SourceSpecedits and rebuildingreproduces the identical error set).
So the fieldtest corpus is pre-existing bit-rot from earlier cycles, not an
otherwise-buildable in-tree path-dep. A partial
SourceSpecmigration neither revivesnor further breaks it, and touching it only adds churn (plus regenerated lockfiles) to
non-building packages. The sweep was therefore reverted; reviving the corpus against
the current engine API is separate work, out of scope here.
The engine + CLI change is complete and workspace-green regardless — the fieldtests
are not workspace members, so they never gated the build.